Data engineering / TPL Media
Ninety-one sources. One dependable feed.
A configurable crawling framework that turns scattered city, police, and public-agency news into publish-ready local stories.
The challenge
Local news is spread across dozens of city, police, health, and transit websites, each with its own layout, date format, and failure modes. One-off scrapers do not scale, and a silent failure means a city quietly stops receiving news.
The engineering
- Designed a configurable base spider so a new source is mostly configuration, then expanded coverage to 91 source-specific crawlers.
- Used Playwright for JavaScript-rendered pages, kept publication dates separate from service dates, and rejected future-dated articles.
- Used Gemini to turn each scraped article into a short, structured entry built only from the source text.
- Published to Payload CMS with deduplication by city, source, and URL, and wrote failure and run-summary reports for every crawl.
The result
91 crawlers feed the local news product through one pipeline. Every run reports which sources failed or found nothing, so coverage gaps are visible instead of silent.
05.Open a connection
Production ML, computer vision, AI infrastructure, or a team looking for an engineer who ships. Tell me what you have in mind. Three fields, one conversation.
Fill out the form → hit send → I’ll reply to your email.
↓ download résumé · PDF