Newsbot – Self-Hosted News-Intelligence Pipeline (Python, SQLite, Ubuntu)
A self-hosted news-intelligence pipeline that polls 59 RSS feeds from 42 outlets every 15 minutes, stores and deduplicates every story in SQLite, and pushes to your phone. It's being extended to rank each day's biggest stories by how widely they're covered and to answer on-demand requests sent from the phone. It uses no paid services and no AI API calls.
- Python
- SQL (SQLite)
- RSS
- Ubuntu Server
- systemd
- ntfy.sh
- Git
- Claude Code
The Question
What's actually the biggest story today, measured by how many outlets are covering it rather than by any one editor's choice?
Every outlet decides for itself what's important. Newsbot measures importance from the outside instead: if dozens of independent outlets are covering the same story, that story is big. That turns "what's the news today" into a data problem. It means collecting broadly, grouping headlines that describe the same event, counting coverage, and handling stories that continue across multiple days.
The goal (in development): a short, ranked digest on the phone each morning, plus the ability to ask the server for fresh data at any time.
System Diagram
Collection, storage, health monitoring, backups, and push delivery are live. Dashed stages are in development.
- 1 Feeds 59 validated RSS feeds across 42 outlets
- 2 Collector Runs every 15 minutes on a systemd timer, writes to SQLite, and deduplicates items
- 3 SQLite + health Tracks the status of every feed, detects stale feeds, logs headline revisions, and keeps source-URL provenance
- 4 In development Clustering & ranking Group headlines into stories and rank them by coverage
- 5 In development Morning digest About 8am, short and ranked, with deduplication of continuing stories
- 6 Push to phone Push notifications to the phone through ntfy.sh, a free, open-source push service
- 7 In development Two-way requests Send a command from the phone and get a fresh update or data pull back
Constraints
- $0 budget.
- No paid AI API calls inside the bot, so ranking is built from data and arithmetic. It keeps the bot free and makes the ranking transparent and explainable from the counts.
- Fully isolated from other services on the same server, with its own directory, virtualenv, database, and user-scope timers.
Data-Quality Story
Most of the work so far has been data quality: catching duplicate records, broken feeds, and counting errors before they could distort the analysis.
Real Defects Found and Fixed
- Duplicate rows: BBC changes a fragment in its item IDs between fetches, which created duplicate rows. Fixed by normalizing the IDs, plus a migration of existing data.
- Inflated counts: the "times seen" metric counted per document instead of per collection run. Fixed in collector v0.2.
- Test polluting production: a validation script wrote to the live database by default. It now uses throwaway copies and needs an explicit flag to touch live data.
- Off-by-a-fraction retention bug: a backup exactly 14 days old was deleted a day early because of a microseconds vs. whole-seconds mismatch. It was caught by a control test and fixed in the script.
- Leftover sidecar files: backups were published in WAL mode, so every integrity check left sidecar files behind. Archives are now single self-contained files.
- Unicode crash: a Unicode character crashed the notification title path. Fixed by sending the title in the JSON body.
How I Validate the Checks
Every check was validated with negative controls, meaning deliberately planted errors that the check had to catch before its results were trusted. The feed validation included deliberately bad URLs that the check had to reject. A check that can't fail proves nothing.
Sample Digest & Exploratory Findings – In Development
Neither the digest nor the exploratory analysis exists yet, so there are no results to show here.
- Deliberately waiting: clustering and ranking will be written after about a week of real data. Tuning them on one evening's headlines would overfit.
- Next: clustering and ranking, then the 8am digest, then two-way phone requests.
- After that, exploratory analysis: output by outlet, publish-to-fetch lag, and headline-revision rates.
Tools & Constraints
Project Facts
| My role | Designed and directed, built with Claude Code. |
|---|---|
| Status | The collection, storage, health, backup, and push layers are live. Clustering, ranking, the morning digest, and two-way requests are in development. |
| Sources | 59 validated RSS feeds across 42 outlets |
| Schedule | Every 15 minutes on a systemd timer; nightly verified backups |
| Budget | $0 |
| AI usage | No paid AI API calls inside the bot |
| Isolation | Fully isolated from other services on the same server |
Tech Stack
- Python: collector, feed validation, migrations, backup and retention scripts
- SQL / SQLite: schema, deduplication, migrations, health and provenance tables
- Linux and automation: Ubuntu Server, systemd timers, self-hosted and unattended
- Reliability: verified backups, restore test, retention, health watchdog (being added)
- Delivery: ntfy.sh, a free, open-source push service
- Git / GitHub: private repo, commits verified by object hash
Current Status
As of 2026-09-20.
- Live: validated and deployed 59 feeds from 42 outlets; collection runs unattended every 15 minutes
- Live: nightly verified backups, with the restore path proved
- Live: ntfy push delivery proved end to end, with notifications received on the phone
- Being added: a health watchdog
- In development: clustering and ranking, then the 8am digest, then two-way phone requests
Source code is currently private.