wxbot – Automated Weather Data Pipeline with QC, Anomaly Detection & Forecast Scorecard

Where is the weather actually unusual today – not just hot?

• Raw extremes are predictable and boring. The hottest place on Earth is almost always the Persian Gulf, Sahara, or Death Valley; the coldest is Antarctica. The newsworthy question is anomaly: "Chicago is 22°F above normal" is news; "Kuwait is 115°F" is not.
• "Hottest right now" is a data-quality problem in disguise. A stuck sensor reading 140°F wins every time unless quality control catches it. Stations go silent, report stale data, disagree with neighbours, and issue corrections.
• Anomaly definitions have traps. Comparing an 8 a.m. temperature to the normal daily high makes every morning look cold. The definition has to be chosen by measurement and stated in every output.
• Forecast accuracy is measurable but rarely measured personally. Logging NWS forecasts and matching them to what actually happened produces an accuracy scorecard by city and lead time – a classic analyst deliverable.

Planned Data Sources (free, government-first)

Source	Data	Role
NOAA Aviation Weather Center	METAR observations from airport stations worldwide	Core observation feed
National Weather Service API	Forecasts, observations, severe alerts	Scorecard + alerts
NOAA Regional Climate Centers (ACIS)	1991–2020 station normals, daily records	Anomalies and records (US)
Storm Prediction Center / National Hurricane Center	Severe-storm outlooks, active hurricanes	Severe-weather layer
NOAA buoys & tides (NDBC, CO-OPS)	Ocean temp, waves, tides	Local brief
EPA AirNow (optional)	Air quality	Local brief

Constraints

• $0 budget
• Terms of use read and documented for every source before use
• No scraping – only published files and documented APIs
• No AI/LLM calls inside the pipeline – all summaries built from data and arithmetic

Planned to run 24/7 on a home server, scheduled with systemd services.

1. 1 Ingest NOAA/NWS data every ~5 minutes, conditional download
2. 2 Parse + QC Quality control with exclusion counts by reason
3. 3 Raw database 14-day rolling
4. 4 Nightly rollup Hourly → daily
5. 5 Durable database Aggregates, daily summaries, events, scorecard
6. 6 Analysis Extremes, anomalies, records, alerts, forecast scorecard
7. 7 Phone push Morning brief, weekly digest, on-demand replies

Data Quality Control

Every reading will pass a set of rules before it can count:

• Physical plausibility bounds, set from measured data, not guessed
• Internal consistency: whole-degree temperature vs. the precise tenths value in the report remarks
• Temporal jumps: change vs. the station's own previous report
• Spatial consistency: disagreement with nearby stations
• Staleness: observations too old to count as "current"
• Corrections: corrected reports will replace originals, with no double counting
• Every exclusion counted by reason – a filter that silently drops data is how a real record gets thrown away
• Negative controls: planted 140°F and stale readings must be excluded; a planted genuine extreme must be kept

Anomaly Definition

• Observed vs. 1991–2020 normals for that station and date
• The definition will be chosen by measurement to avoid the time-of-day trap, and stated in every output
• Records: record highs/lows set today

Forecast Accuracy Scorecard

• Log NWS forecasts daily from Phase 1 (unlogged days are unrecoverable)
• Match the forecast window to the official observation window (a known source of silent error)
• Error by city, lead time, and season; "biggest forecast miss yesterday"

Station Data-Quality Report

Which stations report bad data, go silent, or disagree with neighbours – tracked over time.

Data Lifecycle / Storage

• Tiered retention: raw 14 days → hourly aggregates 90 days → daily summaries 2 years
• Rollup verified before any deletion; disk budget with alarms
• Sizing will be set from measured rows/day and bytes/day, not estimates

• Measure before building. A formal measurement phase will answer every unknown about external services before any pipeline code is written.
• Negative controls on every check – a check that can't fail isn't a check.
• "Couldn't look" ≠ "nothing there." Distinct outcomes for data / no change / empty / error / unreachable; HTTP 200 will not be treated as success.
• Staleness is its own failure mode – the pipeline will flag data that stops updating instead of passing it off as current.
• Provenance: every run row stamped with the system invocation ID, traceable to logs.
• Verified backups: integrity-checked copies, test restores, retention only after verification.
• Reliability: long-running listener with auto-restart, reconnect without replaying old commands, and a heartbeat + health check to be proven by deliberately killing the service.
• Security hygiene: secrets outside the repo, never logged; phone commands are read-only by design.

Project Facts

My role	Designed and directed; to be built with Claude Code.
Status	Design complete as of 2026-09-20.
Budget	$0
Runtime	Planned to run 24/7 on a home server
AI usage	No AI/LLM calls inside the pipeline

Planned Tech Stack

• Python (standard library first): ingest, parsing, QC, analysis
• SQLite: schema, aggregations, rollups, scorecard queries
• Linux (Ubuntu Server): systemd scheduled services, backups, monitoring
• ntfy push notifications
• Git/GitHub

Build Roadmap

Phase	Scope
0	Measurement (Step 0): answer every unknown about external services before any pipeline code is written
1	Ingest, QC, storage, forecast logging
2	Morning brief
3	On-demand phone commands – V1 milestone
4	Anomalies & records
5	Severe weather
6	Home (local) brief
7	Forecast scorecard & station QC report
8	Weekly digest

Source code is not yet published.