— Case study 12
A live instrument panel for renewable electricity, built only on real grid data
project overview
A live dashboard of how much electricity already comes from renewables, starting with Brazil's hydro-heavy grid and comparing it with Europe and the USA, fed by a failure-resistant streaming pipeline over real public data.
shipped outcome
Live, with Brazil, Europe and USA deep-dives, a plant registry map, country comparison and a pipeline-health panel showing the dead-letter queue and the last successful poll per source. The ONS spine ingested 366,336 real plant/hour readings with zero duplicates on replay.
Grid operators publish generation data, but in different formats, cadences and units: Brazil's ONS by plant and hour, ENTSO-E in 15 to 60-minute series for European zones, the EIA hourly for the USA. Turning that into one comparable, trustworthy picture means a pipeline that survives bursty batches, malformed rows and replays, without ever filling a gap with invented numbers.
A Go ingestion edge polls each source and normalizes it to one canonical event schema; Redpanda carries the events to idempotent TypeScript consumers that write to TimescaleDB, keyed on source, zone, asset, metric and timestamp, with a dead-letter queue and bounded backpressure. A separate live consumer fans readings out over WebSocket to the Next.js dashboard. The README states plainly that the sources are polled, not pushed: the real-time character comes from the pipeline, not the data.
Live, with Brazil, Europe and USA deep-dives, a plant registry map, country comparison and a pipeline-health panel showing the dead-letter queue and the last successful poll per source. The ONS spine ingested 366,336 real plant/hour readings with zero duplicates on replay.
— Interface evidence
These are the verified interface captures from the project build. Scroll inside taller frames to inspect each complete page.
— Delivery process
The work is sequenced around risk. Domain rules and failure states come before interface polish; automation arrives before handoff.
Go poller, Redpanda, a persist consumer and TimescaleDB, live-verified end to end on real Brazilian plant data.
Idempotent upserts, a dead-letter queue for malformed events and bounded backpressure, each verified, not assumed.
ENTSO-E pollers across six zones and the EIA for the US48 and seven regional grid operators, normalized to the same schema.
Deep-dives per region, a plant map, volatility charts and a pipeline-transparency panel, with data prefetched on the server for real edge caching.
— System architecture
Each layer has one job and a narrow contract. That keeps external services replaceable and product behavior testable.
ons / plant-hour
366,336 readings
zero duplicates on replay
idempotency key
source · zone · asset · metric · time
sources / 3
One canonical event schema, hand-mirrored from the Zod contracts the TypeScript side uses.
Partitioned topics with independent consumer groups for persistence and live fan-out.
Idempotent writes and an hourly continuous aggregate for the charts.
The dashboard reads REST through TanStack Query and live readings over a socket.
— What shipped
Every reading traces back to an ONS, ENTSO-E or EIA response; a missing source shows as missing, never as a plausible number.
Writes are keyed on source, zone, asset, metric and timestamp, so re-ingesting a batch changes nothing.
Dead-letter depth, consumer lag and the last successful poll per source are on the dashboard, not hidden in logs.
— Engineering pressure
None of the sources stream; ONS republishes its plant file twice a day. The product says so, and the streaming standard is in the pipeline's engineering instead.
The full-depth ONS backfill outgrew the database volume during live operation; the volume was resized and the incident recorded with recommendations for the next backfill.
ONS switched to per-month files in 2022, so the backfill broke on earlier years until the year-file case was handled and the pilot chunk ran clean.
verified project destinations
next transmission
Bring the operating problem. We will shape it into a clear product brief and a buildable next step.
Start a project