Cases · Senior data engineer · Renewable energy · power trading · Europe · anonymized
From 5-minute micro-ETL to Kafka — realtime data for European power traders
IoT feeds from solar and wind were stuck in a micro-ETL loop every five minutes. Rebuilt the platform around Kafka on Kubernetes, streamed clean data into the warehouse, and helped build dashboards for day-ahead and futures traders — 200+ pipelines live.
The problem
- Data arrived from renewable IoT devices — solar farms, windmills, and related sources — but landed through a micro-ETL architecture on a ~5-minute cycle.
- That delay was too slow for power trading. Day traders and futures traders needed fresher signals to act in the European market.
- The platform beneath the pipelines was not built for streaming volume or low-latency paths into the warehouse and dashboards.
- Batch-shaped processing limited how many reliable pipelines the team could run in production.
What I did
- Replaced the micro-ETL path with a Kafka-centred streaming architecture.
- Stood up Kafka on Kubernetes and hardened the platform so ingestion, processing, and delivery could scale with IoT volume.
- Streamed device and market-relevant events from Kafka into the Azure data warehouse with Python processing and dbt models on a clear publish path.
- Built and operated 200+ production pipelines on that platform — not one-off jobs.
- Helped design and build the dashboards day traders and futures traders actually used — so decisions sat on near-realtime renewable and market data.
- Kept the European client context operational — latency and reliability mattered more than prettier batch reports.
What changed
- Moved from ~5-minute micro-ETL cycles to a Kafka-based realtime path.
- 200+ pipelines live on the new platform.
- Day-trader and futures-trader dashboards I helped build ran on fresher data from solar, wind, and related IoT sources.
- During the engagement, the client reported roughly 5× revenue growth as trading moved onto fresher data — attribution was not isolated in a controlled study.
Stack
Need a similar rebuild?
Book a free 30-minute teardown. Bring the current stack. Leave with a clearer next step.