Cases · Data lead · Enterprise B2B · anonymized
From a 30-year private-cloud monolith to a reusable cloud data platform
A horizontal, decades-old data platform with manual runs and no quality gates. Redesigned as cloud-native data products — so multi-day batch cycles finished in hours, and the same frameworks fed the website, analytics, and ML.
The problem
- The data platform was a horizontal monolith on a private cloud — layered for roughly 30 years.
- Processing was mostly manual. No automation. No QA checks in the path.
- Everything lived in one tangled flow, so teams could not own a slice of the data or ship changes safely.
- Downstream consumers — the main website, analytical dashboards, and ML/AI workflows — depended on that fragile path.
What I did
- Redesigned the end-to-end data process to run on cloud, with a clear path from ingestion to publish.
- Split the estate into logical data products so each product could own its own pipeline and lifecycle.
- Remodelled the warehouse around fact and dimension design so analytics and products shared one trustworthy shape.
- Built a metadata-driven platform: one driver module wired to reusable ingestion, quality, standardization, and transformation frameworks.
- Automated QA checks in the path — not after someone noticed a broken dashboard.
- Ran processing with PySpark and Pandas, orchestrated through Dagster.
- Migrated data from the private cloud to Azure with validation and cutover discipline.
- Stood up Dagster, MLflow, and the processing stack on Kubernetes — platform and pipelines, together.
- Partnered with AI engineers so model workflows received correct, governed inputs — not ad-hoc extracts.
What changed
- Data processing that previously took multi-day batch cycles (often spanning a week or more end-to-end) completed in hours after the rebuild.
- Reusable frameworks replaced one-off scripts — new pipelines plugged into the same driver.
- Website, analytical dashboards, and ML/AI workflows all consumed the same transformed products.
- Quality gates and data-product boundaries made ownership and change safer across teams.
Stack
Need a similar rebuild?
Book a free 30-minute teardown. Bring the current stack. Leave with a clearer next step.