work / case
The Company's First BI / Data Warehouse
Built the company's first analytics warehouse from nothing — a ClickHouse pipeline that compressed ~600 GB of transactional data to ~60 GB and gave the business its first cross-product reporting.
Problem
The business ran on transactional databases but had no analytics layer — no warehouse, no cross-product reporting, no way to ask a question that spanned more than one system. Raw transactional data sat in the hundreds of gigabytes per product, in schemas designed for live operations rather than for analysis, and there was no prior art to build on. Querying production directly was slow, risky, and gave the business no consolidated view across a portfolio of separate products.
Approach
I designed and built the warehouse end to end. Because no documentation of the analytical shape of the data existed, I read through the product code and cloned live databases to reverse-engineer what was worth extracting and how it should be modeled, then wrote Python ETL to extract, transform, and load it into ClickHouse — chosen for its columnar storage and compression on append-mostly transactional data.
The first pipeline targeted a flagship transaction-processing product: roughly 600 GB of source data, reshaped and loaded into about 60–70 GB in ClickHouse — close to a 10× reduction through column types, ordering, and codec choices. I then generalised the loader into a reusable, schema-driven ETL that onboarded further products and external marketing data without bespoke code per source, and used Airbyte for parts of the flow such as customer-feedback and web-analytics extraction.
To host it, I stood up a Kubernetes cluster from scratch in the datacentre (one control plane, three workers) running the ETL jobs, ClickHouse, and Apache Superset for dashboards — giving non-engineers self-serve access to the data.
Outcome
The warehouse went from zero to covering eight production products at around 120 GB total — the company’s first consolidated analytics platform. The flagship dataset compressed roughly 10× (~600 GB → ~60 GB), and storage and query patterns were tuned so dashboards stayed responsive as the dataset grew. Superset turned what had been inaccessible transactional data into reporting the business could actually use, and the schema-driven ETL made adding the next product a configuration change rather than a new project.