work / case
Production Kubernetes Platform, Built From Scratch
Designed and operates the 7-node, self-managed Kubernetes platform that runs the company's AI, development, and production workloads end to end.
Problem
The organisation’s AI, development, and production workloads needed a real platform to run on — not a hand-managed pile of VMs. Managed Kubernetes wasn’t a fit for the GPU-heavy, on-premise compute involved, so the cluster had to be built and operated from the ground up: networking, storage, ingress, certificates, backup, and observability all owned in-house, with no platform team to lean on.
Approach
Stood up a production cluster from scratch with kubeadm and the Calico CNI — 7 nodes on a 10GbE fabric, partitioned into namespaces for AI, development, production, experimental, and analytics workloads. Built out the full platform layer rather than just a bare control plane:
- Storage — Rook-Ceph providing replicated block and shared filesystem storage to stateful workloads, with snapshot and disaster-recovery workflows.
- GPU scheduling — the GPU operator to expose accelerators to the AI namespaces as first-class schedulable resources.
- Edge & networking — Traefik for ingress, MetalLB for bare-metal load balancing, and cert-manager driving automated TLS via ACME DNS-01.
- Backup & DR — Velero with Restic for cluster and persistent-volume backup/restore, targeting a dedicated TrueNAS system: 180 TB raw across two RAID-Z2 (ZFS) arrays, ~140 TB usable. It holds the automated cluster backups, the model library, the agentic platform’s per-user workspace snapshots, and the other server-component backups.
- Observability — Prometheus in agent mode fronted by Thanos for long-horizon, low-overhead metrics, Grafana for dashboards, Loki for logs, and Alertmanager plus uptime monitoring for service health.
- Delivery — GitLab CI/CD builds, tests, and security-scans each change and publishes to a private registry; ArgoCD then syncs every workload onto the clusters as GitOps, so the running state always matches what is in git — reproducible, auditable, and self-correcting.
The same patterns run across substrates: self-managed kubeadm clusters on-premise and Amazon EKS on AWS, so workloads move between a self-managed and a managed control plane from one delivery flow.
Outcome
A single self-managed platform now carries the company’s agentic AI service, internal tooling, and production workloads on shared, observable infrastructure — replacing ad-hoc per-VM deployments with a reproducible, backed-up, monitored system. The full stack (compute, storage, ingress, TLS, backup, and metrics/logs) is operated by one engineer, and the same kubeadm + Calico patterns were carried across to additional clusters.