work / case
An Automated 24/7 Incident-Response Pipeline
Built an in-house automation that detects severity-1 incidents, finds whoever is on call, and places escalation phone calls 24/7 — replacing a paid outsourced answering service.
Problem
Severity-1 production incidents at the organisation depended on a third-party on-call answering service: monitoring fired, a human at an outside desk read the alert, looked up who to call, and relayed it onward. That handoff cost ~$3–5k/month and added a layer of latency and translation between the systems that knew something was wrong and the engineer who could act. It also put the most time-critical path in the stack — getting the right person on the phone at 3am — outside our own control and impossible to tune.
Approach
Built an in-house automation that closes the loop from alert to phone call without a human relay in the middle, running 24/7. It reuses the platform’s existing observability rather than bolting on a parallel monitoring stack:
- Detection — sev-1 conditions are picked up from the existing Prometheus / Alertmanager / Loki layer already running on the Kubernetes platform, so the signals driving escalation are the same ones the platform is judged on.
- Roster lookup — on a qualifying incident the automation checks the on-call roster to resolve who is actually on duty at that moment, across the day and the night.
- Escalation — it then places programmatic voice calls to that person, a PagerDuty-style call-out flow owned end to end, so a sev-1 reaches a real engineer’s phone directly instead of routing through an outside answering desk.
Because the whole path is in-house, escalation rules and timing are ours to change rather than something negotiated with a vendor.
Outcome
The outsourced answering and escalation service was retired, saving ~$3–5k/month — a clear build-versus-buy win — while escalation got faster and fully owned: detection, roster, and call-out now live in one system we operate and tune. It also set the operational pattern for how I work today as the primary escalation point for complex production incidents across cloud and Kubernetes. When the pipeline itself breaks, I own it and engineer the fix; reliability here isn’t outsourced, it’s accountable.