AI-driven reliability engineering for an overnight data platform

Problem

A nightly pipeline serving hundreds of automated client reports failed in ways nobody could see until morning: silent transient errors, jobs hanging for hours until a cost-saving capacity pause killed them, hidden resource contention.

What I built

Used AI agents to audit months of run history through the platform’s APIs, producing a defensible failure-mode taxonomy (transient nightly errors with no retry configured, multi-hour hangs terminated by the scheduled capacity pause, throttling from over-parallelised job graphs) and a costed fix plan.

Then implemented the substrate for ongoing self-observation: a structured event log every pipeline stage writes to, a morning digest that turns the night’s events into a plain-English status email with held work flagged red rather than silently skipped, and a living troubleshooting knowledge base that grows with each incident.

Outcome

The platform went from “we find out when a client complains” to a daily, evidence-based digest. The audit methodology (prove the API contract, then build on it) is reusable at any client.