On-call for agents: 3 a.m. and something is wrong

Everything in this book has been preparation for the moment it goes sideways in production with a human asleep. This closing chapter of Part 9 is about that moment: what pages, what the runbook says, the two switches that must always work, and the postmortem culture that turns an incident into a fix instead of a shrug. It is where the whole production bar is cashed, because an operable platform is finally defined by one property, that the person paged at 3 a.m. can actually resolve it, and everything else in the book existed to make that true.

What pages, and what does not

Agent platforms have their own failure signatures, distinct from ordinary services, and the on-call design starts by naming them. Ordinary alerts (host down, error rate up, latency spiked) still apply, but four agent-specific ones matter more, and each maps to a metric the earlier chapters made you emit:

Read that table backwards and it validates the whole book: every page is something an earlier chapter built the instrumentation to catch, which is what "operable" means in practice. A platform that cannot page on a verification collapse never built the decoy monitor; one that cannot page on a budget blowout never made the ledger a metric stream. The alerts you can define are exactly the failures you designed to see.

And the discipline of what does not page matters as much: a single unit dead-lettering is normal (failure semantics made it boring), a run exhausting its budget is a designed landing not an incident, and paging on either trains the on-call to ignore the pager. Page on rates and trends (dead-letter rate climbing, budget burn accelerating), not on individual expected events.

The two switches

When a page is real and getting worse, the operator needs two controls that always work, and "always work" is a hard requirement: they must be enforced by the platform, reachable even when the fleet is melting, and independent of the agents' own cooperation.

• The kill switch stops the fleet now. It halts dispatch, cancels in-flight units, and brings spend to zero, accepting that in-flight work is lost (the ledger means it is not forgotten, just stopped). This is for "we are actively harming something": exfiltrating, spending catastrophically, acting on a compromised input.
• The drain switch stops intake while letting in-flight work finish. New work is refused; running units complete and settle; the fleet empties gracefully. This is for the far more common "we need to stop starting things": a bad deploy to roll back, a dependency degrading, a maintenance window.

Most incidents want drain, not kill, which is why both exist: reaching for the kill switch when drain would do throws away good work and settled state for no reason. The kill switch is the fire alarm; the drain switch is the closing-time announcement, and an on-call who knows which is which under pressure is an on-call the runbook prepared.

Runbooks that resolve, not describe

A runbook is not documentation of how the system works; it is a decision tree for a specific alert, written so a tired human executes it correctly. Each of the pages above gets one, and they share a shape:

1. Confirm the signal is real (the metric, the trace, the ledger, from Chapter 40, not a hunch).
2. Contain with the right switch: drain to stop the bleeding without losing work, kill only if active harm is underway.
3. Diagnose from evidence: pull the offending runs' traces and ledgers, and for the strange ones, replay them.
4. Resolve: roll back the change (the pinned model or config is one edit away), raise the quota, top up the budget, fix the poison input.
5. Recover: undrain, and let checkpoint-and-resume (Chapter 25) restart only the incomplete work, not the whole night.

The runbook's value is that step 3 is possible at all. On most platforms, "the agent did something weird" bottoms out in guesswork; on this one it bottoms out in a replayable trace, which is the difference between an incident that ends in a fix and one that ends in a shrug.

The banned root cause

Which brings us to the cultural rule that makes the whole discipline stick: "the model behaved unexpectedly" is not an acceptable root cause. It is a description of every agent incident and an explanation of none, and a postmortem that stops there guarantees the incident recurs, because nothing was actually found.

The book spent Part 1 building the tools to do better. Every incident has a replayable trace: the recorded run, re-executable deterministically, with the ability to inject a changed input and watch the divergence. So the postmortem question is never "why did the model get weird" but "what did the model see, and what would it have done differently if it had seen something else", and that question has a mechanical answer. The fault-injection lab was building this exact muscle at toy scale: change one tool result, watch the run take a different, explicable path. A mature agent postmortem is that lab pointed at a production incident, and its output is a concrete fix (a tightened tool description, a scoped credential, a guardrail at the injection point, a poison input added to the golden set), not a sentence about the model's mood.

Part 9, and the platform, made operable

Part 9 in one arc: observability made the run readable as a trace; evals made behavior regression-testable; cost engineering made spend a gated metric; and on-call made the 3 a.m. failure resolvable from evidence. Together with the defensibility of Part 8, the swarm machinery of Part 6, and the foundations beneath them, the platform now passes all twelve gates of the production bar, which was the entire promise of the book: not a demo that impresses, but a system a team can be on call for.

Don't be confused: reliability incidents vs safety incidents. They page the same person and feel alike at 3 a.m., but they are different failures needing different postmortems. A reliability incident (stuck fleet, quota storm, budget blowout) assumes everyone was honest and something broke; the fix is engineering (raise a limit, fix a bug, add a reserve). A safety incident (an injection succeeded, an agent exceeded its scope, data leaked) assumes an adversary and the fix is containment (Part 8: tighten a scope, cut a leg of the trifecta, add a gate). Triaging them the same way is how a security breach gets closed as "flaky fleet, added a retry" while the attacker is still inside. Ask early which one you are in.

👉 Next, per the map: Part 10 reads the protocols and frameworks (Strands, MCP, A2A) as a design review of everything built by hand, and Part 11's capstones exercise the whole platform on four real jobs.