Situation walkthrough: an incident, minute by minute

Incident communication is where every rule in this book runs at combat speed. This chapter replays a realistic incident channel from first alert to postmortem, annotated. The scenario is the one from Chapter 20: a payment provider blip that uncapped retries amplify into a checkout brownout.

The cast: Ravi, on call. Alex and Sam, engineers. Dana, engineering manager. Marta, support lead.

Don't be confused: mature incident response separates three roles that small teams collapse into one. The first responder (usually the on-call) acknowledges and starts investigating. The incident commander (IC, awkwardly colliding with "individual contributor") owns coordination and communication, and deliberately does not debug. Subject matter experts debug. The commander role exists because the person elbow-deep in logs is the worst-placed person to also run updates, and the transcript below shows the handoff working.

14:02 to 14:20: detection and mobilization

14:02  [bot] ALERT: checkout 5xx rate 4.1% (threshold 1%), eu-west
14:04  Ravi: Ack, looking. Thread here please.
14:09  Ravi: Real, not a blip: 5xx climbing, 6% now, eu-west only.
       Checkout p99 also degraded, 400ms -> 2.1s. Opening INC-214,
       severity 2 for now. I need a second pair of eyes on the
       payment path; Alex, you're up if available.
14:10  Alex: Here. Taking the payment provider angle.
14:12  Dana: I'll run comms as of now so Ravi can debug. Updates from
       me every 15 min in this channel. Current status:
       investigating, impact ~6% of eu-west checkouts, no data loss
       suspected. Next update 14:27.
14:16  Alex: Provider status page shows elevated errors on their side
       since 13:58. But our 5xx rate is 6x their reported error rate,
       so something of ours is amplifying. Suspect the retries;
       checking.
14:19  Marta: Support here: 11 tickets so far, all eu-west, all
       checkout. Using the wording "we're aware and investigating";
       shout if you want different.

Annotations:

"Ack, looking. Thread here please" does three jobs in six words: claims ownership, starts the record, and routes chatter out of the main channel. Nothing kills incident comms like four side conversations.
Ravi's 14:09 is the description formula under fire: rate with baseline, percentile with baseline, scope, and an honest severity guess "for now". No adjectives, no panic.
Dana's 14:12 is the commander handoff in one message: who runs comms, the cadence promise, the current state word (investigating), and the next-update time. From this point the debuggers never break focus to answer "any update?", because the answer is always "at :27, from Dana".
Alex's 14:16 models suspicion discipline: a fact ("their page shows errors"), the reasoning gap ("6x their rate"), and a hypothesis explicitly labeled ("suspect the retries; checking"), not asserted.
Marta's message is the support ally pattern in reverse: she brings data (ticket count as an impact signal) and syncs the external wording instead of freelancing it.

14:27 to 14:45: diagnosis and the call

14:27  Dana: Status: cause identified with high confidence. The 14:00
       deploy raised the retry ceiling; combined with provider
       errors, retries are amplifying load ~6x. Two options on the
       table, decision in this thread in <10 min. Impact steady at
       6-8%. Next update 14:42.
14:31  Ravi: Options: (A) roll back the 14:00 deploy, ~8 min, high
       confidence, also reverts two unrelated fixes; (B) hotfix just
       the retry config, ~20 min incl. review, surgical but slower.
       I lean A: reversibility beats elegance mid-incident.
14:33  Sam: +1 A. The unrelated fixes are cosmetic; nothing depends
       on them today. Confirmed by checking the deploy manifest.
14:34  Dana: Decision: A. Ravi executes, Alex watches provider-side
       recovery so we don't misattribute. Go.
14:42  Dana: Status: rollback deployed 14:39. 5xx rate falling, 3.1%
       and dropping. Calling it mitigated, not resolved: the retry
       config that allowed this is still in the codebase, and the
       provider is still degraded. Next update 14:57.

Annotations:

14:27 upgrades the state word deliberately: identified, with a confidence marker, and pre-announces a decision with a deadline. Compare the anti-pattern: silence until someone asks.
Ravi's option framing is the escalation decision-request shape, compressed: two options, costs, a lean, and a principle ("reversibility beats elegance mid-incident") that makes the lean auditable.
Sam's "+1 A" carries weight because it arrives with its evidence ("confirmed by checking the manifest"), not as a vibe.
Dana's decision message assigns the watching job too. "Alex watches provider-side recovery so we don't misattribute" prevents the classic post-rollback error: declaring victory over a graph that was recovering anyway.
14:42 refuses the premature all-clear: mitigated, not resolved, with the reason spelled out. This one distinction (Chapter 10) is the most-audited sentence in any incident record.

15:10: standing down

15:10  Dana: Status: resolved. 5xx back to 0.1% baseline for 20+
       minutes, provider recovered, retry cap PR is up for tomorrow
       (#4821). Standing down the incident. Timeline notes are in the
       doc while memories are fresh: go/inc-214. Postmortem Thursday
       10:00, blameless as always; if you touched this incident,
       please add your timestamps to the doc today. Thanks all;
       textbook response, and Marta's ticket feed was our best
       impact gauge for the first ten minutes.

The stand-down message closes every loop it opened: the metric that justifies resolved, the follow-up work already tracked, the postmortem scheduled with its ground rule restated, the same-day timeline ask (Chapter 17), and specific credit (Chapter 2) to the least glamorous contribution in the room.

Thursday: the postmortem, two extracts

Dana:  Ground rule reminder: we're debugging the system, not the
       people. Everything anyone did made sense at the time with the
       information they had, or we have a different problem to fix.

Alex:  Contributing factor two: the retry ceiling change shipped
       without a load test because our load tests don't model
       provider degradation at all. I wrote the change, and given
       what the tooling showed me, I'd write it again. That's the
       point: the tooling has to show the next person more.

Sam:   Proposed action items: (1) retry cap with a config maximum,
       owner Alex, this week, already in review. (2) Add provider-
       degradation profile to load tests, owner me, two weeks.
       (3) The runbook said "check the queue depth" but not where;
       one-line fix, owner Ravi, today. Filing all three now.

Annotations: Dana's ground rule is the blameless frame said out loud, not assumed. Alex demonstrates the partner move: plain first-person facts ("I wrote the change") wrapped in system analysis, neither hiding nor flagellating. And Sam's action items each carry an owner and a date and become tickets in the meeting, which is the entire difference between a postmortem and a support group.

The external note

The customer-facing translation of everything above is four sentences, and writing it is Chapter 20's discipline:

Resolved: Between 14:02 and 14:39 UTC, some customers in Europe saw
errors or slow loading during checkout. Completed orders were not
affected. The issue was resolved by reverting a configuration change,
and we are adding safeguards against this class of problem. We're
sorry for the disruption.

No INC numbers, no provider blame, no percentages that invite arithmetic, one plain apology, and only claims an engineer would sign.

👉 One walkthrough remains: the conversations everyone dreads. Slipping a date to your manager's face, reclaiming credit that walked away, being put on the spot by an executive, taking a harsh review, and saying no to a colleague, each played out with the words that work. On to Chapter 23.