Incident management, blameless postmortems, and per-tenant SLAs

What it is

Incident management is the set of roles, decisions and rituals for handling a production failure while it is happening, and learning from it afterwards. It is a discipline because the instinct under pressure (everyone debugging at once, the loudest voice deciding, fixing the root cause before stopping the bleeding) is wrong, and the discipline corrects it.

The three parts:

DURING:   roles, severity, and the rule "mitigate before you diagnose"
AFTER:    the blameless postmortem and the repeat-incident rate
CONTRACT: SLAs, which turn availability into an obligation, and the
          observability that per-tenant SLAs require

What this is confused with: incident management as debugging. Debugging is finding the cause; incident management is restoring service, which is often a different and faster action. The single most important rule is that those are separate, and mitigation comes first: a rollback that restores service in two minutes beats a root-cause fix that takes an hour, and the root cause is a postmortem's job, not the incident's.

The problem it solves

Under pressure, without structure, an incident goes badly in predictable ways:

- everyone debugs the same thing; nobody communicates with stakeholders
- the person who shouts loudest decides, regardless of context
- someone tries to fix the root cause while the site is down, when a
  rollback would have restored it in two minutes
- no one is tracking what has been tried, so things are retried
- the postmortem blames a person, so the next person hides the next mistake

The measurable cost:

Without structure:      mean time to RESTORE dominated by coordination,
                        not by the actual fix
                        repeat incidents high, because postmortems produce
                        blame rather than action items
With structure:         mitigation decoupled from diagnosis; restoration
                        time driven by the mitigation, not the fix

Per-tenant SLAs add a second problem: an aggregate 99.9 percent availability can hide that your largest customer had 98 percent, and if their contract says 99.9, that is a breach nobody saw. A per-tenant SLA is unenforceable without per-tenant observability, which is the constraint the contract creates.

Mechanics

Roles

INCIDENT COMMANDER (IC):  decides. Does NOT debug. Coordinates, delegates,
                          and makes the call (roll back, escalate, page).
                          The one person with authority.
COMMUNICATIONS LEAD:      updates stakeholders and the status page, so the
                          IC and responders are not interrupted.
OPERATIONS / RESPONDERS:  the ones actually investigating and acting.
SCRIBE:                   records the timeline as it happens (for the
                          postmortem, and so nothing is retried).

The IC not debugging is the counterintuitive part and the load-bearing one. The instinct is that the most senior engineer should be hands-on, and the discipline is that someone must be holding the whole picture, deciding, and communicating, and that person cannot also be head-down in a terminal. On a small incident one person wears several hats; the roles still exist.

Severity, and why it has explicit triggers

SEV1:  customer-facing outage, revenue-impacting, or data loss.
       Page immediately, IC assigned, all-hands.
SEV2:  significant degradation, a subset of users, a workaround exists.
SEV3:  minor, internal, or single-customer with low impact.

Severity must have explicit, objective triggers, or every incident is argued into a lower severity to avoid the process:

SEV1 triggers (examples, objective):
  - error rate > 5% for > 5 min on a customer-facing service
  - any data-loss or data-corruption event
  - a Tier-1 customer fully down
  - checkout/payment path unavailable

Objective triggers remove the negotiation. Without them, the person who would have to run the SEV1 process has an incentive to call it a SEV2, and the incident is under-resourced. The trigger fires the severity, not a judgement call under pressure.

Mitigate before you diagnose

The order, non-negotiable:
  1. STOP THE BLEEDING: rollback, failover, feature-flag off, scale up,
     shed load. Restore service by ANY means.
  2. THEN diagnose: find the root cause, in a postmortem, at leisure.

Example: a deploy causes a 30% error rate.
  wrong: "let me find what in the deploy broke it" (60 min, site down)
  right: "roll back" (2 min, site up), then investigate the deploy in
         the postmortem

The mitigation you reach for should not require understanding the cause. A rollback works whether or not you know what broke; a targeted fix requires diagnosis first. This is why "can we roll back" is the IC's first question, and why deploys must be rollback-able (see expand-contract): the fastest mitigation is undoing the change, and it only works if the change is reversible.

The blameless postmortem

The premise: incidents are caused by SYSTEMS that allowed a mistake, not
by the person who made it. If a single human error can take down
production, the SYSTEM is the problem.

A blameless postmortem asks:
  - what happened (timeline, from the scribe's notes)
  - what was the impact (users, duration, revenue)
  - why did it happen (contributing factors, not "who")
  - why did our systems ALLOW it (the real question)
  - what will we change (action items with owners and dates)

"Blameless" is not "consequence-free"; it is "the mistake is a system signal." If an engineer ran a destructive command against production, the postmortem asks why production accepted a destructive command from one person with no confirmation, not why the engineer was careless. The blame version teaches people to hide mistakes; the blameless version teaches the organisation to remove the sharp edges.

The metric that matters is the repeat-incident rate:

repeat-incident rate = incidents with the same root cause as a prior one
                       / total incidents

A high repeat rate means postmortems are producing blame rather than fixes, because a fixed root cause does not recur. It is the single number that tells you whether the postmortem process works, and it is more honest than "we did a postmortem," which measures activity rather than outcome.

SLA, SLO, SLI, and the per-tenant version

SLI (Indicator):   a measurement. "the fraction of requests served in
                   under 300 ms."
SLO (Objective):   an internal target. "99.9% of requests under 300 ms."
SLA (Agreement):   a CONTRACT with a customer, with PENALTIES. "99.9%
                   uptime, or a service credit."

The SLO is stricter than the SLA, deliberately: you promise the customer 99.9 and target 99.95 internally, so you have margin to detect and fix a problem before you breach the contract. See SLA vs SLO vs SLI.

Per-tenant SLAs create an observability requirement that aggregate SLOs do not:

Aggregate availability:  99.92%   (looks fine against a 99.9% SLA)
But per-tenant:
  Tenant A (enterprise, 99.9% SLA):  98.4%   <- BREACH, invisible in the aggregate
  Tenant B:                          99.99%
  ...

A large customer's downtime is a rounding error in the aggregate and a contract breach in their view. So a per-tenant SLA requires per-tenant SLIs: availability and latency computed per tenant, which is a cardinality problem (a metric per tenant) that pushes toward the wide- event model (see observability pipeline).

The per-tenant SLA -> per-tenant observability chain:
  contract promises per-tenant availability
  -> you must MEASURE per-tenant availability
  -> which is a high-cardinality slice (tenant_id)
  -> which a time-series database holds badly
  -> which is an argument for an event store

The contract you signed determines the observability you need, and teams sign per-tenant SLAs without building per-tenant measurement, so the first they learn of a breach is the customer's email.

A worked example: an incident that took 4 hours and should have taken 5 minutes

A SaaS platform. A deploy to the auth service caused a 40 percent login failure rate.

What happened, without discipline:

14:02  deploy completes; login errors start
14:05  three engineers independently start debugging the auth service
14:10  a manager joins and asks for status; the engineers are interrupted
14:15  someone suggests it might be the database; two engineers pivot there
14:30  the database is fine; back to the auth service
14:40  a VP asks in the channel "is this the deploy?"; nobody had checked
15:10  someone finds the deploy changed a token-validation library
15:40  they attempt a forward fix to the library config
16:20  the forward fix has a bug; more errors
17:50  finally roll back the deploy; service restored

Three hours and forty-eight minutes, and the rollback that fixed it took two minutes. The site was down for the entire time it took to diagnose, when a rollback would have restored it at 14:05.

The failures, each a discipline gap:

- no IC: three people debugging the same thing, nobody coordinating
- no comms lead: engineers interrupted by status requests
- diagnosis before mitigation: 90 minutes spent finding the cause with
  the site down, when rollback needed no diagnosis
- a forward fix attempted under pressure, which introduced a new bug
- no scribe: the database was investigated twice

The changes:

1. Roles, assigned at declaration.

On SEV1 declaration:
  - the on-call IC takes command and does NOT debug
  - a comms lead is assigned; all stakeholder updates go through them
  - responders investigate; a scribe records the timeline

2. Mitigate-first as the IC's first question.

The IC's script:
  1. "Can we roll back?" -> if yes, roll back NOW, diagnose after
  2. "Can we feature-flag it off?"
  3. "Can we fail over?"
  4. only if none: diagnose to find a mitigation

3. Objective severity triggers, so the incident is resourced correctly from the start.

The same incident, replayed in a game day:

14:02  deploy completes; login errors start
14:03  error-rate alert fires; on-call paged
14:04  on-call declares SEV1 (>5% error rate trigger), takes IC role
14:04  IC's first question: "can we roll back?" -> yes
14:06  rollback initiated
14:07  service restored. Login errors stop.
14:07  IC hands off to a postmortem; the deploy is investigated at leisure
total time to restore: ~5 minutes

Five minutes instead of nearly four hours, and the difference was entirely process: the IC asked "can we roll back" before anyone diagnosed anything.

4. The postmortem, blameless.

The blame version would have asked: "why did the engineer ship a broken
token-validation change?"

The blameless version asked:
  - why did the change reach production without a canary catching the
    login-failure spike? (the deploy skipped the canary stage for auth)
  - why did it take 90 minutes to consider rollback? (no IC, no
    mitigate-first culture)
  - why was the database investigated twice? (no scribe)

Action items:
  - auth deploys must go through canary (owner, date)
  - IC training and the mitigate-first script (owner, date)
  - a runbook: "SEV1 first questions" (owner, date)

The action items address the system, not the person. The engineer's change was a contributing factor; the absence of a canary on auth deploys and the absence of a mitigate-first culture were the causes.

5. Per-tenant SLAs, driven by a contract.

The platform then signed an enterprise customer with a 99.9 percent per-tenant SLA.

existing observability:  aggregate availability only
the gap:                 a breach for that one tenant would be invisible
                         in the aggregate

added: per-tenant SLIs (availability and p99 latency by tenant_id),
       computed from wide events rather than Prometheus (the tenant_id
       cardinality broke the TSDB), with a per-tenant error budget and
       an alert when any tenant's budget burned fast.

first month: the aggregate was 99.94% and one tenant was at 99.87%,
             which was within their SLA but on track to breach. The
             per-tenant alert caught it; the aggregate never would have.

Final:

                              before      after
mean time to restore (SEV1)   hours       minutes (mitigate-first)
incidents with an IC          rarely      every SEV1
diagnosis-before-mitigation   default     eliminated
repeat-incident rate          ~30%        ~8%     (blameless action items)
per-tenant SLA visibility     none        per-tenant SLIs + budget alerts

The mitigate-first rule was the single largest change, turning a four-hour outage into a five-minute one with no new technology, only the discipline of restoring service before understanding it.

Production evidence

Google's SRE book and the PagerDuty and Atlassian incident-management guides all codify the same structure: an incident commander who coordinates rather than debugs, explicit severity levels, and mitigation before root-cause analysis. The convergence across independent sources is the evidence that the structure is not one team's preference.

The Incident Command System that these borrow from is the framework used by emergency services for decades, and the adaptation to software incidents is deliberate: the separation of command from execution is the part that transfers.

"Mitigate before diagnose" is explicit in Google's SRE practice ("stop the bleeding") and in every mature incident process. The rollback-first instinct is the operational expression of it, and it is why progressive delivery emphasises reversibility.

Blameless postmortems were popularised by John Allspaw at Etsy, and the argument (that blame drives mistakes underground and systems, not people, are the correct unit of analysis) is the foundation of the practice. The repeat-incident rate as the effectiveness metric is the practical test of whether it works.

Per-tenant SLAs and the observability they require are a standard challenge in multi-tenant SaaS, and the cardinality problem (per-tenant metrics) is a recurring driver toward event-based observability, which is the connection the observability pipeline page makes from the other direction.

The debate

Should the most senior engineer be the IC? No, and this is the counterintuitive position. The IC coordinates and decides and does not debug, so the most senior engineer is often more valuable as a responder. The IC role is about holding the whole picture and communicating, and those are incompatible with being head-down in a terminal. On a small incident one person wears both hats; the point is that the coordination role exists and is filled.

Mitigate or diagnose first? Mitigate, always, and the mitigation should not require understanding the cause. A rollback restores service whether or not you know what broke; a targeted fix requires diagnosis, which happens with the site down. The exception is a mitigation that could make things worse (a failover to a replica that is also affected), which is why the IC's judgement is on which mitigation, not on whether to mitigate before diagnosing.

Are blameless postmortems soft? The opposite: they are the version that actually reduces incidents, because blame drives mistakes underground and a system that a single human error can break is the real defect. The test is the repeat-incident rate: a blame culture produces postmortems that recur because they fix a person, not a system. Blameless is the more rigorous practice, not the gentler one.

How strict should severity triggers be? Objective and specific, because subjective severity is negotiated downward to avoid the process. "Error rate above 5 percent for five minutes on a customer-facing service" fires without argument; "significant impact" is debated while the incident continues. The trigger removes the incentive to under-declare.

Do you need per-tenant SLAs? Only if you sign them, and if you sign them you need per-tenant observability, which is a cardinality commitment. The failure is signing the contract without building the measurement, so the first breach is the customer's email. If your SLA is aggregate, aggregate SLIs suffice; the moment it is per-tenant, the observability must follow the contract.

What is the highest-value single practice? Mitigate before diagnose, because it decouples restoration time from diagnosis time, and diagnosis is the slow part. In the worked example it turned a four-hour outage into five minutes, and it is a cultural change with no technology cost.

Follow-up Q&A

"What are the incident roles and why doesn't the IC debug?"

The incident commander coordinates and decides, a communications lead updates stakeholders, the responders investigate and act, and a scribe records the timeline. The IC does not debug because someone must hold the whole picture, make the calls (roll back, escalate, page) and communicate, and that is incompatible with being head-down in a terminal. The instinct that the most senior engineer should be hands-on is exactly what leaves nobody coordinating, which is how three people debug the same thing while the database gets investigated twice.

"Why mitigate before diagnose?"

Because restoring service and finding the cause are different actions, and the first is usually faster. A rollback restores service whether or not you know what broke; a targeted fix requires diagnosis, which happens with the site down. In one incident, diagnosis took ninety minutes with the site down and the rollback that fixed it took two minutes. So the IC's first question is "can we roll back," and the mitigation you reach for should not require understanding the cause.

"What makes a postmortem blameless, and why does it matter?"

It treats the incident as caused by a system that allowed a mistake, not by the person who made it: if a single human error can take down production, the system is the defect. It matters because blame drives mistakes underground, so the next person hides the next one, and because a fixed system does not recur while a blamed person does. The test is the repeat-incident rate: a high rate means postmortems are producing blame rather than fixes.

"Why do severity levels need objective triggers?"

Because subjective severity is negotiated downward to avoid the process. If SEV1 means "significant impact," the person who would have to run the SEV1 process argues it is a SEV2, and the incident is under-resourced. An objective trigger, "error rate above 5 percent for five minutes on a customer-facing service," fires the severity without a judgement call under pressure, and it resources the incident correctly from the start.

"Why do per-tenant SLAs create an observability requirement?"

Because an aggregate availability number hides individual tenants. An aggregate 99.92 percent can contain a single enterprise tenant at 98.4 percent, which is a breach of their 99.9 percent SLA that is invisible in the aggregate. So a per-tenant SLA requires per-tenant SLIs, which is a high-cardinality slice (a measurement per tenant) that a time-series database holds badly and that pushes toward an event store. The contract determines the observability, and signing the SLA without building the measurement means the first breach is the customer's email.

Common misconceptions

"The most senior engineer should run the incident." The IC coordinates and does not debug, so the senior engineer is often more valuable investigating. The coordination role is separate from the execution.

"Find the root cause first." Restoring service and finding the cause are different actions. Mitigate first, with a mitigation that does not require diagnosis, and find the cause in the postmortem.

"Blameless means no accountability." It means the mistake is a system signal. The action items hold the organisation accountable for removing the sharp edge, which is more rigorous than blaming a person.

"A good aggregate SLO means every customer is fine." An aggregate hides individual tenants. A per-tenant SLA requires per-tenant measurement, or a large customer's breach is a rounding error you never see.

"A postmortem means the process works." Doing a postmortem measures activity. The repeat-incident rate measures whether it produced fixes.

Interview delivery note

Say this verbatim: "The single most important rule is mitigate before diagnose, because restoring service and finding the cause are different actions and the first is faster. In one incident diagnosis took ninety minutes with the site down and the rollback that fixed it took two minutes, so the incident commander's first question is 'can we roll back.'" The rule and the concrete cost of violating it.

The senior-versus-staff separator is the IC not debugging. A senior engineer knows incidents need coordination. A staff engineer knows the coordination role must be separate from execution, that the instinct to put the most senior person head-down in a terminal is exactly what leaves nobody holding the picture, and that mitigate-first is the IC's job precisely because they are not debugging. Understanding why the roles are separated is the depth signal.

The second signal is connecting per-tenant SLAs to observability cardinality. Saying "a per-tenant SLA requires per-tenant SLIs, which is a high-cardinality slice a time-series database holds badly, so the contract you sign determines the observability you need" shows you follow the obligation to its infrastructure consequence rather than treating the SLA as a legal artifact.

The Staff & Team Lead Interview Reference