Declaring failover, the runbook, failback and split-brain

What it is

Four things that decide whether a working DR capability produces a short outage or a long one:

THE DECISION      who is allowed to declare failover, against
                  what pre-agreed threshold, without asking
                  anyone else.

THE RUNBOOK       an executable sequence written for someone who
                  did not write it, at 3am, under stress, with
                  a verification step after each action.

FAILBACK          returning to the original region, which is
                  harder than failing over and is the step
                  people have never practised.

SPLIT-BRAIN       both regions accepting writes. Prevented by
                  fencing, detected by divergence checks, and
                  reconciled by a rule chosen in advance.

The decision is the part that is almost never designed, and in practice it is the largest component of real-world RTO.

What this is confused with: having a runbook and having a rehearsed one. A runbook not exercised in six months is a document, not a capability, because the infrastructure it describes has drifted and the people who can execute it have changed.

Also confused: failover and failback as symmetric operations. Failing over goes to a region whose data is behind by the replication lag. Failing back goes to a region whose data is behind by the entire duration of the outage, and which must catch up while the active region keeps moving.

The problem it solves

The decision delay is usually longer than the mechanism.

A real incident, decomposed:

  detection                4 min
  DECISION (debate)       21 min   <- the largest component
  database promotion       1.5 min  (worked exactly as designed)
  routing failover        25 min   (a separate, fixable problem)

The team had rehearsed the promotion. They had never rehearsed
the decision, because nobody thought of it as a thing that
could be rehearsed.

The debate is predictable and it has the same shape every time:

"Is this bad enough to fail over?"
"We don't know the cause yet. If we fail over and it's a bad
 deploy, we'll take the bug with us."
"How long will failover take?"
"Twelve minutes, in theory."
"Let's give it another ten minutes and see."
...twenty minutes later...
"Okay, let's do it."

Nobody behaved unreasonably. There was no threshold, so the
decision required consensus among people with incomplete
information and asymmetric fear.

And the failback problem, which produces second outages:

Region A recovers at 14:00. Someone fails back at 14:30,
during business hours, on the same day.

Region A's database is 4 hours behind. Nobody checked. Four
hours of orders vanish from the customer's view, some of which
were already fulfilled.

The reconciliation takes three days and involves finance.

Mechanics

The decision: a written threshold and a named decider

The threshold must be written BEFORE the incident, must be
observable without root-cause knowledge, and must not require
consensus.

  "If the regional deep health check fails for 3 consecutive
   evaluations (90 seconds) AND the error rate for critical
   journeys exceeds 5% for 5 minutes, the incident commander
   declares failover. No further approval is required.

   The incident commander MAY delay only if a bad deploy in
   the last 30 minutes is a plausible cause, in which case
   rollback is attempted first, with a 10-minute cap, after
   which failover proceeds regardless."

Four properties, and each removes a specific part of the observed debate:

1. OBSERVABLE WITHOUT DIAGNOSIS. "Error rate above 5% for 5
   minutes" is checkable now. "Is this a regional failure?" is
   a diagnosis and it is exactly the question that takes 20
   minutes.

2. A SINGLE NAMED ROLE, with authority. Not a committee. The
   role is the incident commander, and it is a role rather
   than a person so it always exists.

3. NO-REGRET BIAS, WRITTEN. Failing over unnecessarily is a
   recoverable annoyance; not failing over is an outage.
   Stating that in advance is what lets someone act on
   incomplete information.

4. THE ONE LEGITIMATE HESITATION, BOUNDED. "It might be our
   deploy" is a real concern and it deserves ten minutes and a
   rollback attempt, not an open-ended debate.

"The incident commander declares failover, no further approval required" is the sentence that removes the twenty minutes, and it costs one meeting to agree in advance and is impossible to agree during an incident.

The runbook

Written for a stranger, executed under stress, verified at every step.

Properties that separate a runbook from documentation:

  - COPY-PASTEABLE commands, not descriptions of commands
  - a VERIFICATION step after every action, with the expected
    output shown
  - explicit PRE-CONDITIONS at the top, so you know before
    starting whether this runbook applies
  - a STOP condition: what to do if a step's verification
    fails, which is usually "stop and escalate", not
    "continue"
  - NO decisions inside it. Decisions belong in the threshold.
    A runbook that says "assess whether..." is a document.
  - a stated DURATION per step, so the executor knows whether
    they are on track

## DR-01: Regional failover, us-east-1 -> us-west-2

PRE-CONDITIONS (verify all before starting)
  [ ] Failover declared by the incident commander (see DR-00)
  [ ] us-west-2 deep health check is GREEN
        aws cloudwatch get-metric-data --profile dr ... 
        EXPECT: HealthCheckStatus = 1
  [ ] Replication lag < 60s
        psql -h replica.us-west-2 -c "SELECT
          extract(epoch from (now() - pg_last_xact_replay_timestamp()));"
        EXPECT: < 60
        IF NOT: STOP. Escalate to the DBA on call. Failing over
        with high lag loses that much data; the IC must accept
        it explicitly.

STEP 1 (est. 30s) Freeze writes in us-east-1
  kubectl --context us-east-1 -n payments \
    patch cm feature-flags --type merge \
    -p '{"data":{"writes_enabled":"false"}}'
  VERIFY: curl -s https://api.internal/us-east-1/_flags | jq .writes_enabled
  EXPECT: false
  WHY: prevents split-brain. Any write accepted after the
  promotion decision is a write that will be lost or will
  conflict.

STEP 2 (est. 90s) Promote the us-west-2 replica
  aws rds promote-read-replica --db-instance-identifier payments-usw2
  VERIFY: aws rds describe-db-instances ... | jq '.[].StatusInfo'
  EXPECT: no replication status; instance available
  IF IT FAILS: STOP. Do not retry. Escalate.

STEP 3 (est. 10s) Shift traffic at the edge
  ...
STEP 4 (est. 60s) Verify a real transaction end to end
  ...
STEP 5 Post-failover
  [ ] Record the promotion LSN / timestamp. YOU WILL NEED THIS
      FOR FAILBACK.
  [ ] Announce on the status page
  [ ] Do NOT begin failback planning today

Step 1 before step 2 is the ordering that prevents split-brain, and the reason to state the WHY line in the runbook is that under stress someone will otherwise skip a step that appears to do nothing.

Recording the promotion point in step 5 is what makes failback possible, and it is the step most commonly omitted because the incident feels over.

Game days

A runbook not executed in six months is fiction. The things
that drift:
  - IAM permissions the executor needs
  - hostnames, ARNs, cluster names
  - the promotion command's flags (managed service APIs change)
  - who is on the rota and whether they have access
  - whether the standby's infrastructure-as-code still applies

EXERCISE, quarterly, on a schedule, in production or in a
production-shaped environment. The measurable output is TIME
PER STEP, compared to the runbook's estimate, plus a list of
every step that did not work as written.

The first game day always finds 5 to 15 broken steps. That is
the point, and it is much cheaper than finding them during an
incident.

Failback

Harder than failover, and it should be boring.

Why it is harder:
  - the original region's data is behind by the ENTIRE outage,
    not by the replication lag
  - the active region has kept moving, so the gap grows while
    you prepare
  - it is a PLANNED operation, so an outage during it is
    inexcusable in a way that the original was not
  - and nobody has practised it, because game days usually
    exercise failover and then restore from a snapshot

The sequence:

1. RE-REPLICATE. Rebuild the original region as a replica of
   the now-primary. For a large dataset this is a fresh
   snapshot plus catch-up, not an incremental resume, because
   the original diverged at the promotion point.
2. VERIFY. Replication lag near zero, sustained, and a
   row-count or checksum comparison on the largest tables.
3. WAIT. Do not fail back on the same day. Let the active
   region prove itself and let the original region's cause be
   understood.
4. SCHEDULE. A low-traffic window, announced, with the same
   runbook discipline as failover.
5. EXECUTE, using the same freeze-promote-shift sequence in
   reverse.
6. MONITOR at elevated attention for a full traffic cycle,
   which means at least a full weekday plus a weekend if your
   traffic has a weekly shape.

The rule: FAILBACK IS SCHEDULED, NEVER REACTIVE. The pressure
to "get back to normal" is exactly the pressure that produces
the second outage.

Split-brain

Two regions both accepting writes, which is a data problem rather than an availability problem, and therefore worse.

HOW IT HAPPENS
  - a network partition, where each region believes the other
    is dead
  - a failover performed while the original region is still
    serving (the freeze step skipped or failed)
  - an automated failover that flaps
  - a stale client or a cached DNS entry still writing to the
    old region after the cutover

PREVENTION, in order of strength:

  QUORUM / CONSENSUS. A majority is required to accept writes,
    so a minority partition cannot. This is what etcd, Raft
    and consensus-backed databases give you, and it is the only
    prevention that is actually a guarantee. It costs a third
    region or a witness.

  FENCING TOKENS. Every write carries a monotonically
    increasing epoch issued at promotion; the storage layer
    rejects writes with an old epoch. A demoted primary
    physically cannot write, even if it thinks it is still
    primary.

  LEASES with a bounded clock assumption. A primary holds a
    time-bounded lease and stops accepting writes when it
    expires. Depends on clock behaviour, so it is weaker than
    fencing.

  A WRITE FREEZE in the runbook, as above. Procedural, so it
    is only as strong as the procedure, and it is what most
    systems actually have.

DETECTION
  - a continuous divergence check: compare a monotonic counter,
    a row count, or a checksum of recent writes across regions
  - alert on ANY writes accepted in a region that is supposed
    to be passive. This should be impossible; alert on it
    anyway, because "impossible" is how you find out the
    freeze did not apply.

Reconciliation, and the rule must be chosen before it is needed:

LAST-WRITE-WINS
  Simple, and it loses data silently. Acceptable for a cache
  or a presence map, unacceptable for anything a customer or
  an auditor cares about.

PER-ENTITY OWNERSHIP
  Each entity has a home region; on conflict, the home
  region's version wins and the other is quarantined. Requires
  the ownership to be recorded, and it is the most defensible
  default for business data.

APPEND-ONLY / EVENT LOG
  If writes are events rather than state, both sides' events
  are retained and the merge is a matter of ordering, which is
  a much smaller problem. This is one of the strongest
  arguments for an event-sourced write model in a
  multi-region system.

MANUAL QUEUE
  Detected conflicts go to a queue with enough context for a
  human to resolve. Slow, correct, and the only acceptable
  answer for money.

Whatever the rule, it must be DECIDED IN ADVANCE and written
down, because choosing it during a reconciliation is choosing
it under pressure with an audience.

A worked example: a failover, a failback that failed, and the fix

A subscription billing platform. Warm standby, a rehearsed promotion, and no written failover threshold.

The first incident:

t+0     region A's managed Postgres begins rejecting new
        connections intermittently.
t+4m    alert.
t+9m    incident declared.
t+9-30m debate. Three engineers, one incident commander, one
        director joining at t+18m and reopening the question.
t+30m   failover declared.
t+31m   promotion runs. 80 seconds.
t+33m   traffic shifted (global load balancer, 6 seconds).
t+35m   service restored.

Decision: 21 minutes. Mechanism: 4 minutes.

The failback, four days later, which produced the second and worse outage:

Region A recovered on day 1. Failback was performed on day 4
at 10:00, a Tuesday.

What went wrong:

  The team rebuilt region A as a replica of region B. The
  replica reported "in sync". It was in sync with a snapshot
  taken 40 minutes earlier, and the catch-up stream had
  stalled 25 minutes into the process on a replication slot
  error that was logged and not alerted.

  Nobody verified lag independently. The runbook for failover
  had a lag pre-condition; the failback had no runbook at all,
  because it had never been practised.

  Failback proceeded. 40 minutes of billing events were
  invisible for 3 hours until a customer support escalation
  surfaced it.

  Recovery: replay from the event log (which existed, and was
  the thing that made this recoverable rather than
  catastrophic), reconciliation with finance, 3 days of
  cleanup.

The failover had a pre-condition checking replication lag and the failback did not, which is the asymmetry this page exists to prevent: teams write a runbook for the operation they fear and not for the one they perform calmly.

What was changed:

1. A WRITTEN THRESHOLD, agreed in a 45-minute meeting with the
   director, the IC rota and the DBA lead:

     "Deep health check red for 3 consecutive checks AND
      critical-journey error rate >5% for 5 minutes -> the IC
      declares failover. No further approval. The IC may
      attempt a rollback first if a deploy landed in the last
      30 minutes, capped at 10 minutes."

   Plus, explicitly: "Failing over unnecessarily is an
   accepted cost. The IC will not be second-guessed for a
   failover that turned out to be avoidable."

   That last sentence was requested by the IC rota and it is
   the one that made the threshold usable.

2. A FAILBACK RUNBOOK, with the same pre-conditions as
   failover plus two more:
     [ ] independent lag verification from BOTH sides
     [ ] row-count comparison on the 5 largest tables, within
         tolerance
     [ ] at least 48 hours since the original failover
     [ ] scheduled in a low-traffic window, announced

3. A DIVERGENCE CHECK, running continuously: a monotonic
   sequence per region compared every 30 seconds, alerting on
   any write accepted by a passive region.
   It fired twice in the first month, both times catching a
   stale internal client still pointed at the old region after
   a cutover. Neither would have been noticed otherwise.

4. FENCING. The write path acquired an epoch, incremented at
   promotion and checked by the storage layer, so a demoted
   primary physically cannot write.

5. QUARTERLY GAME DAYS, exercising failover AND failback.

The divergence check firing twice in a month, catching stale clients nobody knew about, is the return on a control that was supposed to detect an "impossible" condition, and it is the argument for alerting on impossibilities.

Game day results:

                     GD1        GD2        GD3
                  (+3 months) (+6 mo)   (+9 mo)
decision to declare   3 min     70 s      55 s
failover mechanism    4 min     3.5 min   3.5 min
failback (rehearsed) 41 min    22 min    18 min
steps that failed
  as written            11         3         1

GD1's 11 broken steps included: an IAM permission the IC did
not have, a cluster name that had changed, a promotion flag
deprecated by the provider, and a runbook link to a dashboard
that had been deleted.

None of those would have been found any other way, and every
one of them would have added minutes during a real incident.

Eleven broken steps on the first rehearsal of a runbook the team believed was current is the normal result, and it is the strongest available argument for game days.

One thing that was tried and reverted:

Automatic failover was implemented after the first incident,
triggering on the same threshold.

It fired twice in six weeks, both times on transient
degradations that self-resolved within 90 seconds. Each
automatic failover was a real failover: a promotion, a
divergence, and a scheduled failback three days later.

Net effect: two unnecessary failbacks, each carrying the risk
the failback incident had demonstrated.

Reverted to human declaration with the written threshold. The
decision time was already 70 seconds by then, so automation
was optimising a component that was no longer the bottleneck.

The lesson recorded: automate the DETECTION and the
PREPARATION (health checks, pre-condition verification, the
runbook's first steps queued and ready), and leave the
irreversible step to a human with a written threshold.

Production evidence

Google's SRE Book on incident management specifies a single incident commander with clear authority and separates the roles of command, operations and communications, which is the structure the written threshold depends on: a decision that requires consensus is a decision with no owner.

AWS's disaster recovery guidance treats failback as a distinct planned operation requiring its own procedure and its own verification, and warns explicitly that the recovered region's data is stale by the duration of the outage rather than by the replication lag.

Fencing tokens as the correct mechanism against a demoted primary continuing to write are argued in Martin Kleppmann's analysis of distributed locks and in Designing Data-Intensive Applications: without a monotonically increasing token checked at the storage layer, a paused or partitioned node can resume and write with stale authority.

Quorum-based systems (Raft, etcd, consensus-backed databases) prevent split-brain by construction because a minority partition cannot achieve a majority, which is why a third region or a witness is the standard price of the guarantee.

Chaos engineering and game-day practice at Netflix, Amazon and Google is documented specifically as the mechanism for discovering that a documented capability does not work, and the consistent finding that first exercises reveal a substantial list of broken steps is the reason the practice is scheduled rather than occasional.

Event-sourced write models as a reconciliation aid appear in multi-region designs precisely because retaining both sides' events reduces a merge to an ordering problem, whereas reconciling two divergent current-state snapshots loses information that was never recorded.

The debate

Should failover be automatic? Automate detection and preparation; leave the irreversible step to a human with a written threshold. The worked example is the argument: automatic failover fired twice in six weeks on transient degradations, producing two unnecessary failbacks, each carrying real risk. The counter-argument, that humans took twenty-one minutes, is answered by the written threshold, which took the decision to seventy seconds without automating it.

Is a no-regret bias correct? For most systems yes, and it must be stated in writing, including the sentence that the decider will not be second-guessed for a failover that turned out to be avoidable. Without that sentence the threshold is unusable, because the person with the authority also has the career exposure. It is not correct where failover itself carries data-loss risk, which is why the replication-lag pre-condition is a hard stop rather than a guideline.

Should the runbook contain decisions? No. A runbook step that says "assess whether" is a document pretending to be a procedure, and it reintroduces the debate at the worst moment. Decisions belong in the threshold, before the incident; the runbook executes.

How often should game days run? Quarterly, and the argument is drift rather than skill: IAM permissions, resource names, provider API flags and rota membership all change on that timescale. The measurable output is the count of steps that did not work as written, which was eleven on a first exercise of a runbook everyone believed was current.

Is last-write-wins ever acceptable? For a cache, a presence map or a preference that the user can reset, yes. For anything a customer or an auditor cares about, no, because it loses data silently and you find out from a reconciliation months later. Per-entity ownership is the defensible default, and a manual queue is the only honest answer for money.

Should you fail back at all? Not automatically and not quickly. The pressure to return to normal is exactly what produces the second outage, and in the worked example the failback caused a worse incident than the failure it was recovering from. If the standby region can serve indefinitely, running there until a scheduled low-traffic window is strictly better, and if it cannot, that is a capacity finding worth acting on separately.

Follow-up Q&A

"What is usually the biggest component of real-world RTO?"

The decision. In one incident, detection took 4 minutes, the debate took 21, the database promotion took 90 seconds exactly as designed, and routing took 25 because DNS was in the path. The team had rehearsed the promotion and had never rehearsed the decision, because nobody thought of a decision as something you could rehearse. The fix is a written threshold that is observable without diagnosis, a single named role with authority and no requirement for consensus, and an explicit statement that failing over unnecessarily is an accepted cost.

"What does a good failover threshold look like?"

Observable without root-cause knowledge, owned by one role, and bounded on its one legitimate hesitation. For example: deep health check red for three consecutive checks and critical-journey error rate above five percent for five minutes, at which point the incident commander declares failover with no further approval, and may attempt a rollback first only if a deploy landed in the last thirty minutes, capped at ten. The clause that makes it usable in practice is the written promise that the decider will not be second-guessed for a failover that turned out to be avoidable, because the person with the authority also carries the career exposure.

"What separates a runbook from documentation?"

Copy-pasteable commands rather than descriptions, a verification step after every action with the expected output shown, explicit pre-conditions at the top, a stop condition when a verification fails, a duration estimate per step so the executor knows whether they are on track, and no decisions inside it. A step that says "assess whether" reintroduces the debate at the worst possible moment. It should be executable by someone who did not write it, at 3am, under stress.

"Why is failback harder than failover?"

Because the recovered region is behind by the entire duration of the outage rather than by the replication lag, the active region keeps moving while you prepare, it is a planned operation so an outage during it is inexcusable, and almost nobody practises it. In one case a failback rebuilt the original region as a replica, the catch-up stream stalled 25 minutes in on a replication slot error that was logged and not alerted, nobody verified lag independently because the failback had no runbook, and 40 minutes of billing events were invisible for three hours. The failover runbook had a lag pre-condition; the failback had none.

"How do you prevent split-brain?"

In order of strength: quorum, so a minority partition cannot accept writes at all, which is the only real guarantee and costs a third region or a witness. Fencing tokens, a monotonically increasing epoch issued at promotion and checked at the storage layer, so a demoted primary physically cannot write. Leases, which depend on clock behaviour and are weaker. And a procedural write freeze in the runbook, which is what most systems actually have and is only as strong as the procedure. Detect it regardless, with a continuous divergence check that alerts on any write accepted by a region that is supposed to be passive.

"How do you reconcile after a split-brain?"

With a rule chosen in advance, because choosing it during a reconciliation means choosing under pressure with an audience. Last-write-wins is simple and loses data silently, which is fine for a cache and not for anything a customer or auditor cares about. Per-entity ownership, where the home region wins and the other version is quarantined, is the defensible default for business data. An append-only event log reduces the merge to an ordering problem, which is one of the strongest arguments for an event-sourced write model in a multi-region system. And a manual queue is the only honest answer for money.

Common misconceptions

"We have a runbook." Unexercised, it is a document. The first game day on a runbook everyone believed was current found eleven steps that did not work, including a missing IAM permission and a deprecated provider flag.

"Automatic failover is the goal." It fires on transient degradations and each false trigger produces a real failback with real risk. Automate detection and preparation; leave the irreversible step to a human with a written threshold.

"Failback is just failover in reverse." The recovered region is stale by the whole outage, not by the replication lag, and it is a planned operation where an outage is inexcusable.

"Get back to normal as soon as the region recovers." That pressure is what produces the second outage. Failback is scheduled, in a low-traffic window, at least a couple of days later.

"The write freeze prevents split-brain." It is procedural, so it is only as strong as the procedure. Fencing tokens or quorum are the mechanisms; the freeze is a practice.

"A divergence check is unnecessary because passive regions cannot write." Alert on the impossible anyway: in one system it fired twice in a month, both times catching a stale internal client still pointed at the old region.

Interview delivery note

Say this verbatim: "The biggest component of real RTO is usually the decision. We had a rehearsed promotion that took ninety seconds and a twenty-one-minute debate in front of it, and the fix was a written threshold that is observable without knowing the cause, owned by one role, with an explicit statement that failing over unnecessarily is an accepted cost." It identifies the unmeasured component and gives the three properties that fix it.

The senior-versus-staff separator is the sentence about not being second-guessed. A senior engineer writes a failover threshold. A staff engineer notices that the person holding the authority also holds the career exposure, and writes into the policy that a failover which turned out to be avoidable will not be held against the decider. That clause was requested by the on-call rota in the worked example and it is what took the decision time from twenty-one minutes to seventy seconds; the threshold alone would not have.

The second signal is treating failback as the more dangerous operation. Saying "our failover runbook had a replication-lag pre-condition and our failback had no runbook at all, which is how a stalled catch-up stream cost us forty minutes of billing events" shows you know teams write procedures for the operation they fear rather than the one they will perform calmly, and that the calm one is where the second outage comes from.

The Staff & Team Lead Interview Reference