The DR ladder and global routing
What it is
Disaster recovery strategies form a ladder, and each rung buys a shorter recovery time for a predictable multiple of cost. The rung is chosen by the RTO and RPO, which are chosen by the cost of downtime, not by ambition.
RUNG RTO RPO Ongoing cost
(as % of prod)
-----------------------------------------------------------------
0 Multi-AZ only n/a for a n/a baseline
(one region) regional
failure
1 Backup and hours to hours ~2-5%
restore days (storage + egress)
2 Pilot light tens of seconds to ~10-20%
minutes to minutes (replicating data,
hours almost no compute)
3 Warm standby minutes seconds ~30-50%
(scaled-down but
running)
4 Active-active seconds to near zero 200%+, plus a
zero PERMANENT engineering
tax for conflict
handling
And the routing layer is a separate decision from the rung, because a warm standby you cannot send traffic to in under fifteen minutes has an RTO of fifteen minutes regardless of how ready it is.
What this is confused with: high availability and disaster recovery. Multi-AZ within a region is HA: it survives an instance, a rack or an availability zone. DR is about losing a region, or losing your account, or a regional control plane failing, and multi-AZ does nothing for any of those.
Also confused: "active-active" as a description of a system whose database has one writable primary. That is active-active reads with single-region writes, which is a legitimate and much cheaper architecture. Say which one you have, because the difference is the entire conflict-resolution problem.
The problem it solves
Without an explicit rung, the organisation has an implicit one, and it is usually rung 1 with an untested restore.
"What's our DR plan?"
"We have backups."
"How long to restore?"
"...we've never done it."
That is rung 1 with an unmeasured RTO, which is functionally
"we do not know", and it is the most common actual state of
systems whose documentation claims warm standby.
And the failure of choosing a rung by ambition rather than by arithmetic:
A team builds active-active for a system whose downtime costs
roughly $4,000/hour.
Cost of active-active: a second region at full capacity, plus
roughly 2 engineer-quarters to handle write conflicts, plus a
permanent tax on every new feature that touches shared state.
Cost of warm standby with a 12-minute RTO: ~35% of a region,
and no conflict handling at all.
Expected annual downtime avoided by moving from 12 minutes to
30 seconds, at a realistic 1-2 regional events per year:
roughly 20 minutes, worth about $1,300.
They spent low six figures a year plus a permanent engineering
tax to avoid $1,300 of expected loss.
The arithmetic is not hard and it is almost never done, which is why the rung should be derived from the priced RTO (see RTO and RPO, extracted and priced) before anything is built.
Mechanics
The four rungs, concretely
Rung 1: backup and restore.
What runs in the second region: nothing.
What exists: backups, replicated cross-region, immutable.
RTO is dominated by RESTORE TIME, which nobody measures until
the incident:
4 TB at a sustained 200 MB/s = ~5.5 hours of transfer alone
plus provisioning, plus index rebuild, plus warm-up, plus
DNS and verification
So a claimed "4-hour RTO" for a 4TB database is usually
fiction. Measure it, once, and put the measured number in the
runbook.
Right when: downtime costs little, the data is small, or the
system is genuinely rebuildable (a stateless service with data
that can be re-derived).
Rung 2: pilot light.
What runs: continuous data replication, and nothing else.
The compute, the load balancers, the caches all exist as
infrastructure-as-code and are provisioned on demand.
RTO = provision + deploy + warm + cut over.
Realistically 20 minutes to 2 hours depending on how much of
the estate is scripted and how honest the scripts are.
The failure: the IaC has not been applied to that region in
eight months, so it does not work. Fix: apply it on a
schedule, to a real (if tiny) footprint, so drift is caught by
a cron job rather than by an outage.
Rung 3: warm standby.
What runs: a scaled-down but FUNCTIONAL copy, receiving
replicated data, and continuously exercised by synthetic
traffic.
RTO = scale up + cut over. Minutes.
Two properties that make it real rather than nominal:
- it serves some real traffic continuously (even 1%), so
"does it work" is answered every minute rather than during
the incident
- it is scaled up regularly under load, so the scale-up path
is tested
Warm standby with zero traffic is pilot light with a bigger
bill.
Rung 4: active-active.
What runs: both regions serving production traffic at full
capacity.
The infrastructure cost is the smaller half. The real cost is
that every piece of mutable shared state now needs a conflict
story:
- which region owns a given entity, or
- a CRDT, or
- last-write-wins with a clock you trust, which loses data,
or
- a global consensus store in the write path, which
reintroduces the latency you were avoiding
And it is a PERMANENT tax: every new feature touching shared
state pays it, forever, not once.
Right when: the RTO requirement is genuinely seconds, or the
traffic genuinely needs to be served locally in both regions
for latency reasons, in which case you were building it
anyway.
The intermediate rung people forget: active-active reads, single-region writes.
Reads served locally from a replica in every region.
Writes routed to one home region.
RTO for reads: seconds (the routing layer fails over).
RTO for writes: whatever your promotion takes, minutes.
Conflict handling: NONE, because there is one writer.
This covers a large fraction of real workloads at a fraction
of the cost and complexity of true active-active, and it is
what most systems describing themselves as active-active
actually are.
Global routing, and why DNS is not a failover mechanism
GeoDNS
Return different A records based on the resolver's location.
MECHANISM OF FAILOVER: change the record, wait for TTL.
ACTUAL failover time: TTL, plus resolver non-compliance
(many resolvers enforce their own minimums), plus OS
caching, plus browser caching, plus negative caching.
With a 60-second TTL, observed full drain routinely takes
5 to 15 minutes, and a long tail continues for hours.
So: fine for planned migrations, unusable as the mechanism
for a sub-5-minute RTO.
Also: it routes by RESOLVER location, not client location,
which is wrong for anyone using a public resolver.
ANYCAST
The same IP announced from many locations; BGP delivers the
packet to the topologically nearest announcement.
FAILOVER: withdraw the announcement. BGP converges in
seconds to tens of seconds, with no client involvement.
This is why CDNs and public DNS resolvers use it.
Caveat: you do not control which PoP a client reaches, and a
route change mid-connection can break a long-lived TCP
connection, which matters for WebSockets and gRPC streams
and not for short HTTP requests.
GLOBAL L7 LOAD BALANCER (anycast front door)
An anycast IP terminates at the nearest edge; the edge
health-checks regional backends and forwards over the
provider's backbone.
FAILOVER: at the edge, in seconds, with no DNS change and no
client cache to wait for.
This is the modern default (Cloudflare, Google Cloud's
global LB, AWS Global Accelerator, Azure Front Door), and
it is the answer to "how do I fail over in seconds".
CLIENT-SIDE
The client holds a list of regional endpoints and does its
own health checking and failover.
FAILOVER: as fast as the client's checks, and it works even
if your edge is the thing that failed.
Right for mobile and desktop clients you control. Wrong for
a browser, and it requires shipping a client update to
change the endpoint list, which is its own RTO.
The rule worth memorising: your RTO cannot be shorter than your routing layer's failover time. A warm standby with a 90-second RTO behind a GeoDNS record with a 300-second TTL has a real RTO of several minutes at best.
Layer Realistic failover
--------------------------------------------
BGP anycast seconds to tens of seconds
Global L7 LB seconds
Client-side seconds (client's check interval)
DNS, TTL 60s 5 to 15 minutes, long tail hours
DNS, TTL 300s 15 to 60 minutes, long tail hours
Health checks, and the thing they usually get wrong
A health check that only proves the process is up will happily
route traffic to a region whose database replica is broken.
The health check that a failover decision depends on must
assert the DEPENDENCIES, not the process:
- can I read from the primary datastore?
- can I write, if this region is supposed to accept writes?
- is replication lag within tolerance?
- can I reach the identity provider?
And it must be shallow enough not to cascade: if the health
check calls three downstream services and any of them being
slow marks the whole region unhealthy, one slow dependency
takes out both regions.
The usual shape: a deep check for the failover decision,
evaluated centrally at a low frequency; a shallow check for
per-instance load balancing, evaluated constantly.
Conflating those two checks is how a single slow dependency causes a global outage, because the load balancer's per-instance check starts failing everywhere at once.
A worked example: a 12-minute RTO that was really 47 minutes
A payments platform. Documented DR posture: warm standby in a second region, "RTO 12 minutes, RPO 30 seconds". A regional control-plane degradation forced the first real failover in three years.
What actually happened:
t+0 region A's managed database control plane degrades.
Existing connections work; new connections fail
intermittently.
t+4m error rate crosses the alert threshold.
t+9m incident declared. Debate begins about whether this
warrants a failover, because "it's only intermittent".
t+21m failover declared.
t+22m standby database promoted. Took 90 seconds. This part
worked exactly as documented.
t+24m DNS records updated. TTL was 300 seconds.
t+29m ~40% of traffic on region B.
t+38m ~85%.
t+47m ~99%. A long tail continued for 3 hours from clients
with aggressive caching.
Documented RTO: 12 minutes.
Actual: 47 minutes to 99%, and the first 21 of those were a
conversation.
Two separate failures: a decision that took 21 minutes, and a routing layer that took 25. The database promotion, which was the part the team had practised, took 90 seconds.
The routing fix:
Replaced GeoDNS + health checks with a global anycast L7 load
balancer, health-checking both regions and forwarding over
the provider backbone.
failover mechanism: edge stops forwarding to region A
measured in a game day: 6 seconds to 99% of traffic on
region B
DNS now points at a single anycast IP that never changes, so
DNS is no longer in the failover path at all.
Cost: ~$3,100/month for the global load balancer, against a
downtime cost of roughly $22,000 per hour for this system.
Break-even: about 8.5 minutes of avoided downtime per month,
which one avoided incident covers.
Taking DNS out of the failover path entirely, rather than tuning its TTL, is the structural fix, and the TTL-tuning alternative was measured too: dropping to 30 seconds got 99 percent of traffic moved in about 9 minutes, still far above the requirement.
The health-check fix, which was found during the same work:
The existing region health check called /health, which
returned 200 if the HTTP server was up.
During the incident, region A's /health returned 200
throughout, because the process was fine and only NEW database
connections were failing. So no automated system had any
signal, which is part of why the decision took 21 minutes:
every dashboard the responders looked at said region A was
healthy.
New deep check, evaluated centrally every 10 seconds:
- acquire a NEW connection from the pool and run SELECT 1
- write and read back a heartbeat row
- assert replication lag < 5s
- assert the IdP token endpoint is reachable
Any failure for 3 consecutive checks marks the region
unhealthy for failover purposes.
Kept separate from the per-instance load balancer check, which
remained shallow, so a slow dependency cannot mark every
instance in both regions unhealthy simultaneously.
"Every dashboard said the region was healthy" is the finding that explains the 21-minute debate, and it was a health check measuring the wrong thing rather than a failure of nerve.
The rung question, revisited with arithmetic:
After the incident there was a proposal to move to
active-active.
Priced:
second region at full capacity +$41k/month
conflict handling for the ledger ~2 engineer-quarters,
plus a permanent tax on
every feature touching
balances
expected benefit: RTO from ~1 minute (post-fix) to seconds
Downtime cost: ~$22k/hour.
Regional events observed: 1 in 3 years for this provider and
region pair.
Expected annual saving from the RTO improvement: well under
$1k.
REJECTED, in writing, with the arithmetic in the ADR.
What was funded instead:
- the global load balancer (done)
- the deep health check (done)
- a written failover threshold with a named decider, so the
21-minute debate cannot recur
- quarterly game days
Writing the rejected option's arithmetic into the ADR is what stopped the proposal returning every six months, which it had done twice before.
Measured at the next two game days:
incident game day 1 game day 2
(before) (+3 months) (+6 months)
decision to declare 21 min 4 min 90 s
routing failover 25 min 6 s 6 s
database promotion 90 s 85 s 80 s
total to 99% traffic 47 min ~5 min ~2.5 min
The decision time fell further than any technical number, and it fell because the threshold was written down in advance, which cost one meeting.
Production evidence
AWS's published disaster recovery whitepaper defines the four strategies used here, backup and restore, pilot light, warm standby, and multi-site active-active, with the same RTO/RPO and cost ordering, and it is the canonical source for the ladder framing.
Anycast for failover is how public DNS resolvers and CDNs achieve near-instant regional withdrawal: Cloudflare, Google Public DNS and every major CDN announce the same address space from many locations, and BGP convergence rather than DNS TTL is the recovery mechanism.
Global load balancers with anycast front doors (Google Cloud's global external load balancer, AWS Global Accelerator, Azure Front Door, Cloudflare) all document failover at the edge without a DNS change, which is the mechanism that decouples RTO from client-side DNS caching.
DNS TTL non-compliance is well documented: resolvers commonly enforce minimum TTLs, operating systems and browsers cache independently, and measurement studies consistently show a long tail of traffic continuing to the old address for far longer than the configured TTL.
Deep versus shallow health checks are covered in AWS's builders' library article on health checks, including the specific failure mode where a deep check causes correlated failure across an entire fleet when a shared dependency slows, and the recommendation to separate the check used for a large-scale decision from the one used for per-instance routing.
The cost of active-active write conflict handling is visible in the published designs of systems that took it on: DynamoDB global tables use last-writer-wins with an explicit data-loss caveat, Cosmos DB offers configurable conflict resolution policies, and CRDT-based stores accept restricted data types in exchange for automatic convergence. None of them make the problem disappear, which is the argument for pricing it as a permanent tax.
The debate
Which rung is right? Whichever one the priced RTO buys. The arithmetic is straightforward, cost of downtime per hour times expected regional events per year times the RTO improvement, and it is almost never done, which is how organisations end up spending six figures a year to avoid four figures of expected loss. Do the arithmetic and write the rejected option into the ADR, because the proposal returns otherwise.
Is active-active worth it? When the RTO requirement is genuinely seconds, or when you need local writes in multiple regions for latency or residency reasons anyway. Otherwise the honest comparison is against active-active reads with single-region writes, which delivers most of the availability benefit with none of the conflict problem, and which is what many systems calling themselves active-active actually are.
Should failover be automatic? For stateless and read paths, yes. For the write path, a human decision with a pre-agreed threshold is usually right, because automatic promotion risks split-brain and flapping, and because the cost of a wrong automatic failover during a transient blip is a reconciliation problem. The counter-argument is that humans took 21 minutes in the worked example, and the answer to that is a written threshold and a named decider rather than automation.
Is DNS-based failover ever acceptable? For planned migrations and for RTOs measured in tens of minutes, yes, and it is cheap. For anything faster it is not a mechanism, because the failover time is set by resolver and client caching you do not control, with a long tail measured in hours.
Are deep health checks a good idea? For the failover decision, essential: a check that only proves the process is up will report a region healthy while its database is unreachable. For per-instance load balancing, dangerous, because a shared slow dependency marks every instance unhealthy at once. Use both, at different frequencies, for different decisions, and never let the same check drive both.
Does warm standby need real traffic? Yes, or it is pilot light with a bigger bill. The property that makes a standby real is that "does it work" is answered continuously rather than during the incident, and one percent of production traffic plus a periodic scale-up exercise is enough to get it.
Follow-up Q&A
"Walk me through the DR ladder."
Backup and restore, where nothing runs in the second region and the RTO is dominated by restore time, which for a few terabytes is hours and is usually unmeasured. Pilot light, where data replicates continuously and compute is provisioned on demand, giving tens of minutes to hours. Warm standby, a scaled-down functional copy, giving minutes. Active-active, both regions at full capacity, giving seconds at more than double the infrastructure cost plus a permanent engineering tax for write conflicts. And the rung people forget, active-active reads with single-region writes, which delivers most of the availability at a fraction of the cost because there is one writer and therefore no conflicts.
"How do you choose the rung?"
From the priced RTO. Cost of downtime per hour, times expected regional events per year, times the downtime avoided by the RTO improvement. In one case a team proposed moving from a one-minute RTO to seconds at 41,000 dollars a month plus two engineer-quarters plus a permanent tax on every feature touching balances, against an expected annual saving well under a thousand dollars. The proposal was rejected with the arithmetic written into the ADR, which is what stopped it returning every six months as it had twice before.
"Why is DNS not a failover mechanism?"
Because the failover time is set by caching you do not control. Resolvers commonly enforce their own minimum TTLs, operating systems and browsers cache independently, and negative caching adds more. With a 60-second TTL, moving 99 percent of traffic routinely takes 5 to 15 minutes with a tail continuing for hours. In one incident a 300-second TTL took 25 minutes to reach 99 percent. It is fine for planned migrations and for RTOs of tens of minutes, and it cannot deliver a sub-five-minute RTO. It also routes by resolver location rather than client location, which is wrong for anyone on a public resolver.
"What do you use instead?"
An anycast front door: a global L7 load balancer with an anycast IP that never changes, health-checking regional backends and forwarding over the provider's backbone. Failover happens at the edge in seconds, with no DNS change and no client cache to wait for. In one case that took routing failover from 25 minutes to 6 seconds, at about 3,100 dollars a month against a downtime cost of 22,000 dollars an hour, which breaks even at roughly eight minutes of avoided downtime a month. The general rule is that your RTO cannot be shorter than your routing layer's failover time.
"What is wrong with most health checks used for failover?"
They prove the process is up rather than that the region can serve. In one incident the region's health endpoint returned 200 throughout, because the HTTP server was fine and only new database connections were failing, so every dashboard said the region was healthy and the failover decision took 21 minutes. A failover health check must assert dependencies: acquire a new connection and query, write and read back a heartbeat, check replication lag, check the identity provider. And it must be separate from the shallow per-instance check used for load balancing, because a deep check driving instance routing means one slow shared dependency marks every instance in every region unhealthy at once.
"What makes a warm standby real rather than nominal?"
Continuous traffic and a regularly exercised scale-up path. If the standby serves no traffic, "does it work" is a question first answered during the incident, and the answer is frequently no because infrastructure-as-code has not been applied there in months. One percent of production traffic plus a periodic scale-up under load turns both into continuously answered questions. A warm standby with zero traffic is pilot light with a bigger bill.
Common misconceptions
"Multi-AZ is disaster recovery." It is high availability within a region. It does nothing for a regional failure, an account-level problem, or a regional control-plane outage.
"We're active-active." Usually it means active-active reads with a single writable primary, which is a different and much cheaper architecture. Say which one you have.
"Our RTO is 12 minutes." Not if the routing layer takes 25. The RTO is bounded below by the slowest link in the failover path, and it is usually DNS or the decision.
"Lower the TTL and DNS failover is fine." Resolver minimums, OS and browser caching, and negative caching mean the observed drain is much longer than the configured TTL, with a tail of hours.
"A backup is a recovery plan." A backup nobody has restored is a hypothesis, and restore time for a few terabytes is hours that nobody has measured.
"Active-active costs twice as much." The infrastructure does. The conflict handling is a permanent tax on every future feature touching shared state, and that is the larger cost.
Interview delivery note
Say this verbatim: "Your RTO cannot be shorter than your routing layer's failover time. We had a documented twelve-minute RTO, the database promotion took ninety seconds exactly as designed, and it still took forty-seven minutes to move traffic, because DNS with a five-minute TTL was in the failover path and the decision itself took twenty-one." It names the constraint and gives the decomposition that makes it concrete.
The senior-versus-staff separator is pricing the rung and writing the rejected option down. A senior engineer proposes active-active after an incident. A staff engineer prices it, 41,000 dollars a month plus two engineer-quarters plus a permanent tax on every feature touching shared state, against an expected annual saving under a thousand dollars given one regional event in three years, rejects it in writing, and funds the global load balancer and the written failover threshold instead. Recording the arithmetic is what stops the proposal returning.
The second signal is separating the deep failover health check from the shallow routing one. Knowing that a health check which only proves the process is up will report a region healthy while its database is unreachable, and that making the routing check deep instead means one slow shared dependency marks every instance everywhere unhealthy at once, is the detail that distinguishes someone who has run a failover from someone who has read about one.
Further reading
- AWS's disaster recovery whitepaper, for the four-strategy ladder with RTO, RPO and cost ordering.
- AWS Builders' Library, "Implementing health checks," for the deep-versus-shallow distinction and the correlated-failure risk.
- Documentation for a global anycast load balancer (Google Cloud global external LB, AWS Global Accelerator, Azure Front Door) on edge failover without DNS changes.
- The RTO and RPO, extracted and priced page, which produces the numbers this page's rung choice depends on.
- The multi-region write path and active-active conflict resolution pages, for the cost that rung 4 actually carries.