RTO and RPO, extracted and priced

"The business says we can't lose any data and can't be down. What do you say next?"

What it is

RTO (Recovery Time Objective) is how long the service may remain unavailable after a disaster before the impact becomes unacceptable. It is a time to restore.

RPO (Recovery Point Objective) is how much data you may lose, expressed as time: an RPO of five minutes means that after a disaster, the last five minutes of writes may be gone. It is a distance back to the last usable copy.

They are independent. A system can restore in 30 seconds and lose an hour of data (fail over to a stale replica), or take eight hours to restore and lose nothing (replay a synchronous log onto rebuilt infrastructure). Conflating them is the most common error in this conversation, and keeping them separate is the first thing to demonstrate.

Both are commonly confused with availability SLOs, which cover ordinary failures: a bad deploy, an instance dying, a dependency degrading. RTO and RPO cover disasters: losing a region, losing a database irrecoverably, a ransomware event, an accidental DROP TABLE propagated to every replica. Different failure class, different mechanisms, different budget.

The problem it solves

"We can't lose any data and can't be down" is not a requirement, it is a wish. Zero RTO and zero RPO are unachievable at any budget: the speed of light bounds synchronous replication distance, and any single failure domain can be lost.

What the two numbers do is convert an argument about feelings into an engineering problem with a price attached. Every disaster recovery architecture is a point on a cost curve, and the numbers select the point. Without them, teams either over-engineer (multi-region active-active for an internal tool) or under-engineer (nightly backups for a payments system), and both failures are expensive.

The second thing the numbers do is force the business to own the tradeoff. An engineer choosing RPO is an engineer deciding how much of the company's money may evaporate, which is not an engineering decision.

Mechanics

The extraction conversation

You will not be handed these numbers. You extract them, and the way to do it is to ask about consequences rather than preferences, because everyone prefers zero.

For RTO, ask what happens during the outage:

• "If the service is down for one hour on a Tuesday afternoon, what happens? Walk me through who notices and what they do."
• "At what point do customers leave rather than wait?"
• "Is there a manual fallback? How long can people run it?"
• "Is there a contractual or regulatory deadline we would breach?"

For RPO, ask what happens to the lost work:

• "If we lose the last hour of writes, can we reconstruct them from somewhere else? Emails, a partner's records, the customer's own copy?"
• "Which specific data would be unrecoverable, and what is that worth?"
• "Is anything irreversible? A payment sent, a message delivered, a document signed?"

The answers are almost always different per data class, which is the insight that makes the conversation productive. A single company-wide RPO is nearly always wrong:

Getting a business stakeholder to say "we need zero RPO on the ledger and four hours is fine for analytics" is the whole objective, because it converts an infeasible global requirement into a feasible per-tier one, and it usually reduces the cost by an order of magnitude.

The ladder, with prices

The percentages are of primary-region infrastructure cost and are indicative, not quoted. The shape is what matters: the cost is not linear in the guarantee. Going from hours to minutes roughly triples cost; going from minutes to near-zero roughly quadruples it again, and adds permanent engineering complexity that does not appear on the infrastructure bill at all.

That last cost is the one to name, because it is invisible in a budget conversation and dominant in practice: active-active means every feature must be designed for concurrent writes in two places, every schema migration must run in two regions, every engineer must understand conflict resolution, and your correctness bar rises permanently.

The physics that bound RPO

Synchronous replication gives RPO of zero and costs latency. Every write waits for acknowledgement from the remote replica, so write latency includes a round trip. Light in fibre travels at roughly 200,000 km/s, so about 1 ms per 100 km round trip. London to Frankfurt is roughly 8 ms round trip, London to Virginia roughly 75 ms. Adding 75 ms to every write is not viable for an interactive service, so cross-continent synchronous replication is off the table for most workloads, and that is a physical constraint rather than a budget one.

Asynchronous replication gives you latency back and an RPO equal to your replication lag. So the honest formulation is: your RPO is your p99 replication lag at the moment of the disaster, which means replication lag is not a health metric, it is your RPO measured live. Alerting on it is alerting on whether you can meet your commitment.

The middle ground is synchronous within a region or metro (single-digit milliseconds, RPO zero for the common failure of losing one datacentre) and asynchronous across regions (RPO of seconds, for the rarer failure of losing a region). That covers the failure you will actually have without paying for the one you probably will not.

The dependency audit

Your real RTO is the maximum of your critical dependencies' RTOs, not your own. Three that every team forgets:

• The identity provider. If nobody can log in, the service is down whether or not it is running. Does your IdP fail over to your DR region? Have you tested it?
• DNS. Failover is usually a DNS change, so DNS must work and its TTL bounds how fast the change propagates. A 3600-second TTL means an hour of clients still resolving to the dead region regardless of how fast you flipped the record.
• The secrets manager. Services that cannot fetch credentials cannot start. If secrets live only in the failed region, your DR environment comes up and immediately fails health checks.

Add to those: the container registry (can you pull images in the DR region?), the CI system (can you deploy a fix?), and the observability stack (can you see what is happening?). A DR plan that assumes these all work is a plan that has never been tested.

Backup hygiene

The 3-2-1 rule: three copies, on two media types, one off-site. The modern addition is immutable and air-gapped, because ransomware deletes backups first and an attacker with your cloud credentials can delete an S3 bucket as easily as a database. Object lock or equivalent write-once storage is the control.

And the rule that matters more than any of the above: a backup that has never been restored is a hope, not a backup. Restore testing must be automated, scheduled, and produce a measured restore time, because that measured number is your RTO. Teams that assume a four-hour RTO from a backup they have never restored routinely discover it is eleven hours the first time they try, in the middle of the incident.

A worked example: the conversation

An e-commerce platform. The VP of Engineering says "we can't lose orders and we can't be down".

Extract, per data class. Twenty minutes of questions produces:

Note what happened: the infeasible global requirement became six feasible ones, and only the first row needs the expensive architecture.

Price it.

• Orders and payments: RPO 0 requires synchronous replication, which requires proximity. Solution: synchronous across availability zones within the region (RPO 0 for zone loss, which is the failure that actually happens), plus asynchronous cross-region with a measured lag under two seconds. Honest statement: for a full region loss, the RPO on orders is our replication lag, roughly two seconds, not zero. Achieving genuine zero across regions means paying 75 ms on every write, and the business should decide whether it wants that. It will not.
• Catalogue and accounts: warm standby. Scaled-down stack in the second region, data replicating asynchronously, scale up on failover. 15 to 20 minutes of RTO, dominated by scaling time, not data.
• Search, analytics, models: pilot light or backup-and-restore. Rebuild from sources.

Total: roughly 45 percent uplift on infrastructure, versus the 200 percent-plus that full active-active for everything would have cost. That number is the deliverable, and producing it is what turns the conversation from "we need everything" into a decision.

Then the uncomfortable part, which you must say: these numbers are aspirations until tested. The commitments become real when we have (a) run a region evacuation game day, (b) automated restore tests reporting actual restore times, and (c) verified the identity provider, DNS and secrets manager fail over with us. Until then the honest RTO is "unknown, probably several times our estimate".

Production evidence

AWS's Well-Architected disaster recovery guidance defines exactly the four-tier ladder above (backup and restore, pilot light, warm standby, multi-site active-active) with indicative RTO and RPO ranges for each, and it is the standard vocabulary in this conversation. Google Cloud and Azure publish equivalent frameworks with the same shape, which is good evidence the taxonomy is settled.

Google's SRE practice contributes the discipline point: DiRT (Disaster Recovery Testing) exercises, in which teams deliberately fail components and sometimes entire datacentres, exist because untested recovery procedures reliably fail. The SRE book's treatment of testing for reliability makes the argument that an untested claim is a hypothesis.

AWS Fault Injection Service, Gremlin, Chaos Mesh are the productised form of the same idea, and the existence of a market for region-evacuation tooling tells you how often the claim and the reality diverge.

Published post-incident reports from cloud providers are the best material on what actually goes wrong in a region event, and the recurring theme is dependency surprises: the failover worked and something upstream did not.

The debate

The alternative to a tiered DR plan is active-active for everything. Its advocates make a fair point: a standby you never use is a standby that does not work, and a system serving traffic from two regions continuously is exercising its failover constantly rather than hypothetically. There is no failover event to get wrong because there is no failover.

The costs are the ones people underestimate. Doubled infrastructure is the visible one. The invisible ones: every write path needs conflict resolution or single-writer-per-key routing, every schema migration runs twice, data residency becomes a constraint on where a user's data may live, and the engineering team carries a permanently higher correctness bar. Teams adopt active-active for availability and discover they have adopted a distributed systems problem.

My position: tier by data class, because a single company-wide RTO and RPO is always wrong in one direction or the other. Buy synchronous replication within a region and asynchronous across regions, which covers the failure that actually happens (losing a zone) at a cost proportional to its likelihood. Reserve active-active for the small set of systems that genuinely justify it, and be explicit that the price includes permanent engineering complexity, not just compute.

The tiered approach is wrong when the system is small enough that tiering costs more in complexity than it saves in infrastructure, when regulation mandates a specific posture regardless of cost, or when the organisation has repeatedly failed to execute failover, in which case active-active's "no failover event" property is worth paying for precisely because the failover is the thing you cannot get right.

Follow-up Q&A

"The business says they can't lose data and can't be down. What do you say next?" I ask what happens if we do, per system, because zero of both is unachievable at any budget and the useful conversation is about consequences. Different data classes get different answers: the ledger is unreconstructable and needs an RPO near zero, analytics can lose six hours and nobody notices. Then I price each tier on the standard ladder (backup and restore, pilot light, warm standby, active-active) and bring back a number. The goal is to convert an infeasible global requirement into feasible per-tier ones, which usually cuts the cost by an order of magnitude.

"Why can't you have RPO of zero across regions?" Physics and then economics. Zero RPO requires synchronous replication, which means every write waits for the remote acknowledgement, and light in fibre gives you roughly a millisecond per 100 km round trip. London to Virginia is about 75 ms added to every write, which destroys an interactive service. You can have synchronous within a metro (a few milliseconds, tolerable) and asynchronous across regions, in which case your RPO equals your replication lag. Which means replication lag is not a health metric, it is your RPO measured live, and you should alert on it as such.

"Who decides to fail over, and when?" This is the hard part, and it is organisational rather than technical. Name a single role that can declare (the incident commander, not a committee), give them objective triggers rather than judgement (region unreachable for N minutes, error rate above X for Y minutes), and pre-authorise the decision so nobody is looking for a VP at 3am. The failure mode I have seen most is a two-hour outage where the failover would have taken ten minutes but nobody felt empowered to pull the trigger, and every additional minute made the decision feel riskier.

"Why is failback harder than failover?" Because during the failover the secondary accumulated writes the primary never saw, so bringing the primary back means reconciling divergent state rather than resuming. If both regions took writes at any point, you have conflicts to resolve with no obvious authority. And failback is usually done under time pressure by a tired team who have just handled the original incident. The mitigations: make the failover direction symmetric so failback is the same procedure, keep a change log during the failover so reconciliation is mechanical, and practise failback in game days, which almost nobody does.

"What does a DR test actually look like?" Not a tabletop. A scheduled exercise where you evacuate a region in production, with an abort switch and a defined blast radius, and you measure the actual RTO. Announce it the first few times, then stop announcing it. The output is a measured number and a list of surprises, and the surprises are the value: the secrets manager that only existed in the primary, the DNS TTL nobody had lowered, the runbook step that referenced a decommissioned host. Do the same for backups: automated restore tests on a schedule, with the restore time as a reported metric, because that measured number is your real RTO.

Common misconceptions

The most common is that RTO and RPO are the same conversation. They are independent, they are met by different mechanisms (restore speed versus replication), and a system can be excellent at one and terrible at the other.

The second is that a replica is a backup. A replica faithfully replicates a DROP TABLE, an application bug that corrupts rows, and a ransomware encryption pass. You need point-in-time recovery and immutable copies for the failures that replication propagates rather than protects against.

The third is that a documented RTO is an RTO. It is an estimate until a restore test produces a measured number, and measured numbers are routinely several times the estimate.

Interview delivery note

Say this: "First I'd separate the two, because they're independent: RTO is how long we can be down, RPO is how much data we can lose. Then I'd ask what happens if we do, per system, because zero of both is unachievable and the answer is almost always different per data class. The ledger is unreconstructable, so it needs RPO near zero; analytics can lose six hours and nobody notices. Then I price each tier on the ladder and bring back a number, because the point of extracting the numbers is to turn 'we need everything' into a decision somebody can make."

The depth signal is the physics: "true RPO of zero across regions means synchronous replication, which means every write pays the round trip. That's about 75 milliseconds London to Virginia, so what we can actually offer is synchronous within the region and asynchronous across, which means our cross-region RPO is our replication lag. That makes replication lag our RPO measured live, and I'd alert on it as such."

Then close with the honesty that makes the whole answer credible: "and none of these numbers are real until we've run a region evacuation and an automated restore test, because a measured restore time is usually several times the estimate."

Further reading

• AWS Well-Architected Framework, the disaster recovery whitepaper, for the four-tier ladder with indicative RTO and RPO ranges.
• Google, Site Reliability Engineering, on testing for reliability and the DiRT exercise programme.
• Cloud provider post-incident reports for region-level events, which are the best public material on what actually fails during a failover.
• The 3-2-1 backup rule and object-lock / write-once storage documentation, for the ransomware-resistant variant.