Composite SLOs and dependency availability math

What it is

The arithmetic for computing a service's achievable availability from its dependencies', and the design decisions that follow from it.

Two combining rules, and everything else is an application of them:

SERIAL (all required)     A_total = A_1 × A_2 × ... × A_n
                          Availability MULTIPLIES DOWN.

PARALLEL (any suffices)   A_total = 1 - (1-A_1)(1-A_2)...(1-A_n)
                          UNavailability multiplies down.

Three dependencies at 99.9% each:
  Serial:   0.999^3           = 99.70%   (worse than any one)
  Parallel: 1 - 0.001^3       = 99.9999% (far better than any one)

Commonly confused with a modelling nicety. The serial rule is why a service composed of several 99.9 percent dependencies cannot itself be 99.9 percent, and that is a hard ceiling you discover either by arithmetic or by missing your SLO.

Also commonly confused: the rules assume independence, and real failures are correlated. Two replicas in the same rack, two regions on the same provider, or two services calling the same database are not independent, and the parallel formula over-states the benefit substantially. Naming that assumption is the difference between using the arithmetic and believing it.

The problem it solves

Availability targets are usually set by aspiration and then discovered to be impossible.

"The API should be 99.99% available."

It calls:
  auth service          99.95%
  user database         99.99%
  product catalogue     99.9%
  recommendations       99.5%
  payments provider     99.9%

All required?  0.9995 × 0.9999 × 0.999 × 0.995 × 0.999
             = 99.33%
*** The ceiling is 99.33%. The target is 99.99%. ***
*** It is not 66x away, it is unreachable by construction. ***

The arithmetic ends the conversation and, more usefully, points at the fix: the recommendations service at 99.5 percent costs more availability than everything else combined, and it is the least important dependency in the list.

Mechanics

Working the serial chain

Contribution of each dependency to total UNavailability:

  dependency        A          U = 1-A      share of total U
  ----------------------------------------------------------
  recommendations   99.5%      0.00500      74.7%
  auth              99.95%     0.00050       7.5%
  catalogue         99.9%      0.00100      14.9%
  payments          99.9%      0.00100      14.9%   (see note)
  user database     99.99%     0.00010       1.5%
  ----------------------------------------------------------
  total U ≈ sum of U (for small U)  = 0.00760 -> A = 99.24%

The approximation worth knowing: for small unavailabilities, total unavailability is approximately the sum, because the cross terms are second-order. That makes the arithmetic doable in your head:

0.9995 × 0.9999 × 0.999 × 0.995 × 0.999
  ≈ 1 - (0.0005 + 0.0001 + 0.001 + 0.005 + 0.001)
  ≈ 1 - 0.0076 = 99.24%

Exact: 99.243%. The approximation is accurate to three
decimal places, and it is far faster.

And it makes the design conclusion obvious: unavailability adds, so the worst dependency dominates. Improving the 99.99 percent database to 99.999 buys 0.00009; removing the 99.5 percent dependency from the critical path buys 0.005, which is fifty times more.

Removing a dependency from the critical path

The highest-leverage move, and it is usually cheap.

BEFORE: recommendations required
  A = 99.24%

AFTER: recommendations optional, with a cached fallback
  Its failure no longer fails the request; the page renders
  with popular items instead.
  A = 0.9995 × 0.9999 × 0.999 × 0.999 = 99.74%

  *** +0.5 percentage points, which at 99.24% is going from
      5.5 hours of downtime a month to 1.9. ***
  Cost: a fallback path and a cache. Days, not quarters.

The general principle: every dependency on the critical path multiplies its unavailability into yours, so the question for each one is "what happens if this is down" and the answer should not be "the request fails" unless it genuinely must.

The classification to apply to every dependency:

  REQUIRED       the request is meaningless without it
                 (auth for a private resource, the database
                 for the primary data)
  DEGRADABLE     the request works less well without it
                 (recommendations, related items, personalisation)
  ASYNCHRONOUS   the request does not need it at all
                 (analytics, audit logging, notifications)

Most dependencies teams treat as required are degradable, and reclassifying them is where the availability is.

Parallel redundancy, and the independence trap

Two payment providers, either sufficient, each 99.9%:
  A = 1 - (0.001)^2 = 99.9999%

That number is almost certainly wrong.

The correlation correction:

If a fraction c of failures are COMMON to both (a shared
network path, a shared upstream, a correlated traffic spike):

  A_effective ≈ 1 - [ c·U + (1-c)·U² ]

With U = 0.001 and c = 0.10 (10% of failures are common):
  = 1 - [0.10 × 0.001 + 0.90 × 0.000001]
  = 1 - 0.0001009
  = 99.99%

*** Not 99.9999%. Two orders of magnitude worse, from a 10%
    correlation. ***

That sensitivity is the point. Redundancy's benefit is dominated by the correlated fraction, not by the individual availabilities, which means the engineering that matters is reducing correlation, not adding a third replica.

Sources of correlation people miss:
  same availability zone            -> power, network, cooling
  same cloud provider               -> control plane, IAM, DNS
  same deployment pipeline          -> a bad deploy hits both
  same configuration source         -> a bad config hits both
  same certificate authority        -> expiry hits both
  same upstream dependency          -> the shared thing fails
  correlated demand                 -> the spike hits both

A deploy pipeline shared between two "redundant" regions is the one that catches people, because it looks like redundancy on the architecture diagram and a bad rollout takes both regions simultaneously.

When several services contribute to one user journey, how is the error budget divided?

User journey: "search and add to cart"
  frontend  ->  search API  ->  ranking  ->  cart API  ->  database

Journey SLO: 99.9% (43 minutes per 30 days)

OPTION A: EQUAL SPLIT
  Each of 5 services gets 1/5 of the budget: 8.6 minutes.
  Each needs 99.98% individually.
  Simple, and it ignores that services have different
  difficulty and different traffic shares.

OPTION B: PROPORTIONAL TO DIFFICULTY
  The database is easier to keep up than the ranking service,
  which depends on a model server and a feature store.
  Allocate: db 3 min, cart 5 min, search 10 min, ranking 20 min,
  frontend 5 min.
  More realistic, and it needs negotiation.

OPTION C: MEASURE THE JOURNEY DIRECTLY
  Instrument the end-to-end journey and set ONE SLO on it.
  Individual services have their own SLOs for their own
  operational purposes, and the journey SLO is what the
  business commits to.
  *** This is the right answer, and it requires
      journey-level instrumentation that most systems lack. ***

Why option C wins: the sum of component SLOs is not the journey SLO, because failures are correlated (one dependency taking down three services), because not every request touches every service, and because a component's SLO measures its own requests rather than the journey's.

Concretely: the ranking service is 99.9% available, and 40%
of searches do not call it because they hit the cache. Its
unavailability contributes 0.4 × 0.001 to the journey, not
0.001.

Weighting by the fraction of journeys that actually touch a dependency is the correction most composite calculations omit.

Where the arithmetic misleads

1. IT ASSUMES BINARY UP/DOWN.
   Real services degrade: slow, partially failing, failing
   for one tenant. A service at "99.9% available" may be
   serving 99.9% of requests successfully while being
   unusable for 5% of users.

2. IT ASSUMES INDEPENDENCE.
   Addressed above, and it is the largest error.

3. IT IGNORES RETRIES.
   A dependency at 99.9% per attempt, with one retry and
   independent failures, is 99.9999% per logical call. Retries
   are cheap redundancy and they change the arithmetic
   substantially, which is why they belong in the model.

4. IT IGNORES TIMEOUTS AND FALLBACKS.
   A dependency that is "down" but whose failure is detected
   in 50 ms and served from cache has not caused an outage.
   The availability that matters is the availability of the
   USER-VISIBLE OUTCOME, not of the dependency.

Point 4 is the reframe: you are not composing dependency availabilities, you are composing the availability of outcomes, and a fast failure with a good fallback contributes almost nothing to unavailability.

A worked example: raising a ceiling

STARTING POINT
  API SLO target: 99.9%. Measured: 99.31%. Consistently missing.

STEP 1: compute the ceiling.
  auth 99.95, db 99.99, catalogue 99.9, recs 99.5, payments 99.9
  All on the critical path.
  U_total ≈ 0.0005 + 0.0001 + 0.001 + 0.005 + 0.001 = 0.0076
  A_ceiling = 99.24%
  *** The SLO was never achievable. We were not failing to
      execute; we had committed to arithmetic that does not
      work. ***

STEP 2: rank by contribution to unavailability.
  recs      0.005   66%   <- and it is the least important
  catalogue 0.001   13%
  payments  0.001   13%
  auth      0.0005   7%
  db        0.0001   1%

STEP 3: reclassify.
  recs       REQUIRED -> DEGRADABLE. Cached popular items on
             failure. 3 days of work.
  payments   REQUIRED at checkout, NOT required for browsing.
             Split the SLO: the browse journey does not
             include it.
  catalogue  REQUIRED, but 60% of requests are served from
             cache, so its effective contribution is
             0.4 × 0.001 = 0.0004.

STEP 4: recompute.
  Browse journey:
    U ≈ 0.0005 (auth) + 0.0001 (db) + 0.0004 (catalogue, weighted)
      = 0.0010
    A = 99.90%   <- now exactly at the target, with no margin

  Checkout journey:
    U ≈ 0.0010 + 0.001 (payments) = 0.0020
    A = 99.80%
    -> Set a SEPARATE, lower SLO for checkout. It is a
       different journey with a different dependency set and
       pretending it is the same journey was part of the
       original error.

STEP 5: buy margin where it is cheapest.
  auth is 0.0005, half the remaining browse budget, and it is
  a single-region deployment. Making it multi-AZ takes it to
  ~99.99% for a week of work.
  New browse U ≈ 0.0001 + 0.0001 + 0.0004 = 0.0006 -> 99.94%.
  Now there is 0.04 points of margin against the 99.9% SLO.

Two findings worth stating from that. The original SLO was arithmetically impossible, so the team's failure to meet it was a planning error rather than an execution one, and that reframing matters for how the conversation goes. And splitting one SLO into per-journey SLOs was as valuable as any reliability work, because browse and checkout have genuinely different dependency sets and a single number for both was wrong for each.

Production evidence

Google Cloud's and Azure's composite SLA documentation both state that the combined SLA of services used together is the product of the individual SLAs, and provide worked examples. It is the serial rule as vendor guidance.

Google's SRE Workbook, chapter 2, covers SLOs for services with dependencies and makes the point that a service cannot be more available than the product of its critical dependencies.

AWS's well-architected reliability pillar documents the correlated-failure problem explicitly in its guidance on multi-AZ and multi-region design, including that a shared deployment pipeline undermines regional independence.

The 2021 Facebook BGP outage and the 2021 AWS us-east-1 control-plane events are the canonical illustrations of correlation: redundant infrastructure that failed together because it shared a control plane, and in Facebook's case because the recovery tooling depended on the network that was down.

Netflix's and Amazon's published work on graceful degradation documents the degradable-dependency classification in production, with Netflix's fallback hierarchy being the reference example.

The debate

The case for computing composite SLOs formally: it prevents committing to impossible targets, it ranks improvement work by actual contribution rather than by intuition, and it turns "should we make X more reliable" into arithmetic.

The case against: the model assumes binary up/down and independent failures, both of which are false, so the number is confidently wrong. Real availability is determined by correlated failures and partial degradation, neither of which the formula captures, and a precise wrong number is worse than an admitted uncertainty.

The case for measuring the journey instead: skip the composition entirely and instrument the end-to-end user journey. That is what the business cares about, it captures correlation and degradation automatically, and it does not require a model.

My position: use the arithmetic to find the ceiling and to rank the work, measure the journey to set the SLO.

The arithmetic's value is diagnostic rather than predictive. Computing that the ceiling is 99.24 percent against a 99.9 percent target tells you the commitment was impossible, which is a different conversation from "we keep missing our SLO", and it ranks the fixes: the recommendations dependency was 66 percent of unavailability and the least important service in the list. That ranking is worth having even though the absolute number is unreliable.

The approximation I would always use is that for small unavailabilities, total unavailability is the sum, because it makes the arithmetic doable in a meeting and it is accurate to three decimal places at these values. And it makes the design conclusion immediate: unavailability adds, so the worst dependency dominates and improving the best one is nearly worthless.

The correction I would insist on is the independence assumption. Two 99.9 percent providers in parallel is 99.9999 percent only if failures are independent, and at a 10 percent correlated fraction it is 99.99 percent, two orders of magnitude worse. So the engineering that matters is reducing correlation, not adding replicas, and the correlation people miss most often is a shared deployment pipeline, which looks like redundancy on the diagram and fails both regions on one bad rollout.

The move I would make before any of the reliability work is reclassifying dependencies as required, degradable or asynchronous. Most dependencies teams treat as required are degradable, and moving one off the critical path bought half a percentage point in days where a quarter of reliability engineering on the others would have bought a fraction of that.

And I would split by journey. Browse and checkout have different dependency sets and one SLO for both is wrong for each, which is a modelling error that looks like an execution problem.

Follow-up Q&A

"How do you compute a service's achievable availability?" Serial dependencies multiply, so all-required availability is the product. Parallel redundancy multiplies the unavailabilities. And for the small numbers involved, total unavailability is approximately the sum of the individual unavailabilities, which makes it doable in your head: five dependencies at 0.0005, 0.0001, 0.001, 0.005 and 0.001 sum to 0.0076, so the ceiling is 99.24 percent. That is accurate to three decimals against the exact product.

"What does that tell you to do?" Rank by contribution to unavailability, because it adds, so the worst dependency dominates. In that example the recommendations service at 99.5 percent was 66 percent of total unavailability and was also the least important service in the list. Improving the database from 99.99 to 99.999 would have bought 0.00009; making recommendations optional bought 0.005, fifty times more, for a few days of work.

"So the fix is a fallback?" Usually, and I would frame it as reclassification. Every dependency is required, degradable or asynchronous, and most that teams treat as required are degradable. The question for each is "what happens if this is down", and the answer should not be "the request fails" unless it genuinely must. Moving one dependency off the critical path is routinely worth more than a quarter of reliability engineering on the ones that stay.

"Two providers at 99.9 percent each. What's the combined availability?" 99.9999 percent if the failures are independent, and they are not. With a 10 percent correlated fraction, a shared network path or upstream or deploy pipeline, it is about 99.99 percent, two orders of magnitude worse. That sensitivity is the point: redundancy's benefit is dominated by the correlated fraction rather than by the individual availabilities, so the engineering that matters is reducing correlation, not adding a third replica.

"What correlations do people miss?" Shared availability zone, so power and cooling. Shared cloud provider, so control plane, IAM and DNS. Shared certificate authority, so an expiry hits both. Shared upstream. Correlated demand, so the spike hits both. And the one that catches people most: a shared deployment pipeline, which looks like redundancy on the architecture diagram and takes both regions out on one bad rollout.

"How do you set an SLO for a multi-service user journey?" Measure the journey directly rather than composing component SLOs, and give components their own SLOs for their own operational purposes. Composition is wrong for three reasons: failures are correlated, not every request touches every service, and a component's SLO measures its own requests rather than the journey's. Concretely, if 40 percent of searches skip the ranking service because they hit cache, its contribution is 0.4 times its unavailability, and that weighting is the correction most composite calculations omit.

"Where does the arithmetic mislead you?" Four places. It assumes binary up or down, and real services degrade, so a service at "99.9 percent available" may be serving most requests while being unusable for five percent of users. It assumes independence, which is the largest error. It ignores retries, which are cheap redundancy that changes the numbers substantially. And it ignores timeouts and fallbacks, which is the reframe: you are composing the availability of outcomes, and a dependency that fails in 50 milliseconds into a good fallback contributes almost nothing.

"Your SLO is 99.9 and you keep missing it at 99.31. What do you do first?" Compute the ceiling, because the likely finding is that the SLO was never achievable, which is a completely different conversation from "we keep failing to execute". In the case I worked the ceiling was 99.24 against a 99.9 target, so the team had committed to arithmetic that does not work. Then rank by contribution, reclassify dependencies, and split by journey, because browse and checkout had different dependency sets and one SLO for both was wrong for each.

Common misconceptions

"Adding a replica doubles reliability." Only if failures are independent. A 10 percent correlated fraction removes most of the benefit.

"The composite SLO is the sum of component SLOs." Failures correlate, not every request touches every component, and component SLOs measure their own traffic. Measure the journey.

"Improve the least reliable dependency." Correct, but the stronger move is usually to remove it from the critical path entirely.

"99.9 percent availability means it works 99.9 percent of the time for everyone." It can mean it works perfectly for 95 percent of users and not at all for 5 percent.

"Multi-region means independent." Not with a shared control plane, a shared deploy pipeline or a shared configuration source.

Interview delivery note

Give the two rules and the approximation that makes them usable in a meeting: "Serial dependencies multiply, parallel redundancy multiplies the unavailabilities. And for small numbers, total unavailability is approximately the sum, which is accurate to three decimals and means you can do it in your head."

Then use it to reach a conclusion rather than stating it abstractly: "So five dependencies at 99.95, 99.99, 99.9, 99.5 and 99.9 sum to 0.0076 unavailability, so the ceiling is 99.24 percent. If the SLO is 99.9, it was never achievable, which is a different conversation from 'we keep missing it'."

Then the ranking and the move it produces: "And because unavailability adds, the worst dependency dominates. Recommendations at 99.5 was two thirds of total unavailability and the least important service in the list. Making it degradable with a cached fallback bought half a percentage point in three days; improving the database from four nines to five would have bought 0.00009."

Volunteer the independence correction, because it is the largest error in the model: "and I'd be careful with parallel redundancy. Two providers at 99.9 is 99.9999 only if failures are independent. At a ten percent correlated fraction it's 99.99, two orders of magnitude worse. So the work that matters is reducing correlation, and the one people miss is a shared deploy pipeline, which looks like redundancy on the diagram and takes both regions out on one bad rollout."

Close on the modelling point: "and I'd split by journey. Browse and checkout had different dependency sets, and one SLO for both was wrong for each. That was a modelling error that looked like an execution problem."

The Staff & Team Lead Interview Reference