SLI selection and SLO targets

What it is

An SLI is a measurement of service health from the user's point of view. An SLO is a target for that measurement, derived from what users actually tolerate, not from what the system currently achieves and not from a round number.

SLI     "the proportion of checkout requests that returned a
         2xx or 4xx within 800ms, measured at the edge"

SLO     "99.5% of checkout requests over a rolling 28 days"

The two design questions, and both are usually answered badly:
  WHAT do you measure, and WHERE?
  WHAT TARGET, and derived from what?

What this is confused with: an SLI as a system metric. CPU utilisation, queue depth and cache hit rate are useful and they are not SLIs, because a user cannot experience them. An SLI must be something a user would notice if it got worse.

Also confused: an SLO as an aspiration. An SLO is a decision about how much unreliability you will tolerate in exchange for velocity, and it is only real if breaching it changes what the team does (see the error budget policy). A target that produces no consequence is a number on a dashboard.

The problem it solves

Measuring in the wrong place produces a green dashboard during an outage.

An SLI measured at the application server:
  "99.98% of requests returned 200"

What the user experienced that hour:
  - the CDN was serving 502s for 8 minutes for one region,
    which never reached the application
  - TLS handshakes were failing for a subset of clients after
    a certificate rotation
  - the JavaScript bundle 404'd, so the page rendered blank
    while every API call it never made stayed at 100%

Every one of those is invisible to a server-side success-rate
SLI, and all three are total outages from the user's seat.

The general rule: THE FURTHER FROM THE USER YOU MEASURE, THE
MORE FAILURE MODES YOU EXCLUDE BY CONSTRUCTION.

And picking a target without deriving it produces one of two failures:

TOO HIGH:  "99.99% because that sounds serious"
  = 4.4 minutes of downtime per month.
  Every deploy, every dependency blip and every certificate
  rotation consumes it. The team is permanently over budget,
  the policy is permanently suspended, and the SLO now means
  nothing.

TOO LOW:   "99.5%, we hit that easily"
  = 3.6 hours per month, and you will never notice a real
  degradation because the budget absorbs it. Users churn while
  the SLO stays green.

Both come from choosing a number first and reasoning afterwards.

Mechanics

Choosing what to measure

Start from what the user is trying to do, not from what the system emits.

For each critical user journey, ask: what does "working" mean?

  checkout      the order is placed, and it feels immediate
  search        results come back, and they are relevant
  data pipeline yesterday's numbers are there when I look
  streaming     the video plays without stalling

Then find the smallest set of measurements that would go bad if
any of those stopped being true.

The four SLI families, and which one fits:

AVAILABILITY / CORRECTNESS
  proportion of valid requests served successfully
  fits: request-response APIs, page loads

LATENCY
  proportion of valid requests served faster than a threshold
  fits: anything interactive
  NOTE: expressed as a PROPORTION UNDER A THRESHOLD, not as a
  percentile value. See below.

FRESHNESS / CORRECTNESS OF DATA
  proportion of records updated more recently than X, or
  proportion of outputs that match a reference
  fits: pipelines, caches, search indexes, replicas

THROUGHPUT / COVERAGE
  proportion of expected work actually processed
  fits: batch, streaming, anything with an input queue

Latency as a proportion rather than a percentile is the detail that matters most in practice:

BAD SLI:  "p99 latency < 800ms"
  - percentiles cannot be averaged or aggregated across
    instances or windows (see the percentiles page)
  - it produces no error budget: how much of "p99 was 830ms"
    have you spent?

GOOD SLI: "proportion of requests completing in under 800ms"
  - it is a ratio of good events to valid events, so it
    aggregates correctly by summing numerators and
    denominators
  - a 99.5% target directly yields an error budget: 0.5% of
    requests may be slow
  - the threshold is a product decision you can defend

Every SLI should be expressible as good events / valid events, because that form aggregates, it produces a budget, and it is comparable across services.

Where to measure: as close to the user as you can afford

Measurement point            Catches                     Misses
--------------------------------------------------------------------
Real user monitoring         everything the user          bot traffic
(client-side, RUM)           experiences: DNS, TLS,       needs care;
                             CDN, JS, render              client clock
                                                          skew

Edge / CDN logs              CDN failures, TLS, most      client-side
                             network issues               render and JS

Load balancer                app failures, LB failures    CDN, DNS, TLS
                                                          at the edge

Application server           app failures only            everything in
                                                          front of it

Downstream dependency        nothing user-facing          almost all of
metrics                                                   it

The practical answer for most services: measure at the load balancer or edge as the primary SLI, and add RUM for user-facing surfaces. Server-side is the fallback when you control nothing in front.

Synthetic probes are a complement, not a substitute:

Synthetics give you: coverage during low traffic, a consistent
  baseline, and detection of total outages where real traffic
  goes to zero (and a ratio of 0/0 is not a signal).

Synthetics miss: the real input distribution, the user's
  network, real client diversity, and anything driven by
  specific accounts or data.

Use both. A ratio SLI goes blind at zero traffic, which is
exactly when the outage is total, and that is the specific gap
a synthetic probe fills.

Defining "valid events" carefully

The denominator is where SLIs are quietly gamed, usually by accident.

Decisions that must be explicit and written down:

  Do 4xx responses count as failures?
    Usually NO for 400/404 (the client asked for something
    wrong) and YES for 429 (you refused to serve them) and
    YES for 401 caused by your own token service failing.
    "All 4xx are the client's fault" is how a broken auth
    service reports 100% availability.

  Do health checks and internal probes count?
    NO. They inflate the denominator with traffic that never
    fails and dilute real failures.

  Does bot traffic count?
    Usually no, and the filter must be stable, because a
    change to bot classification silently moves the SLI.

  What about requests the user abandoned?
    A client disconnect after 30 seconds is a failure from the
    user's perspective even though the server logged nothing.
    Count it if you can see it.

  Long-tail endpoints?
    A single SLI over all endpoints lets a high-volume healthy
    endpoint mask a low-volume broken one. Either scope the
    SLI to a journey or add per-journey SLIs.

The masking problem is worth stating as arithmetic:

/api/feed        10,000,000 req/day, 99.99% success
/api/checkout        50,000 req/day, 92% success

Combined SLI: (9,999,000 + 46,000) / 10,050,000 = 99.95%

The SLO is met. Checkout is broken for one user in twelve.

An SLI aggregated over a whole service is an average over
journeys with wildly different importance, and averages hide
exactly the case you care about.

Deriving the target from tolerance

The target answers: how much unreliability do users tolerate before it changes their behaviour? Four sources of evidence, in descending order of quality:

1. OBSERVED BEHAVIOUR CHANGE
   Look at historical periods of degradation and what users
   did. Did sessions drop? Did support tickets rise? Did
   conversion move?
   "In the two hours we were at 98%, checkout conversion fell
   9% and support volume tripled" is the strongest possible
   input, and most companies have this data and never look
   at it.

2. CURRENT PERFORMANCE AND COMPLAINT LEVEL
   If you have been at 99.7% for a year and nobody has
   complained, 99.9% is not obviously justified. If you have
   been at 99.7% and the account team raises it monthly, it is.

3. CONTRACTUAL AND COMPETITIVE FLOOR
   The SLA is the floor and the SLO must be strictly tighter
   (see SLA vs SLO vs SLI). What competitors publish sets an
   expectation independent of what users would tolerate.

4. THE DEPENDENCY CEILING
   You cannot durably exceed the composite availability of
   what you depend on. If three serial dependencies are each
   99.9%, your ceiling is about 99.7% before any of your own
   failures.

Then do the arithmetic before committing, because the numbers are less intuitive than they look:

Error budget per 28 days:

  99%      6h 43m       and 1 in 100 requests
  99.5%    3h 21m
  99.9%      40m        and 1 in 1,000 requests
  99.95%     20m
  99.99%      4m 2s     and 1 in 10,000 requests
  99.999%       24s

Read that as: at 99.99%, a single 5-minute incident consumes
more than a month's budget. If your deploy process can cause a
2-minute blip, you can afford two deploys a month.

This is the calculation that converts "four nines sounds
right" into a conversation about what you would have to stop
doing.

Set the target from the tolerance, then check it against the cost of achieving it, and if the two disagree, that is a business conversation rather than an engineering one.

The latency threshold, derived rather than picked

Do not pick 200ms because it is a round number. Derive it:

1. What is the user's task? An autocomplete has a different
   threshold from a monthly report.
2. What does the current distribution look like? Plot it. A
   threshold inside the body of the distribution produces a
   noisy SLI; a threshold on the shoulder is stable.
3. Where does behaviour change? If you have session data,
   find the latency at which abandonment rises.
4. Consider TWO thresholds rather than one:
     "99% of requests under 500ms" (the good experience)
     "99.9% of requests under 3s"  (the tolerable ceiling)
   Two thresholds catch two different failures: a general
   slowdown, and a heavy tail affecting a small group badly.

The two-threshold form is under-used and it is what catches the "most users are fine and 0.5 percent are timing out" pattern, which a single p99-under-500ms target will report as a small budget spend rather than as a group of users who cannot use the product at all.

The window and the review

ROLLING WINDOWS (28 days) rather than calendar months, because a
  calendar month resets the budget on the 1st and produces an
  incentive to ship recklessly on the 30th.
28 DAYS rather than 30, so the window always contains exactly
  four of each weekday and weekend traffic patterns do not
  shift the baseline.
REVIEW QUARTERLY. An SLO that has never been changed is
  usually not being used. If you have been at 99.99% against a
  99.9% target for a year, either tighten it or admit the
  target is not driving decisions.

A worked example: a green dashboard during an outage

A retail platform. Existing SLO: "99.9% availability", measured as non-5xx responses at the application servers, aggregated across all endpoints, on a calendar month.

The incident that started the redesign:

14:10  a CDN configuration change causes the JS bundle to
       return 404 for one region.
14:10  the page loads, renders a blank container, and makes no
       API calls.
14:52  a customer tweet is escalated internally.
15:05  fixed.

The SLO dashboard for that day: 99.99%.

Because: the application server saw FEWER requests, all of
which succeeded. The SLI improved during the outage.

"The SLI improved during the outage" is the clearest possible demonstration of a measurement-point problem, and it made the redesign uncontroversial.

The redesign, per journey rather than per service:

Four critical journeys identified with the product team:

  BROWSE     user can see products
  SEARCH     user can find products
  CHECKOUT   user can pay
  ACCOUNT    user can see orders

SLIs, all in good/valid form:

  BROWSE availability
    RUM: proportion of page views where the product grid
    rendered, measured client-side
    valid: page views excluding known bots, excluding
    user-initiated aborts under 1s

  CHECKOUT availability
    edge: proportion of POST /checkout returning 2xx or a
    4xx OTHER THAN 429, within 30s
    valid: all requests reaching the edge, excluding health
    checks

  CHECKOUT latency, TWO thresholds
    proportion under 800ms   (target 99%)
    proportion under 4s      (target 99.9%)

  SEARCH freshness
    proportion of catalogue updates visible in search within
    5 minutes

  ACCOUNT availability
    edge, same form as checkout

Deriving the checkout target from tolerance:

STEP 1: observed behaviour change.
  Pulled the last 18 months of degradation events and joined
  them to conversion.

    availability in the hour   conversion vs baseline
      99.9%+                     no measurable change
      99.5% - 99.9%              -1.2%
      99.0% - 99.5%              -7%
      < 99.0%                    -31%, and support volume 4x

  The knee is between 99.5% and 99.9%.

STEP 2: current performance and complaints.
  Trailing 12-month checkout availability: 99.86%.
  Account escalations mentioning checkout: 3 in 12 months, all
  during events below 99%.

STEP 3: contractual floor.
  The enterprise SLA promised 99.5% monthly with credits.
  The SLO must be tighter than the SLA, and by enough that
  breaching the SLO is a warning rather than a bill.

STEP 4: dependency ceiling.
  Checkout depends serially on auth (99.95%), inventory
  (99.9%) and the payment gateway (99.95% per its own SLA).
  Composite ceiling: 0.9995 x 0.999 x 0.9995 = 99.80%.

  THIS WAS THE FINDING. A 99.9% target was arithmetically
  unreachable while those dependencies were serial, regardless
  of the team's own reliability.

The dependency ceiling of 99.80 percent against a 99.9 percent target explained a year of unexplained budget exhaustion, and no amount of work on the checkout service itself could have fixed it.

What was decided:

CHECKOUT SLO: 99.8% over a rolling 28 days.

Justification, written into the SLO document:
  - the tolerance knee is between 99.5% and 99.9%; 99.8% sits
    above the level at which conversion measurably moves
  - it is tighter than the 99.5% SLA by 0.3 points, which is
    ~2 hours a month of warning before credits
  - it is at the current dependency ceiling, which makes the
    ceiling visible as a decision rather than a mystery

AND a companion decision, which is the point of doing the
arithmetic: to go above 99.8% requires removing a serial
dependency. Inventory was made non-blocking (checkout proceeds
optimistically and reconciles), raising the ceiling to 99.90%.
That work was scoped and funded because the SLO arithmetic made
it the only available lever.

The SLO calculation produced an architecture decision, which is the strongest argument for deriving targets rather than picking them.

Results after two quarters:

                              before          after
SLIs                          1 (service-     6 (per journey,
                              wide, server-   edge + RUM)
                              side)
outages invisible to the SLO  3 in prior year   0
budget exhausted              9 of 12 months   3 of 6 months
target justification          "it sounded      written, with the
                              right"           conversion data
error budget policy invoked   never (always    twice, both times
                              exhausted, so    acted on
                              ignored)

Going from "exhausted 9 months in 12" to "exhausted 3 in 6" was not a reliability improvement, it was a target that was achievable, which is what made the error budget policy start functioning at all.

One thing that was harder than expected:

Defining "valid events" for BROWSE took three attempts.

Attempt 1: all page views. Included bots, which were 34% of
  volume and never failed, diluting real failures by a third.
Attempt 2: excluded bots by user-agent. A bot-classification
  update in month two moved the SLI by 0.15 points with no
  change in reliability, which triggered a false investigation.
Attempt 3: excluded bots, AND pinned the classification rules
  as a versioned artifact reviewed alongside the SLO.

The lesson recorded: any filter in the denominator is a
dependency of the SLI, and changing it silently changes your
reliability history.

Production evidence

Google's SRE Book and the SRE Workbook specify the good-events-over-valid-events SLI form, the four SLI families (availability, latency, quality, freshness), the practice of measuring as close to the user as possible, and the argument that latency SLIs should be expressed as a proportion under a threshold rather than as a percentile value.

Google's published SLI menu by service type (request-driven, pipeline, storage) is the standard starting point for choosing which family applies, and the SRE Workbook's worked example explicitly warns about denominators that include health checks and internal traffic.

Rolling windows over calendar windows is documented SRE practice, with the stated reason that a calendar reset creates an incentive to spend the remaining budget before it expires and to be reckless immediately after a reset.

Composite availability arithmetic for serial dependencies is standard reliability engineering, and it is the basis for the dependency-ceiling check: a service cannot durably exceed the product of its serial dependencies' availabilities.

Real user monitoring as the measurement point for user-facing surfaces is standard practice in web performance (and is what Core Web Vitals field data is), and the class of failure it catches that server-side monitoring cannot, CDN, TLS, DNS and client-side rendering failures, is well documented.

The SLA-tighter-than-SLO relationship is universal published practice: the internal target must be strictly tighter than the contractual commitment so that breaching the internal target is a warning rather than a financial event.

The debate

Server-side or client-side SLIs? Client-side for anything user-facing, because the failures that server-side measurement excludes by construction (CDN, TLS, DNS, bundle, render) are total outages from the user's seat. The counter-argument is real: RUM data is noisier, requires bot filtering, depends on client clocks, and only reports from clients that successfully loaded enough JavaScript to report. The practical answer is edge measurement as the primary SLI with RUM as a second, and synthetics to cover the zero-traffic case.

Should 4xx count as failures? Not by default, and the blanket rule is dangerous. 429 is your refusal to serve, and a 401 caused by your own token service failing is your outage reported as the client's fault. Decide per status code, write it down, and revisit it when a new failure mode appears.

One SLO per service or per journey? Per journey. A service-wide SLI averages over journeys of wildly different importance, and the arithmetic is unforgiving: ten million healthy feed requests and fifty thousand checkout requests at 92 percent still produce a 99.95 percent aggregate. The cost is more SLOs to maintain, which is real and is the correct price.

Is 99.99% ever the right target? For infrastructure that many things depend on, sometimes. For a typical product service it is usually a decision nobody has costed: 4 minutes a month means a single 5-minute incident blows the budget, so the deploy process, the dependency set and the on-call response all have to be built for it. The honest test is whether you would fund what it requires.

Should the SLO be set to current performance? No, though current performance is an input. Setting the target at what you already achieve guarantees it never drives a decision, and setting it far above guarantees permanent exhaustion, after which the policy is suspended and the SLO is decoration. Derive from tolerance, sanity-check against the dependency ceiling, and accept a target you can actually defend.

Is deriving a target from conversion data over-engineering? It is usually a day of analysis on data the company already has, and it converts the target from a preference into a finding. When a target is challenged in a planning meeting, "the knee in conversion is between 99.5 and 99.9 percent" ends the conversation and "it felt right" does not.

Follow-up Q&A

"What makes a good SLI?"

It is something a user would notice if it got worse, it is measured as close to the user as you can afford, and it is expressed as good events over valid events so that it aggregates correctly and produces an error budget. CPU and queue depth are useful metrics and not SLIs, because a user cannot experience them. Latency in particular should be a proportion under a threshold rather than a percentile value, because percentiles cannot be aggregated across instances or windows and give you no way to say how much budget a slow period consumed.

"Why does measurement point matter so much?"

Because the further from the user you measure, the more failure modes you exclude by construction. A server-side success-rate SLI cannot see a CDN outage, a TLS failure after a certificate rotation, or a JavaScript bundle 404 that renders a blank page. In one case a CDN misconfiguration produced a blank page for a whole region and the SLI improved during the outage, because the application server received fewer requests and all of them succeeded. Measure at the edge as the primary SLI, add real user monitoring for user-facing surfaces, and keep synthetic probes for the case where real traffic goes to zero and a ratio becomes 0 over 0.

"How do you decide what counts as a valid event?"

Explicitly, in writing, because the denominator is where SLIs get gamed by accident. Exclude health checks and internal probes, which never fail and dilute real failures. Decide per status code rather than by class: 400 and 404 are usually the client's problem, 429 is your refusal to serve, and a 401 caused by your own token service failing is your outage. Filter bots, and pin the classification as a versioned artifact, because in one case a bot-classification update moved the SLI by 0.15 points with no change in reliability and triggered a false investigation. Any filter in the denominator is a dependency of the SLI.

"How do you derive an SLO target?"

Four inputs, in descending order of quality. Observed behaviour change: join historical degradation periods to conversion, session or support data and find the knee. Current performance against complaint level. The contractual floor, since the SLO must be strictly tighter than the SLA so that breaching it is a warning rather than a bill. And the dependency ceiling, because you cannot durably exceed the product of your serial dependencies' availabilities. Then do the budget arithmetic: 99.99 percent is four minutes a month, so a single five-minute incident exhausts it, which turns "four nines sounds right" into a conversation about what you would stop doing.

"What happens if the target is above the dependency ceiling?"

You exhaust the budget permanently for reasons no amount of work on your own service can fix. In one case checkout depended serially on auth at 99.95, inventory at 99.9 and a payment gateway at 99.95, for a composite ceiling of 99.80 percent against a 99.9 percent target, which explained a year of unexplained exhaustion. The useful outcome is that it makes the lever explicit: raising the target required removing a serial dependency, so inventory was made non-blocking with optimistic checkout and reconciliation, which raised the ceiling to 99.90. The SLO arithmetic produced an architecture decision.

"Why two latency thresholds instead of one?"

Because they catch different failures. "99 percent under 500 milliseconds" catches a general slowdown. "99.9 percent under 3 seconds" catches a heavy tail where a small group of users is timing out completely. With a single threshold, the pattern where most users are fine and half a percent cannot use the product reports as a small budget spend rather than as an outage for those users. The thresholds themselves should be derived from the shape of your latency distribution and, where you have the data, from where abandonment rises, rather than picked as round numbers.

Common misconceptions

"CPU and queue depth are SLIs." They are useful system metrics. An SLI is something a user would notice.

"p99 latency under 800ms is an SLI." Percentiles do not aggregate and produce no budget. Use the proportion of requests under the threshold.

"All 4xx are the client's fault." 429 is your refusal to serve, and a 401 from your own broken token service is your outage reported as theirs.

"One SLO per service is enough." It averages over journeys of different importance, so ten million healthy feed requests can hide a checkout that is failing for one user in twelve.

"Higher targets are safer." 99.99 percent is four minutes a month, so one incident exhausts it, the policy gets suspended, and the SLO stops meaning anything.

"Set the SLO where we are today." Then it will never drive a decision. Derive it from tolerance and check it against the dependency ceiling.

Interview delivery note

Say this verbatim: "An SLI has to be measured where the user is, because the further back you measure the more failure modes you exclude by construction. We had a CDN misconfiguration render a blank page for a whole region and the SLI improved during the outage, because the app servers saw fewer requests and all of them succeeded." One sentence of principle and one unforgettable example.

The senior-versus-staff separator is checking the target against the dependency ceiling. A senior engineer sets an SLO from historical performance. A staff engineer computes the composite availability of the serial dependencies, finds the ceiling is 99.80 percent against a 99.9 percent target, and concludes that a year of budget exhaustion had an arithmetic cause no amount of work on the service could fix. Then the SLO produces an architecture decision, making one dependency non-blocking, which is the target becoming a design constraint rather than a dashboard.

The second signal is deriving the target from observed behaviour change. Saying "we joined 18 months of degradation events to conversion and the knee is between 99.5 and 99.9 percent, so we set 99.8" converts the target from a preference into a finding, and it is a day of analysis on data most companies already have and never look at.

The Staff & Team Lead Interview Reference