Burn-rate alerting

What it is

Burn-rate alerting fires on how fast you are consuming your error budget rather than on a raw error rate. Burn rate is a dimensionless multiplier: a burn rate of 1 means you are consuming budget at exactly the pace that exhausts it at the end of the compliance window, and a burn rate of 14.4 means you will exhaust a 30-day budget in about two days.

$$\text{burn rate} = \frac{\text{observed error ratio}}{1 - \text{SLO target}}$$

For a 99.9 percent SLO, the budget is 0.1 percent, so a sustained 1.44 percent error rate is a burn rate of 14.4.

The mature form is multi-window, multi-burn-rate: each alert has a long window that establishes significance and a short window (conventionally one twelfth of the long window) that confirms the problem is still happening. Both must be above the threshold for the alert to fire.

This is commonly confused with "alert when the error budget is exhausted". That alert is useless, because it fires after the damage. Burn-rate alerting fires while there is still budget left to protect.

The problem it solves

Static threshold alerting on error rate has two failure modes and you can only tune away one at a time.

Set the threshold low, say "page if error rate exceeds 0.5 percent for 5 minutes", and a brief blip during a deploy pages someone at 3am for an event that consumed 0.03 percent of the month's budget. That is the false-page problem, and its consequence is not annoyance, it is that the team stops reading pages.

Set it high, say 5 percent, and a slow burn at 0.4 percent runs for a week, never trips the alert, and quietly consumes the entire month's budget. That is the missed-degradation problem.

The two failures have the same root: error rate alone contains no information about impact. A 50 percent error rate for 30 seconds and a 0.3 percent error rate for two weeks are very different events that a rate threshold cannot distinguish. Burn rate multiplied by duration is exactly budget consumed, so alerting on burn rate over a window is alerting on impact.

Mechanics

The numbers to have memorised

A 30-day compliance window is 43,200 minutes.

SLO	Budget	Per 30 days	Per week
99%	1%	7h 12m	1h 41m
99.9%	0.1%	43m 12s	10m 5s
99.95%	0.05%	21m 36s	5m 2s
99.99%	0.01%	4m 19s	1m

The 99.9 percent and 99.99 percent rows are the two worth knowing cold, because they are the two numbers most often quoted in a room. Four and a half minutes a month is the sentence that ends most casual requests for four nines.

The standard alert set

From Google's SRE Workbook, and now effectively the industry default:

Severity	Burn rate	Long window	Short window	Budget consumed when it fires
Page	14.4	1 hour	5 minutes	2%
Page	6	6 hours	30 minutes	5%
Ticket	3	1 day	2 hours	10%
Ticket	1	3 days	6 hours	10%

The budget-consumed column is the derivation, and it is worth being able to do live: burn rate 14.4 sustained over 1 hour out of a 720-hour window consumes $14.4 \times 1/720 = 2%$ of the budget. Burn rate 6 over 6 hours consumes $6 \times 6/720 = 5%$. Burn rate 1 over 3 days consumes $1 \times 72/720 = 10%$.

The design intent: a fast, severe outage trips the first rule within minutes, before it has eaten much budget. A moderate degradation that a human would otherwise argue about trips the second rule within hours. A slow leak that would never trip a rate threshold trips the third or fourth within days, as a ticket rather than a page, because nobody should be woken for something that has been happening for three days.

Why two windows

The long window answers "is this significant". The short window answers "is this still happening".

Without the short window, an alert based on a 6-hour window stays firing for up to 6 hours after the incident is resolved, because the window still contains the bad data. On-call gets a page for an already-fixed problem, and worse, learns to ignore the alert during recovery. The short window resolves within minutes of the errors stopping, so the alert clears promptly.

Without the long window, you are back to a 5-minute threshold with all its noise. The long window is the significance filter, the short window is the recency filter, and the conventional ratio of 1/12 comes from the workbook.

In PromQL

# Recording rules: compute the error ratio once, per window, per service.
# Doing this as recording rules matters: the alert expression below evaluates
# four ratios, and computing them inline on every evaluation is expensive.
- record: job:slo_errors:ratio_rate5m
  expr: |
    sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
      / sum(rate(http_requests_total{job="checkout"}[5m]))
# ... and the same for 30m, 1h, 2h, 6h, 1d, 3d.

# The fast-burn page. Both windows must exceed 14.4 x (1 - 0.999) = 0.0144.
- alert: CheckoutSLOFastBurn
  expr: |
    job:slo_errors:ratio_rate1h  > (14.4 * 0.001)
      and
    job:slo_errors:ratio_rate5m  > (14.4 * 0.001)
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "Checkout burning error budget at >14.4x; 2% of the monthly budget
              consumed in the last hour."

# The slow-burn ticket. Same shape, gentler threshold, longer windows.
- alert: CheckoutSLOSlowBurn
  expr: |
    job:slo_errors:ratio_rate3d > (1 * 0.001)
      and
    job:slo_errors:ratio_rate6h > (1 * 0.001)
  for: 15m
  labels:
    severity: ticket

Note that the SLI is a ratio of good events to total events, measured as close to the user as you can get. Measuring at the application gives you a number that stays beautiful while the load balancer returns 503s, which is precisely the outage your customers experience and your dashboard does not.

A worked example

A checkout service. 99.9 percent availability SLO on a 30-day window, 2,000 requests per second, so about 5.2 billion requests per month and a budget of about 5.2 million failed requests, or 43 minutes of total unavailability.

Scenario A: a bad deploy. Error rate jumps to 40 percent for 8 minutes.

Burn rate is $0.40 / 0.001 = 400$. The 5-minute window crosses 0.0144 within about a minute of the deploy; the 1-hour window crosses it after roughly $0.0144 \times 60 / 0.40 \approx 2.2$ minutes of sustained errors. The page fires around the 3-minute mark. Budget consumed by the time someone acknowledges: $400 \times 3/43200 \approx 2.8%$. Total consumed by the 8-minute mark: about 7.4 percent, or roughly 3 minutes of the 43-minute monthly allowance. Automated rollback would have caught it faster, which is the point of connecting this to canary analysis.

Scenario B: a slow leak. A downstream dependency starts failing 0.25 percent of calls after a config change. Burn rate is 2.5.

The fast-burn rule never fires: 0.0025 is well under the 0.0144 threshold. The 6-hour, burn-rate-6 rule never fires either. The 1-day, burn-rate-3 rule does not fire. The 3-day, burn-rate-1 rule fires after the 6-hour short window and the 3-day long window both exceed 0.001, so within about 6 hours of onset, as a ticket. By then, budget consumed is $2.5 \times 6/720 \approx 2%$. Left unaddressed for the full month it would consume 250 percent of the budget, so the ticket has caught it with 98 percent of the budget intact and without waking anyone. That is the entire argument for the multi-rate set in one example.

Scenario C: the argument this enables. Three weeks in, 85 percent of the budget is gone. The error budget policy, signed by the director before any of this happened, says that below 20 percent remaining, feature releases pause and the team works reliability until the budget recovers. This is not a negotiation in the moment; it is a pre-committed rule being applied. The policy only works because it was agreed while the budget was healthy, which is the sentence to say out loud in an interview.

Production evidence

The multi-window, multi-burn-rate design and the specific 14.4 / 6 / 1 table come from Google's Site Reliability Workbook, chapter 5, "Alerting on SLOs", which walks through six successively better alerting strategies and lands on this one. That chapter is the primary source and it is short enough to read in an evening.

The pattern is implemented by Sloth and Pyrra, two open-source generators that turn an SLO definition into the full set of Prometheus recording and alerting rules, and by OpenSLO as a vendor-neutral specification. Grafana Cloud, Datadog, Nobl9 and Dynatrace all ship burn-rate alerting as a product feature. The convergence across independent implementations is the strongest argument that this is settled practice rather than one company's preference.

Google's public SLA documentation for its own cloud products, and AWS's, are worth reading alongside this for the contractual half: they publish availability commitments with credit schedules, measurement windows and exclusions, which is what the SLA layer turns these engineering numbers into.

The debate

The alternative is symptom-based threshold alerting: page when the thing a user notices crosses a line, without reference to a budget. It has real advantages. It is simpler to explain, it needs no SLO definition exercise, and for a system with no meaningful budget (a batch pipeline, an internal tool) an error budget is ceremony without payoff.

A second alternative is anomaly detection: alert when the error rate deviates from its learned baseline. This catches things a fixed threshold misses, particularly on metrics with strong seasonality, and it fails in the way all unsupervised methods fail: it cannot tell you whether the anomaly matters, it drifts as the baseline absorbs a chronic problem, and it is hard to reason about during an incident.

My position: burn-rate alerting for anything with a user-facing availability or latency SLO, because it is the only scheme that ties paging to user impact and gives you a principled reason not to page. Keep a small number of symptom-based pages alongside it for conditions that are catastrophic regardless of budget: total loss of a region, a queue with unbounded growth, a certificate about to expire. Those are not budget-consumption events, and trying to express them as one is contortion.

Burn-rate alerting is the wrong tool when the SLI is not a ratio of good to total events, when traffic is too low for the ratio to be statistically meaningful (at 5 requests per minute, one failure is a 20 percent error rate and your alert is noise), and when nobody has agreed to an error budget policy. That last one is the important failure: without the policy, the budget is a number nobody acts on, and you have built a dashboard rather than a control.

Follow-up Q&A

"Design burn-rate alerts for a 99.9 percent SLO. Why two windows?" Four rules: page at burn rate 14.4 over 1 hour with a 5-minute short window (2 percent of budget), page at 6 over 6 hours with a 30-minute short window (5 percent), ticket at 3 over 1 day with a 2-hour short window, ticket at 1 over 3 days with a 6-hour short window. Two windows because the long one establishes that the event is significant and the short one establishes that it is still happening, which kills both false pages on brief blips and stale pages that keep firing for hours after recovery.

"Your service does 5 requests per minute. How does this change?" It breaks. One failed request in a 5-minute window is a 20 percent error rate and a burn rate of 200 against a 99.9 percent SLO. Options: lengthen the windows so the denominator is large enough to be meaningful, switch to a count-based rather than ratio-based SLI ("no more than N failures per week"), aggregate several low-traffic services into one SLO if they share a user journey, or use synthetic probes to manufacture a denominator. Naming the low-traffic problem unprompted is a good signal, because it is the most common real-world reason a burn-rate rollout stalls.

"How do you set the SLO target in the first place?" From user tolerance and business need, never from aspiration and never from current performance rounded up. Work backwards: what fraction of failures produces a support ticket or a churn event, what does the product need to promise, and what do your dependencies structurally allow. Then check the last quarter's actual performance: if you are already at 99.95 percent, setting the SLO at 99.9 percent gives you room to move; setting it at 99.99 percent means you start in violation and the budget is meaningless from day one.

"Three services each at 99.9 percent, called in series. What is your availability?" $0.999^3 \approx 0.997$, so about 99.7 percent, which is 2 hours 10 minutes a month rather than 43 minutes. You cannot be more available than the product of your hard dependencies. The fixes are to remove the serial dependency (cache, make it optional, degrade gracefully), add redundancy so the dependency is not a single point, or negotiate your own SLO down to something structurally achievable. Walking that arithmetic is one of the more impressive things you can do in a reliability round.

"What do you do when the budget is exhausted?" Whatever the error budget policy says, which was written and signed before it happened. The standard shape: freeze feature releases, redirect the team to reliability work, and require an explicit exec-level exception to ship anything not related to reliability. The policy is the artifact that makes error budgets real; without it the budget is a metric, and metrics without consequences get ignored.

Common misconceptions

The most common is that burn-rate alerting replaces all other alerting. It replaces threshold alerting on user-facing symptom metrics. You still need alerts on saturation approaching a hard limit (disk, connection pool, quota), on conditions that are catastrophic irrespective of budget, and on the pipeline that produces the SLI itself, because a broken metrics pipeline makes your SLO look perfect.

The second is that you can average percentiles to compute a latency SLI across instances. You cannot; percentiles are not linear, and the average of ten instances' p99 values corresponds to nothing. Merge the underlying histograms instead (HDR histograms, Prometheus native histograms, t-digest). For a latency SLO the cleaner formulation avoids the issue entirely: count the fraction of requests faster than a threshold, which is a ratio of good events to total events and composes correctly.

The third is that the 14.4 is magic. It is $2% \times 720 / 1$, chosen so that a 1-hour window corresponds to 2 percent of a 30-day budget. Change the window or the budget fraction you are willing to spend before paging and the number changes. Being able to re-derive it is much better than remembering it.

Interview delivery note

Say this: "I alert on burn rate, not error rate, because burn rate times duration is budget consumed, which is impact. The standard set is multi-window, multi-burn-rate: page at 14.4 times over an hour with a five-minute short window, which is 2 percent of a monthly budget; page at 6 times over six hours, which is 5 percent; ticket at 1 times over three days. Two windows because the long one proves significance and the short one proves it is still happening, so the alert clears when the incident does."

The depth signal is deriving the 2 percent from 14.4 rather than reciting the table, and then immediately naming the error budget policy: the alerting only matters if leadership pre-committed to what happens when the budget runs out. Candidates who have only implemented this describe the rules. Candidates who have run it describe the conversation with the director that had to happen first.

The Staff & Team Lead Interview Reference