The error budget policy as a signed contract

What it is

A document agreed in advance stating what happens when the error budget is exhausted, who decides, and who can override it. Not the budget, which is arithmetic; the policy, which is a commitment.

SLO              99.9% over 28 rolling days
ERROR BUDGET     0.1% = 43 minutes of allowed unavailability
                 This is arithmetic. Nobody disagrees with it.

ERROR BUDGET     "When the budget is exhausted, feature
POLICY           deploys stop and the team works reliability
                 until it recovers above 20%."
                 This is a COMMITMENT, and it is the part
                 that either exists or does not.

Commonly confused with the error budget itself. A budget without a policy is a report. It tells you how you are doing and changes nothing, and the overwhelming majority of SLOs in production are exactly that.

Also commonly confused with a process document. The distinguishing feature is that it is signed by someone who can be overruled and by someone who could overrule them, which is what makes it a contract rather than a team preference.

The problem it solves

Reliability and velocity compete for the same capacity, and without a pre-agreed rule the argument is won by whoever has more organisational power at the moment it happens.

Without a policy:
  Engineering: "We should stop shipping and fix reliability."
  Product:     "We have a launch in three weeks."
  Outcome:     features ship, reliability work does not, and
               the argument repeats every quarter with the
               same result.

With a signed policy:
  Engineering: "The budget is exhausted. Per the policy we
                agreed in January, feature deploys pause."
  Product:     "We have a launch in three weeks."
  Engineering: "The policy has an override: a VP can sign off
                and we document the risk. Do you want to
                escalate it?"
  Outcome:     a decision gets made by someone with the
               authority to make it, on the record.

The crucial property: the policy is agreed when nobody is in an incident and nobody has a launch next week. A policy negotiated during an outage is not a policy; it is a negotiation conducted from the weaker position.

Mechanics

What the document contains

1. THE SLO AND SLI, stated precisely.
   The exact SLI definition, the target, the window.

2. THE THRESHOLDS AND ACTIONS.
   What happens at each level of remaining budget. Specific
   enough that nobody has to interpret it.

3. WHO DECIDES.
   Which role declares the budget exhausted, which role can
   override, and what the override requires.

4. THE EXIT CONDITION.
   What ends the freeze. "Budget recovers above X" or "the
   contributing causes have a fix merged", stated explicitly,
   because "when things are better" produces an argument.

5. EXCLUSIONS.
   What does not consume budget: agreed maintenance windows,
   failures of a documented third party, a load test.
   Negotiated in advance, never during.

6. SIGNATURES.
   Engineering lead and product lead at minimum, and the
   person who can override.

The thresholds, and why they are graduated

A single cliff at zero is the version that fails, because nothing happens until it is too late and then the response is drastic.

BUDGET REMAINING     ACTION
-------------------------------------------------------------
100% to 50%          Normal operations. Ship.
                     Optionally: this is a signal you may be
                     UNDER-shipping, and could take more risk.

50% to 25%           Review at the weekly. No change to
                     shipping, but reliability work is
                     prioritised into the next sprint and the
                     trend is reported.

25% to 10%           Feature work continues; every deploy
                     requires an explicit risk assessment and
                     a rollback plan reviewed by a second
                     person. Reliability items move to the top
                     of the backlog.

10% to 0%            FEATURE FREEZE. Only reliability fixes,
                     security patches and rollbacks deploy.
                     A named owner produces a recovery plan
                     within 48 hours.

BELOW 0%             Freeze continues. A written review with
                     the causes and the plan goes to the
                     skip-level. The freeze does not lift on
                     the calendar; it lifts on the exit
                     condition.

The 100-to-50 row is the one people omit and it is worth including deliberately. An SLO that is never close to being spent means the service is over-invested in reliability relative to its target, and that investment came out of feature work. The policy should be able to say "ship faster" as well as "stop".

Who can override, and what it costs

A policy with no override is ignored the first time it is inconvenient. A policy with a frictionless override is decoration. The design is deliberate friction.

OVERRIDE REQUIRES
  - A named person at VP level or above, not the product
    manager who wants the launch.
  - A written statement of what is shipping and why it cannot
    wait.
  - An explicit acknowledgement of the risk, in writing.
  - A time bound: the override covers this release, not the
    quarter.
  - Distribution to the same people who signed the policy.

WHAT IT DOES NOT REQUIRE
  - Engineering agreeing. The override exists precisely for
    the case where the business decides the risk is worth it,
    and that is a legitimate decision that someone
    accountable should be able to make.

That last point matters and is often mishandled. The policy's job is not to give engineering a veto. It is to ensure the decision is made deliberately, by someone accountable, on the record. An override that is used twice a year is a working policy; one used every month means the SLO is wrong and should be renegotiated rather than routinely overridden.

Rolling window, not calendar month

CALENDAR MONTH
  A bad incident on the 2nd means 28 days of freeze.
  The budget resets on the 1st, so the last week of a bad
  month has no budget and the first week of the next has all
  of it, which creates an incentive to delay risky deploys
  until the reset.

ROLLING 28 DAYS
  The budget recovers continuously as bad minutes age out.
  A freeze naturally lifts as the incident falls out of the
  window, which matches the intuition that a problem four
  weeks ago is less relevant than one yesterday.
  *** Rolling is correct and is what the tooling should
      compute. ***

Measuring burn, not just remaining

Remaining budget tells you where you are; burn rate tells you where you are going.

BURN RATE = (rate of budget consumption) / (rate that would
             exactly exhaust it over the window)

Burn rate 1   = on track to exactly exhaust the budget at the
                end of the window
Burn rate 14.4 = the budget is gone in 2 hours

The standard multi-window alerting:
  14.4x over 1 hour   (and 5 min)  -> page. 2% of budget in 1h.
   6x over 6 hours    (and 30 min) -> page. 5% of budget in 6h.
   1x over 3 days     (and 6 h)    -> ticket. Slow burn.

The short second window in each pair is the reset condition: it stops the alert firing for hours after the problem has passed. See burn-rate alerting.

The connection to the policy: burn rate is what triggers the graduated thresholds early enough to act. A team that only looks at remaining budget discovers the problem at 10 percent, whereas burn rate flags it while there is still room to respond.

A worked example: writing one that survives

CONTEXT
  Checkout API. 99.9% SLO, 43 minutes per 28 rolling days.
  Two failed attempts at a policy already: one was written by
  engineering alone and ignored, one had a freeze with no
  exit condition and was overridden within a week.

WHAT MADE THE THIRD ATTEMPT WORK

1. Product co-wrote it, and the arithmetic was shared first.
   Before proposing thresholds, present: "we spent 31 of 43
   minutes last month, 68% of which came from deploy-related
   incidents." Product agreeing with the diagnosis precedes
   product agreeing with the remedy.

2. The freeze was scoped narrowly.
   Not "no deploys": no FEATURE deploys. Reliability fixes,
   security patches, rollbacks, config changes and
   experiments behind flags all continue. A freeze that stops
   everything is one that gets overridden, because stopping
   everything is obviously wrong.

3. The exit condition was explicit and measurable.
   "The freeze lifts when the rolling budget recovers above
   20% AND the top contributing cause has a merged fix."
   Not "when things are better".

4. The override was real and had a named holder.
   The VP of Engineering, in writing, time-bounded to one
   release, distributed to the signatories. Nobody had to
   pretend the business could never decide to ship anyway.

5. It included the 'ship faster' clause.
   "If the budget is above 50% remaining for two consecutive
   windows, the team should be taking more deployment risk
   or the SLO should be tightened."
   *** This is what made product sign it. *** The policy was
   not one-directional; it committed engineering to the
   symmetric obligation.

6. A review date.
   Six months, or after any override, whichever is sooner.

OUTCOME OVER THE FOLLOWING YEAR
  Freeze triggered twice. Overridden once, with a written
  risk acceptance, and the release in question caused an
  incident, which made the next override conversation
  substantially shorter.
  The SLO was tightened once, from 99.9 to 99.93, because the
  budget was consistently under-spent.

Point 5 is the one worth extracting. A policy that only ever constrains product will be resisted by product. Making it symmetric, so a consistently unspent budget obliges engineering to take more risk or tighten the SLO, is what converts it from a veto into a shared instrument.

Production evidence

Google's Site Reliability Engineering book, chapter 3, introduces the error budget as the mechanism for resolving the reliability-versus-velocity tension, and is explicit that the budget's purpose is to make the trade-off explicit rather than to maximise reliability.

The SRE Workbook, chapter 5, is dedicated to error budget policies and contains sample policies, the argument for graduated thresholds, and the requirement for executive sign-off. Its stated position is that a policy without leadership agreement will not survive contact with a launch.

Google's own published sample policy includes an explicit escalation path and the condition that the policy is reviewed after each invocation, which is where the review-date practice comes from.

Multi-window multi-burn-rate alerting (SRE Workbook, chapter 5) is the source of the 14.4x/6x/1x thresholds and of the short-window reset condition, and it is implemented in Nobl9, Datadog, Grafana and Sloth.

Nobl9's and Datadog's SLO products both implement rolling windows rather than calendar months as the default, which reflects the operational finding that calendar resets create perverse incentives.

The debate

The case for a strict policy with automatic freezes: without a pre-agreed consequence, the budget is a report and nothing changes. The freeze is what makes reliability work happen, and automation removes the negotiation from the moment when engineering has the least leverage.

The case against freezes: they are blunt. A team frozen out of feature work may have reliability problems that feature work would not worsen, and a long freeze demoralises people and delays commitments to customers who had nothing to do with the incident. And a freeze punishes the team rather than fixing the system.

The case for advisory budgets: report the number, let the team use judgement, avoid the politics. Simple, and it produces the report-only outcome that the whole mechanism exists to avoid.

My position: graduated thresholds, a narrowly scoped freeze, an explicit exit condition, a real override at VP level, and a symmetric clause that obliges tightening the SLO when the budget is chronically unspent.

The property that determines whether it works is that it is signed before it is needed. A policy agreed during an incident or while a launch is three weeks away is a negotiation conducted from the weaker position, and it will produce whatever the more powerful party wants. Agreeing it in a quiet month is the entire mechanism.

The scoping is what stops it being overridden. A freeze that stops all deploys is obviously wrong and everyone knows it, so it gets overridden and the policy loses authority. Freezing feature deploys while explicitly permitting reliability fixes, security patches, rollbacks and flag-gated work is defensible, which means it survives.

The symmetric clause is the part I would not omit, and it is the least common. A policy that only ever constrains product gets resisted by product, reasonably. Committing that a budget consistently above 50 percent means either taking more deployment risk or tightening the SLO makes it a shared instrument rather than an engineering veto, and in the case above it is what got it signed.

And the override has to be real. A policy with no override is ignored the first time it is inconvenient, and a policy where engineering holds a veto is not a policy, it is a power struggle with a document. The business is entitled to decide the risk is worth it; the policy's job is to ensure that decision is made deliberately, by someone accountable, in writing. Two overrides a year is a working policy; monthly overrides mean the SLO is wrong and should be renegotiated.

Where I would push back on the framing: a freeze is a symptom, not a remedy. If it triggers repeatedly, the useful response is not a longer freeze, it is asking whether the SLO is right, whether the architecture can meet it, and whether the deploy process is the actual cause, which in the worked example it was.

Follow-up Q&A

"What is an error budget policy and why does it need to be signed?" The budget is arithmetic that nobody disagrees with; the policy is what happens when it is exhausted, and that is a commitment. It needs signing because reliability and velocity compete for the same capacity, and without a pre-agreed rule the argument is won by whoever has more organisational power at the moment it happens. Signing it in a quiet month, rather than during an incident or three weeks before a launch, is the entire mechanism.

"Why graduated thresholds rather than a freeze at zero?" Because a single cliff means nothing happens until it is too late and then the response is drastic. Graduated thresholds give escalating responses: at 50 percent it is a review, at 25 percent every deploy needs an explicit risk assessment and reliability work moves to the top of the backlog, at 10 percent feature deploys freeze. That gives the team room to respond before the drastic step, and it makes the trend actionable rather than only the endpoint.

"What does the freeze actually stop?" Feature deploys, and nothing else. Reliability fixes, security patches, rollbacks, configuration changes and flag-gated work all continue. Scoping matters because a freeze that stops all deploys is obviously wrong, everyone knows it, so it gets overridden and the policy loses its authority. A narrowly scoped freeze is defensible, which is what makes it survive.

"Who can override, and should engineering be able to block it?" A named person at VP level or above, in writing, with an explicit risk acknowledgement, time-bounded to one release, and distributed to everyone who signed the policy. And no, engineering should not hold a veto. The business is entitled to decide the risk is worth it, and the policy's job is to make that decision deliberate, accountable and on the record, not to prevent it. A policy with no override gets ignored the first time it is inconvenient.

"How often should the override be used?" Twice a year is a working policy. Monthly means the SLO is wrong and should be renegotiated rather than routinely overridden, because an SLO that is overridden every month is not a commitment, it is a formality. That frequency is itself a useful signal and worth putting in the review.

"What ends a freeze?" An explicit, measurable exit condition agreed in advance. "The budget recovers above 20 percent and the top contributing cause has a merged fix" works; "when things are better" produces an argument at exactly the moment when nobody wants one. And the freeze should not lift on the calendar, because a rolling window means the budget recovers continuously as bad minutes age out, which is the correct behaviour.

"Rolling window or calendar month?" Rolling. A calendar month means a bad incident on the 2nd freezes you for 28 days and the budget resets on the 1st regardless, which creates an incentive to delay risky deploys until the reset. A rolling window recovers continuously as old bad minutes fall out, which matches the intuition that an incident four weeks ago is less relevant than one yesterday.

"What makes product sign this?" Two things. Co-writing it, starting from shared arithmetic rather than a proposed remedy: "we spent 31 of 43 minutes last month and 68 percent came from deploy-related incidents" gets agreement on the diagnosis before the remedy. And a symmetric clause: if the budget stays above 50 percent for two consecutive windows, the team should take more deployment risk or the SLO should be tightened. A policy that only ever constrains product gets resisted by product, reasonably, and making it bidirectional is what turns it from an engineering veto into a shared instrument.

"The freeze keeps triggering. Now what?" Treat it as a symptom rather than applying more of the remedy. Three questions: is the SLO right for what users actually notice, can the architecture meet it at all (which is the dependency-ceiling calculation, and the answer is sometimes no), and is the deploy process the actual cause. In the case I worked it was the third: 68 percent of budget consumption traced to deploy-related incidents, so the fix was automated canary analysis rather than a longer freeze.

Common misconceptions

"The error budget is the policy." The budget is arithmetic. The policy is the commitment about what happens, and it is the part that usually does not exist.

"A good SLO means never spending the budget." A chronically unspent budget means over-investment in reliability paid for out of feature work, and the policy should be able to say so.

"The policy gives engineering a veto." It makes the decision deliberate and accountable. The override is a feature, not a loophole.

"A freeze stops all deploys." Scope it to feature deploys, or it is obviously wrong and gets overridden.

"Calendar months are fine for the window." They create an incentive to delay risky deploys until the reset. Use a rolling window.

Interview delivery note

Separate the budget from the policy in the first sentence, because that distinction is the whole answer: "The budget is arithmetic and nobody argues with it. The policy is what happens when it's exhausted, and that's a commitment. Most SLOs in production have the first and not the second, which makes them reports."

Name the property that makes it work: "And the thing that determines whether it works is that it's signed before it's needed. A policy negotiated during an incident, or three weeks before a launch, is a negotiation from the weaker position and it produces whatever the more powerful party wants."

Give the graduation and the scoping together, because both are about surviving contact: "Graduated thresholds rather than a cliff at zero, so at 50 percent it's a review, at 25 percent every deploy needs a risk assessment, and at 10 percent feature deploys freeze. And the freeze is scoped to feature deploys only, with reliability fixes, security patches, rollbacks and flag-gated work continuing, because a freeze that stops everything is obviously wrong and gets overridden."

The two lines that show you have actually made one of these stick: "The override has to be real, at VP level, in writing, time-bounded. Engineering shouldn't have a veto, because the business is entitled to decide the risk is worth it and the policy's job is to make that deliberate and on the record." And: "and I'd include a clause that goes the other way: if the budget stays above fifty percent for two windows, take more deployment risk or tighten the SLO. A policy that only ever constrains product gets resisted by product, and that clause is what got ours signed."

Close on the diagnostic: "and if the freeze keeps triggering, that's a symptom. The answer isn't a longer freeze, it's asking whether the SLO is right, whether the architecture can meet it at all, and whether the deploy process is the cause. In our case sixty-eight percent of budget spend traced to deploy-related incidents, so the fix was automated canary analysis."

The Staff & Team Lead Interview Reference