On-call design: rotation, page budget, interrupt shield, toil cap

What it is

On-call design is the set of explicit numbers that make production responsibility sustainable rather than attritional. Four of them, and each has a defensible target:

ROTATION SIZE     how often each person is on call.
                  1 week in N. N >= 6 is the sustainable floor for
                  a 24/7 rotation.

PAGE BUDGET       how many pages a shift may produce before the
                  rotation is declared broken. Google's SRE
                  guidance: no more than 2 events per 12-hour
                  shift, so there is time to investigate properly.

INTERRUPT SHIELD  a separate, rotating role that absorbs
                  non-urgent interrupts so the rest of the team
                  has uninterrupted time. Distinct from on-call.

TOIL CAP          a ceiling on operational work as a fraction of
                  the team's time, with a defined overflow
                  mechanism when it is exceeded. Google's number
                  is 50 percent.

What this is confused with: an on-call schedule. A schedule says who is responsible. A design says how much responsibility that is, what happens when it exceeds the budget, and who pays for reducing it. Most organisations have a schedule and no design, which is why the load only ever goes up.

Also confused: on-call and support. On-call is production incidents at any hour. Support is questions, requests and small fixes during working hours. Putting both on one person means the incident responder is also the person answering Slack, and neither job gets done.

The problem it solves

A rotation with no budget degrades monotonically, because every party's incentive points the same way.

A new alert is added:      free, to the person adding it
An alert that fires often
  but is rarely actionable: nobody owns deleting it
A service is onboarded to
  the rotation:            free, to the team handing it over
Fixing the underlying
  cause:                   costs a sprint, competes with features

So pages accumulate and nothing removes them.

And the cost is paid in a currency nobody puts on a dashboard:

A 4-person rotation, 24/7, at 11 night pages per week.

Each person is on call 13 weeks a year.
Each on-call week: ~2.75 night pages.
A night page costs the responder the rest of that night's sleep
plus a degraded following day.

Annual per person: ~36 disrupted nights, concentrated into 13
weeks.

Observable consequences, in order of appearance:
  1. people decline to join the team
  2. the best responders leave first, because they have options
  3. the rotation shrinks to 3, so each person is on call 1 week
     in 3
  4. the load per person rises by a third, and step 2 repeats

The failure mode is a doom loop, and the loop's gain is greater
than 1 once the rotation drops below about 5.

The design's job is to break that loop with numbers that make the load visible and give the excess somewhere to go.

Mechanics

Rotation size

N people, 24/7 primary rotation:
  each person is on call 1 week in N = 52/N weeks per year

  N=3   17.3 weeks/yr   one week in three. Unsustainable.
  N=4   13.0            common, and the level where people leave
  N=6    8.7            the practical floor
  N=8    6.5            comfortable
  N=12   4.3            people lose familiarity with the systems

The two failure directions:
  TOO SMALL  burnout, and the doom loop above
  TOO LARGE  each person is on call so rarely that they have not
             seen the system change since their last shift, so
             every page is a cold start

Target 6 to 8 for a 24/7 rotation. Below 6, the answer is not a better schedule, it is more people or fewer hours of coverage.

Secondary rotations matter more than people expect:

PRIMARY    takes the page
SECONDARY  escalation after N minutes unacknowledged, and a
           second pair of hands for a long incident

A secondary is not optional above a certain severity, because
a single primary who is asleep, driving, or already handling
one incident is a single point of failure with a human in it.

The cost is that being secondary is also a constraint on your
evening. Count it: a 6-person rotation with primary and
secondary means each person is constrained 2 weeks in 6, not 1.

That last line is the one omitted from most rotation arithmetic, and it means a "6-person rotation" with a secondary is a 1-in-3 constraint.

Follow-the-sun, and what it actually requires

Removes night pages entirely. Requires:
  - teams in 2 or 3 timezones roughly 8 hours apart
  - each capable of handling the same incidents, which means real
    knowledge distribution, not a runbook handoff
  - a genuine handoff protocol: open incidents, degraded
    components, changes in flight, what to watch
  - alignment of ownership, so a region is not paging about a
    service it cannot change

Where it fails: two regions with 12 hours between them and no
overlap, so the handoff is asynchronous and context is lost. And
a "follow the sun" arrangement where one region owns the code
and the other owns the pager, which is just outsourcing the pain.

The page budget, and enforcing it

The target: <= 2 paging events per 12-hour shift.

The reason for the specific number, from Google's SRE practice:
a page should be followed by a proper investigation and a
follow-up, and that takes hours. More than two per shift and the
responder is triaging, not resolving, so the underlying causes
are never removed and the rate never falls.

The budget only does work if exceeding it has a defined consequence:

The mechanism that works:

  IF pages per shift exceeds the budget for 2 consecutive weeks
  THEN the next sprint's reliability bucket is spent on the top
       page sources, and feature commitments are reduced
       accordingly.

That is an ERROR-BUDGET-SHAPED policy applied to human load, and
it works for the same reason: it converts many individually
losing arguments ("can we fix this alert?") into one policy
argument that was won in advance.

The weekly pager review is the operational half:

Every page from the past week, classified in a 30-minute meeting:

  ACTIONABLE     a human had to decide or act. Keep the alert.
  AUTOMATABLE    the response was mechanical (restart, scale,
                 failover). Automate the response, keep a
                 low-priority ticket for visibility.
  SHOULD BE A
  TICKET         real but not urgent. Downgrade to a ticket. This
                 is usually the largest category.
  SHOULD BE
  DELETED        not actionable, not urgent, or duplicative.
  FALSE          the alert fired without the condition being
                 true. Fix the alert or the threshold.

Every page leaves the meeting with an owner and one of those five
dispositions. A page that is discussed and left alone is the
default failure of this meeting.

Alert criteria worth stating as a rule:

A page must be:
  URGENT      it cannot wait until morning
  ACTIONABLE  there is something a human can do now
  NOVEL       it is not the fifth copy of the same condition

Anything failing one of the three is a ticket, a dashboard, or a
deletion. Symptom-based alerts (see USE and RED) satisfy these
far more often than cause-based ones, because a cause-based
alert fires whether or not users are affected.

The interrupt shield

Separate the interrupt-absorbing role from the pager.

Names vary: "support engineer", "sheriff", "batman", "goalie".
The design:
  - one person per week, rotating, DIFFERENT from the on-call
    primary
  - owns: incoming questions, small bug reports, triage of new
    tickets, the shared inbox and the team's Slack channel
  - explicitly NOT expected to make roadmap progress that week
  - has a written escalation path for anything they cannot
    answer in 15 minutes

The arithmetic that justifies it:

Research on interrupted work (Mark, Gudith and Klocke, CHI 2008)
found it takes on average about 23 minutes to return to the
original task after an interruption.

A team of 7 with 30 interrupts a week, spread across everyone:
  30 x 23 min = 11.5 hours of refocus cost, distributed as
  small losses nobody attributes to interruption.

Concentrated on one person:
  that person loses their week (which is planned for)
  the other 6 lose nothing
  -> the team recovers roughly 11.5 hours a week of focus time,
     at the cost of 1 planned person-week in 7.

The failure mode: the shield is assigned and then everyone is still messaged directly. The fix is social and mechanical: a single channel, a bot that redirects direct messages, and the shield answering publicly so the answer is findable next time.

Second-order benefit: the shield produces the documentation backlog. A person answering the same question three times writes it down, because they are the one who will be asked again.

The toil cap

TOIL is operational work that is manual, repetitive, automatable,
tactical, devoid of enduring value, and scales linearly with
service growth.

The cap: <= 50 percent of a person's time. Google's SRE model
enforces it with a defined overflow: when an SRE team's toil
exceeds the cap, work is handed BACK to the product development
team until it is under control.

The overflow mechanism is the part that matters, and it is what
most imitations omit. A cap with no consequence is a preference.

A cap you can actually operate in a non-SRE organisation:

- Measure toil per person per week from the three-bucket split
  (see capacity math).
- Declare the cap in the team charter: KTLO <= 30 percent, say.
- Define the overflow: sustained breach for 2 sprints means the
  next sprint's feature commitment is reduced and the reliability
  bucket doubled, with the lead's decision documented, not
  negotiated per instance.
- Review quarterly with the number, not the feeling.

The rest of the design

COMPENSATION
  Paid on-call, or time in lieu, or both. It is the honest lever,
  it prices the burden so the organisation feels it, and it makes
  the rotation something people will join. In many jurisdictions
  standby pay also has legal dimensions worth checking.

HANDOFF
  A written handoff at the end of each shift: open incidents,
  degraded components, changes landing, things to watch, and any
  alert that fired more than twice. 10 minutes, and it is the
  difference between a rotation and a relay of cold starts.

RUNBOOKS
  Every alert links to a runbook. A runbook that has not been
  used in 6 months is probably wrong; the pager review is where
  that gets caught.

ONBOARDING
  Nobody takes a primary shift alone until they have shadowed at
  least two and been secondary for one. And a new joiner's first
  shift is paired.

BLAMELESS FOLLOW-UP
  Every page above a severity threshold gets a written follow-up.
  Not a full postmortem for a routine page, but a paragraph with
  an action, because the action is the only thing that reduces
  next quarter's load.

A worked example: a rotation that shrank until it broke

A data-platform team. 5 engineers, 24/7 rotation, ownership of 6 services including the ingestion pipeline everything else depended on.

The state at the start:

Rotation                     5 people, primary only, no secondary
Pages per week                        47
Pages between 22:00 and 08:00         11
Median pages per 12-hour shift       3.4   (budget: 2)
Actionable rate (from a sample)       21%
Interrupt requests per week          ~34   (Slack, direct)
Attrition in the prior 12 months       3   (of 8; the rotation had
                                            been 8)
Open reliability tickets              61, none in the current sprint

Twenty-one percent actionable means four out of five pages did not need a human at that moment, which is the whole finding in one number.

The pager review, four weeks of pages classified:

188 pages classified:

  40   ACTIONABLE       genuine, human decision required
  61   AUTOMATABLE      "restart the consumer", "scale the pool",
                        "clear the stuck partition". Mechanical
                        response, done by hand every time.
  52   SHOULD BE A
       TICKET           real, not urgent: a disk at 78%, a certificate
                        expiring in 21 days, a batch job 40 minutes
                        late with a 6-hour SLA
  27   DELETED          not actionable at all: "CPU > 80%" on a
                        service that is supposed to run hot
   8   FALSE            a threshold that fired on a monitoring gap

Sixty-one automatable pages is a robot's job being done by a person at 3am, and 52 tickets-in- disguise is an alert-severity problem, not a reliability problem.

The work, sequenced by cost:

WEEK 1: DELETE AND DOWNGRADE. Free.
  - 27 deletions, agreed in the review
  - 52 downgraded to tickets with a daily digest instead of a page
  - 8 threshold fixes
  -> pages/week 47 -> 18, in one week, with no engineering work.

WEEKS 2-6: AUTOMATE THE MECHANICAL ONES.
  - stuck-partition detection and reset: 3 days
  - consumer restart on a defined health signal, with a
    circuit breaker so it does not loop: 4 days
  - connection-pool autoscaling: 2 days
  - each automation pages ONLY if it fails or fires more than
    3 times in an hour, which preserves the signal
  -> pages/week 18 -> 9

WEEKS 4-12: THE TOP THREE ACTIONABLE CAUSES.
  Of the 40 actionable pages, 23 came from three causes:
    - an unpartitioned table causing lock contention at month end
    - a dependency with no timeout, so its slowness became ours
    - a manual certificate renewal process
  Fixed: partitioning, timeouts plus a bulkhead, automated cert
  renewal.
  -> pages/week 9 -> 6

The 27-deletion, 52-downgrade week is the highest-return work in the whole programme and cost nothing but a meeting, which is the general shape: most pager pain is classification, not reliability.

Then the structural changes:

ROTATION: merged with the adjacent streaming team's rotation.
  5 + 4 = 9 people. Each person 1 week in 9 primary, and a
  secondary rotation offset so nobody is primary and secondary
  in consecutive weeks.
  Cost: 6 weeks of cross-training, 2 shadow shifts each, and a
  shared runbook set. Real, and it was the largest single cost.

INTERRUPT SHIELD: introduced separately from on-call.
  One person per week, not the on-call primary, owning the
  team channel and triage, with no roadmap commitment.
  34 interrupts/week x 23 min = ~13 hours of refocus cost
  previously spread across the team, now absorbed by one
  planned person-week.

PAGE BUDGET with an overflow rule:
  "<= 2 pages per 12-hour shift. Two consecutive weeks above it
   and the next sprint's reliability allocation doubles, with the
   feature commitment reduced accordingly."
  Written into the team charter and agreed with the PM in
  advance, which is the only time that agreement is cheap.

COMPENSATION: on-call stipend introduced, plus a day in lieu for
  any shift with a page between midnight and 06:00.

Agreeing the overflow rule before it fires is the whole trick, because after it fires the conversation is about a specific feature slipping, and it will lose.

Twelve months later:

                              before      after
rotation size                      5           9 (+ secondary)
on-call frequency         1 wk in 5   1 wk in 9 primary
pages per week                    47           6
night pages per week              11         0.4
median pages per shift           3.4         0.5
actionable rate                  21%         76%
interrupts hitting the
  wider team                  ~34/wk       ~4/wk
KTLO share of capacity           58%         26%
attrition (12 months)              3           0
open reliability tickets          61          19

The actionable rate is the number to watch, not the page count. A page count can be reduced by deleting alerts you needed; an actionable rate rising from 21 to 76 percent says the remaining pages are the right ones.

One thing that got worse, stated plainly:

Merging the rotations meant each person was on call for 11
services instead of 6, and mean time to acknowledge rose from
4.1 to 6.8 minutes in the first quarter, because responders were
less familiar with half the surface.

It returned to 4.4 minutes over the following two quarters as
cross-training took hold and runbooks improved.

This was raised as an objection before the merge, was correct,
and was accepted as a transitional cost against a rotation that
was losing three people a year.

Production evidence

Google's SRE Book specifies both numbers used above: a target of no more than two paging events per 12-hour on-call shift, with the rationale that a page deserves a proper investigation and follow-up, and a 50 percent cap on operational work with an explicit overflow mechanism that returns work to the product development team. The overflow is the part that makes the cap enforceable.

Google's definition of toil (manual, repetitive, automatable, tactical, without enduring value, and scaling linearly with service growth) is the operative one, and its "scales linearly with growth" clause is what distinguishes toil from ordinary operational work.

Mark, Gudith and Klocke, "The Cost of Interrupted Work" (CHI 2008) is the source for the roughly 23-minute resumption time after an interruption, which is the arithmetic behind concentrating interrupts on one person rather than distributing them.

PagerDuty's published operational guidance on rotation design, escalation policies and secondary rotations documents the single-primary risk and the standard remedy, and their incident-response documentation is the widely used reference for handoff practice.

Symptom-based alerting, alerting on user-visible effects rather than on causes, is the position taken in both Google's SRE material and in the USE and RED method literature, and it is the mechanism behind a high actionable rate.

Published on-call compensation practice varies widely by jurisdiction and employer, but paid standby is common in European engineering organisations and is increasingly so in North America, which is the evidence that pricing the burden is a live lever rather than a theoretical one.

The debate

Is a 6-person minimum too rigid? It is a floor, not a target, and the honest version is that below 6 you are choosing between reducing coverage hours, merging rotations, or accepting attrition. The counter-argument, that a small team with almost no pages can run a 4-person rotation comfortably, is true, and the test is the page rate rather than the headcount: 4 people with 2 pages a month is fine, 4 people with 47 pages a week is the doom loop.

Should on-call be paid? Yes, and the argument that it is "part of the job" ignores that it is a constraint on non-working hours. Payment does two things: it compensates the individual, and it puts the cost on the organisation's books where it can be compared against the cost of fixing the pages. An unpriced burden is optimised away last.

Should the interrupt shield be the same person as on-call? No. Combining them means the person handling a production incident is also the person expected to answer questions, and the incident loses. On small teams where separating them is impossible, the honest answer is to reduce one of the two loads rather than to pretend one person can do both.

Is a page budget gameable? Yes, by deleting alerts that were needed, which is why the actionable rate is the companion metric. A falling page count with a falling actionable rate means you deleted the useful ones. Report both.

Does follow-the-sun solve this? It removes night pages, which is the largest single harm, and it requires genuine capability in each region plus a real handoff. The failure is a region that owns the pager without owning the code, which relocates the pain without reducing it and adds a handoff to every incident.

Should automation be preferred to fixing the cause? Automating a mechanical response is the right immediate move, because it removes the 3am human, and it is not a fix. The risk is that automation hides the signal, so an automated remediation should still record its occurrences and page if it fires repeatedly or fails, which keeps the underlying rate visible for the quarterly review.

Follow-up Q&A

"What makes an on-call rotation sustainable?"

Four explicit numbers. A rotation of at least six for 24/7, because one week in four is where people begin to leave and the rotation then shrinks, raising the load, which is a loop with gain above one. A page budget of about two events per twelve-hour shift, so there is time to investigate rather than triage. An interrupt shield that is a separate person from the on-call primary. And a cap on operational work with a defined overflow, because a cap without a consequence is a preference. Every one of these is a number in a charter, agreed in advance, rather than a judgment made under pressure.

"What do you do first with a rotation getting 47 pages a week?"

Classify four weeks of pages in a meeting, into actionable, automatable, should-be-a-ticket, delete and false. In one case that was 40, 61, 52, 27 and 8 out of 188. The deletions and downgrades take one meeting and no engineering work and took the rate from 47 a week to 18. Then automate the mechanical responses, which is a robot's job being done by a person at 3am, and only then work the top actionable causes, which is the expensive part. Most pager pain turns out to be a classification problem, not a reliability problem.

"Why is the actionable rate more important than the page count?"

Because the page count is trivially gameable by deleting alerts you needed. An actionable rate, the fraction of pages where a human genuinely had to decide or act, tells you whether the remaining pages are the right ones. Going from 21 percent to 76 percent while the count fell from 47 to 6 is a strong result; the same count reduction with a falling actionable rate would mean you had deleted signal. Report them together, always.

"Why separate the interrupt shield from on-call?"

Because they are different jobs with different urgency, and combining them means the person handling a production incident is also the person expected to answer questions in Slack. The arithmetic also favours concentration: an interruption costs roughly 23 minutes of resumption time, so 30 interrupts a week spread across seven people is about eleven and a half hours of refocus cost invisibly distributed, whereas concentrating them on one planned person-week costs one week and returns that time to the other six.

"What makes a toil cap enforceable?"

The overflow mechanism. Google's SRE model caps operational work at 50 percent and, when a team exceeds it, hands work back to the product development team until it is under control. Most imitations copy the number and omit the consequence, which makes it advisory. A workable non-SRE version is a declared KTLO ceiling in the charter, measured from the three-bucket split, with a written rule that a sustained breach doubles the next sprint's reliability allocation and reduces the feature commitment accordingly, agreed with the product partner in advance, because after it fires the conversation is about a specific feature slipping and it will lose.

"What criteria should a page meet?"

Urgent, actionable and novel. If it can wait until morning it is a ticket. If there is nothing a human can do right now it is a dashboard or an automation. If it is the fifth copy of the same condition it should be deduplicated. Symptom-based alerts, on user-visible effects, satisfy these far more often than cause-based ones, because a cause-based alert fires whether or not anyone is affected, which is where most of the non-actionable volume comes from.

Common misconceptions

"A rotation is a schedule." A schedule says who. A design says how much, what happens when it is exceeded, and who pays to reduce it.

"Fewer pages is the goal." Fewer non-actionable pages is the goal. A count that falls while the actionable rate falls means signal was deleted.

"On-call is part of the job, so it should not be paid." It is a constraint on non-working hours, and pricing it is what puts the cost on the organisation's books next to the cost of fixing it.

"Automating the response fixes the problem." It removes the human at 3am, which is worth doing immediately, and it can hide the underlying rate unless the automation records and escalates on repetition.

"Small teams cannot afford an interrupt shield." Small teams pay for interrupts anyway, spread thinly and unmeasured. The shield makes the cost one planned person-week instead of an unattributed loss across everyone.

"A 50 percent toil cap is the control." The overflow mechanism is the control. The number without it is a preference.

Interview delivery note

Say this verbatim: "The first move on a broken rotation is not reliability work, it is classification. We classified four weeks of pages into actionable, automatable, ticket, delete and false, and the deletions and downgrades alone took us from 47 pages a week to 18 in one meeting with no engineering work. Twenty-one percent of pages were actionable, which meant four out of five did not need a human at that moment." It shows you diagnose before you spend, and the numbers are memorable.

The senior-versus-staff separator is agreeing the overflow rule before it fires. A senior lead proposes a page budget. A staff lead writes into the charter, with the product partner's agreement in advance, that two consecutive weeks above budget doubles the next sprint's reliability allocation and reduces the feature commitment, because after the budget is breached the conversation is about a named feature slipping and it will lose. Converting many individually losing arguments into one policy argument won in advance is the same mechanism as an error budget, and saying so shows you know why it works.

The second signal is naming a cost your own change created. Merging two rotations took mean time to acknowledge from 4.1 to 6.8 minutes for a quarter, because responders covered eleven services instead of six. That objection was raised before the merge, was correct, and was accepted against a rotation losing three people a year. Volunteering the downside makes the rest of the numbers credible.

The Staff & Team Lead Interview Reference