Load shedding, backpressure and degradation as a ladder

What it is

When demand exceeds capacity, a system has exactly five options, and they form a ladder you climb as pressure rises. Naming them as one ordered set is the point, because teams usually implement one rung and are surprised by the failure the other four were for.

RUNG 5  FAIL FAST        circuit break, return an error in
                         microseconds. Preserve the dependency,
                         preserve yourself.
RUNG 4  SHED             reject cheaply, at admission, by
                         priority. Some requests get nothing.
RUNG 3  BACKPRESSURE     slow the producer down. Bounded queues,
                         TCP flow control, request(n), 429 with
                         Retry-After. Everyone gets served,
                         later.
RUNG 2  DEGRADE          serve a cheaper answer. Stale cache, no
                         personalisation, lower resolution,
                         skipped enrichment. Everyone gets
                         something, worse.
RUNG 1  DO THE WORK      normal operation.

The ladder's ordering is by how much the user loses, and the discipline is to climb only as far as you must, per request, rather than applying one response to everything.

What this is confused with: autoscaling. Scaling adds capacity on a timescale of tens of seconds to minutes. Overload arrives in milliseconds, and the queue that forms in the meantime is what kills you. The ladder is what you do in the window before capacity arrives, and it is also what you do when capacity cannot arrive because the constraint is a downstream database.

Also confused: backpressure and load shedding. Backpressure slows the producer and preserves the work; shedding discards the work. Backpressure only exists when the producer can be slowed, which is true of a Kafka consumer or an internal RPC caller and false of the public internet.

The problem it solves

Without a deliberate response, an overloaded system does the worst possible thing: it accepts everything and completes almost nothing useful.

Capacity 460 req/s. Arrival 700 req/s.

With unbounded queueing and no shedding:
  queue grows 240/s
  after 60s: 14,400 queued requests
  a request entering the queue now waits 31 seconds
  the client's timeout is 2 seconds

So the system is spending 100% of its capacity producing
responses for clients that left 29 seconds ago.

  throughput  458/s   (looks healthy)
  goodput       0/s   (nobody receives anything)

And it does not recover when the load falls, which is the property that turns a spike into an outage:

Arrival returns to 300/s, well under capacity.
The queue still holds 14,400 doomed requests.
Draining it at 460/s takes 31 seconds during which every
arriving request also queues behind them.
Clients time out and retry, adding load.
The system stays down long after the cause is gone.

That is a metastable failure, and the sustaining loop is
"work that will not be delivered consuming the capacity needed
to catch up".

Rejecting a request in 2 milliseconds instead of serving it in 31 seconds is the entire intervention, and it costs the user the same thing (no answer) while costing the system 15,000 times less.

Mechanics

Rung 2: degradation, and designing the tiers in advance

Degradation only works if you decided in advance what to drop, because nobody designs it under pressure.

A product page, ranked by cost and by value:

  tier 0  product name, price, buy button      NEVER dropped
  tier 1  images (full resolution)             -> lower res
  tier 2  inventory ("3 left")                 -> "in stock"
                                                  from a cached
                                                  approximation
  tier 3  personalised recommendations         -> a static
                                                  bestseller list
  tier 4  "customers also viewed"              -> omit entirely
  tier 5  review summary sentiment (an LLM
          call)                                -> omit entirely

Written down, with the fallback for each, and each one behind a
flag that a human or an automated controller can flip.

Two properties make a degradation tier real rather than aspirational:

1. THE FALLBACK IS TESTED. A cached-approximation path that has
   never run in production will not run correctly the first
   time, at 3am, under load. Exercise it: run 1% of traffic on
   the degraded path continuously, or flip it deliberately once
   a month.

2. IT IS TRIGGERED AUTOMATICALLY, not by a human noticing.
   A human takes 5 to 15 minutes; the incident is decided in
   the first 60 seconds.

The strongest form of degradation is serving stale. A cache entry that is 10 minutes old is almost always better than an error, and stale-if-error and stale-while-revalidate make it a configuration rather than a code path.

Rung 3: backpressure, and where it is available

BOUNDED QUEUES, everywhere. An unbounded queue is not a buffer,
  it is a device for converting a latency problem into an
  out-of-memory problem while destroying goodput on the way.

TCP flow control. Already there, and it works: if you stop
  reading from a socket the sender's window closes. Frameworks
  that read eagerly into an unbounded buffer defeat it.

REACTIVE STREAMS request(n). The consumer tells the producer how
  much it can take. This is what the specification is FOR, and
  it is the only backpressure mechanism that composes across
  async boundaries.

CONSUMER LAG as backpressure. A Kafka consumer that stops
  polling stops receiving. The producer is unaffected, so the
  "pressure" is absorbed by the log, which is exactly what a
  log is for.

429 WITH Retry-After. The HTTP-level version, and it only works
  if clients honour it. Assume many will not.

The critical property: backpressure requires a producer that can be slowed. For a public API the producer is the internet, which will not slow down, so rung 3 does not exist and you go straight from degradation to shedding.

Rung 4: shedding, and how to do it cheaply

Reject as early and as cheaply as possible. A request rejected after doing 90 percent of the work costs almost as much as serving it.

Where to shed, best first:

  at the edge / load balancer   costs microseconds, protects
                                everything behind it
  at admission control in the
    service (before dequeue,
    before any I/O)             costs ~1ms
  after partial work            costs almost as much as
                                serving. This is not shedding.

Prioritisation, because not all requests are equal:

Criticality, propagated with the request (Google's scheme):

  CRITICAL_PLUS   the thing the business cannot lose. A payment.
  CRITICAL        default for user-facing requests.
  SHEDDABLE_PLUS  partially degraded is acceptable. Batch,
                  prefetch, background sync.
  SHEDDABLE       fully droppable. Speculative work, logging
                  enrichment, best-effort analytics.

Shed from the bottom up. The criticality MUST propagate to
every downstream service, or a critical request will be shed by
a service three hops down that has no idea what it is carrying.

Criticality propagation is the part that gets skipped and the part that makes prioritised shedding work at all, because the shedding decision is usually made by a service that did not receive the user's request directly.

Adaptive concurrency limits

A fixed concurrency limit is wrong at every moment except the one you tuned it in. Derive it instead, using the same idea TCP uses for congestion.

Little's Law: limit = throughput x latency.

The adaptive version (Netflix's concurrency-limits, the Vegas
and gradient algorithms):

  measure RTT continuously
  keep RTT_noload = the minimum observed (the uncontended
    latency)
  gradient = RTT_noload / RTT_current

  if gradient ~ 1  the system is not queueing -> raise the limit
  if gradient < 1  latency is above the floor, so a queue has
                   formed -> lower the limit

  new_limit = current_limit * gradient + queue_allowance

This is AIMD on a latency signal, and it needs no tuning,
adapts to a degraded dependency automatically, and shrinks the
limit when a downstream slows without anyone changing config.

// The shape, without the algorithm's details.
if (!limiter.tryAcquire(request.criticality())) {
    // Rejected at admission: no I/O has happened, no worker is
    // held, and the cost is a counter increment.
    return Response.status(429)
        .header("Retry-After", "1")
        .entity(SHED_BODY)      // preallocated; do not serialise here
        .build();
}
try { return handle(request); } finally { limiter.release(); }

Note the preallocated response body. Under overload, the shed path is the hot path, and allocating or serialising in it is how a load-shedding mechanism becomes a source of load.

Queue discipline: LIFO beats FIFO under overload

This is the least intuitive item on the page and one of the most valuable.

Queue of 5,000 requests, capacity 460/s, client timeout 2s.

FIFO: serve the oldest first.
  The oldest request has been waiting 10 seconds.
  Its client timed out 8 seconds ago.
  You serve it, then the next-oldest, also doomed, and so on.
  EVERY response you produce is for a client that has left.
  goodput = 0.

LIFO: serve the newest first.
  The newest request has been waiting 5ms.
  Its client is still there.
  You serve it, and the next-newest, and so on.
  The oldest requests starve, and they were already doomed.
  goodput = capacity.

Under overload, FIFO is maximally unfair in the sense that
matters: it distributes the outcome "nobody gets anything" to
everyone, where LIFO gives full service to as many as capacity
allows.

LIFO is wrong under normal load (it starves the oldest for no reason), so the useful form is adaptive: FIFO normally, switch to LIFO when queue delay exceeds a threshold. CoDel (controlled delay) is the principled version: if the queue's minimum sojourn time has stayed above a target (say 5ms) for an interval (say 100ms), start dropping from the head.

Deadline propagation, which makes all of it cheaper

Every request carries a deadline. Every hop:
  - checks whether the deadline has passed; if so, drop
    immediately without doing the work
  - passes the REMAINING budget downstream, not a fresh timeout

Without it, a service happily performs a 400ms database query
for a request whose client gave up 3 seconds ago, and the
database's capacity is consumed by work that cannot be
delivered.

With it, doomed work is discarded at the cheapest possible
point, which is exactly the shedding principle applied
recursively.

Rung 5, and the retry problem

Retries during overload are the most common way a degradation becomes an outage.

A 3-attempt retry policy, no budget:
  offered 500/s, capacity 460/s
  40/s fail -> retried -> 120/s of extra load
  more failures -> more retries
  effective arrival climbs to 1,500/s against 460/s capacity

The multiplier arrives at precisely the moment you can least
afford it.

The three controls, all needed:
  RETRY BUDGET: cap retries at a fraction of request volume
    (10% is a common figure). Above the cap, do not retry.
  CIRCUIT BREAKER: stop calling a failing dependency entirely
    for a period, so you fail in microseconds instead of after
    a timeout.
  JITTERED BACKOFF: without jitter, the retries of all failed
    requests re-synchronise into a second spike.

Measuring the ladder

The four numbers to have on the dashboard:

  goodput          responses delivered within the client's
                   deadline. THE metric. Throughput alone
                   cannot distinguish working from wasting.
  shed rate        by criticality. Shedding SHEDDABLE is
                   healthy; shedding CRITICAL_PLUS is an
                   incident.
  queue delay      the p50 and p99 sojourn time, which is what
                   the adaptive controller acts on.
  degradation
    tier active    which tiers are currently degraded, as a
                   first-class signal rather than a log line.

A worked example: 40 minutes of unavailability from a 13 percent overspike

A checkout API. Capacity around 460 req/s. A marketing campaign produced a peak of 520 req/s for eleven minutes, and the service was unavailable for 40 minutes, including 25 minutes after traffic returned to 300 req/s.

The state before:

- unbounded internal work queue
- FIFO
- fixed thread pool of 200, tuned 18 months earlier
- client retry policy: 3 attempts, exponential backoff, NO
  jitter, NO budget
- no admission control, no shedding, no criticality
- no deadline propagation: each hop set a fresh 30s timeout
- degradation: none designed. Recommendations, inventory
  precision and review sentiment were all inline and mandatory.
- autoscaling: HPA on CPU, 3-minute reaction time

The 40 minutes, reconstructed:

t+0     arrival 520/s against 460/s capacity. Queue grows 60/s.
t+30s   queue 1,800. Queue delay 3.9s. Client timeout is 2s, so
        every response is now being delivered to a client that
        has left. GOODPUT REACHES ZERO HERE, at 30 seconds,
        while throughput is still 460/s and every dashboard is
        green.
t+45s   clients begin retrying. No jitter, so the retries
        arrive as a spike. Effective arrival ~1,100/s.
t+2m    queue 40,000+. Autoscaler adds pods; each new pod
        joins, receives a share of a hopeless queue, and its
        CPU pins immediately. Scaling made the failure wider,
        not shorter.
t+3m    the pricing service's connection pool is exhausted,
        because checkout holds connections for 30 seconds
        (fresh timeout per hop) while doing work for departed
        clients. Pricing now fails for OTHER callers.
t+11m   campaign traffic ends. Arrival 300/s.
t+11m   no recovery: the queue holds ~180,000 doomed requests
        and retries continue.
t+36m   operators drain the queue by restarting every pod,
        which is the only tool available.
t+40m   recovered.

Goodput hit zero at 30 seconds and nobody knew for eleven minutes, because the only metrics were throughput and error rate, and throughput was normal while every response was going to nobody.

The ladder, implemented over a quarter:

RUNG 2, DEGRADATION (2 weeks)
  Five tiers written down with the product owner and each put
  behind a flag with an automatic trigger on queue delay:
    >200ms  drop review sentiment (an LLM call, 180ms p50)
    >400ms  static bestsellers instead of personalised recs
    >600ms  cached inventory approximation
    >800ms  lower-resolution images
  Each fallback exercised on 1% of traffic continuously, so it
  is a tested path rather than a hypothesis.
  Effect measured in isolation: shedding the LLM call and the
  recommendations raised effective capacity from 460/s to
  610/s, because both were on the critical path for no
  necessary reason.

RUNG 3, BACKPRESSURE (1 week)
  Bounded queue, 2,000 items. Reactive streams request(n)
  between the ingest stage and the pricing client, so the
  pricing client's slowness propagates as reduced demand
  rather than as accumulated work.

RUNG 4, SHEDDING (3 weeks)
  Adaptive concurrency limiter (gradient algorithm on measured
  RTT) at admission, before any I/O.
  Criticality propagated in a header, set at the edge:
    CRITICAL_PLUS  POST /checkout/pay
    CRITICAL       the rest of checkout
    SHEDDABLE_PLUS recommendations, review summaries
    SHEDDABLE      analytics enrichment, prefetch
  Shed from the bottom, with a preallocated 429 body and
  Retry-After.

QUEUE DISCIPLINE (3 days)
  CoDel-style: FIFO normally; if minimum sojourn time exceeds
  5ms for 100ms, switch to LIFO and drop from the head.

DEADLINE PROPAGATION (2 weeks)
  A deadline header set at the edge, decremented at each hop,
  checked before dequeue and before every outbound call.

RETRY CONTROLS (1 week)
  Retry budget capped at 10% of request volume, full jitter,
  and a circuit breaker on the pricing client.

Retesting with an open-model load generator:

             BEFORE                    AFTER
offered  completed  goodput      completed  goodput   shed
 460/s     460/s     460/s         460/s     460/s      0
 520/s     459/s      12/s         520/s     520/s      0   (deg.
                                                            tiers)
 700/s     458/s       0/s         608/s     608/s     92/s
1,000/s    310/s       0/s         610/s     610/s    390/s
recovery after a 5-minute
  overload at 1,000/s:
           25+ min                    11 seconds

Capacity above the knee went from "collapses to zero goodput" to "goodput equals capacity, and everything above it is rejected in about 2 milliseconds", and recovery went from 25 minutes to 11 seconds.

The 520 req/s row is the one that mattered commercially: the campaign load that caused a 40-minute outage is now served completely, because degradation raised effective capacity above it before any shedding was needed.

Three findings worth recording:

1. THE LLM CALL WAS THE CHEAPEST WIN. A review-sentiment
   summary added 180ms p50 to every product page and was
   inline and mandatory. Making it tier 0 for degradation, and
   later asynchronous entirely, raised capacity by ~22% on its
   own. Nobody had noticed because it was fast enough at
   normal load.

2. AUTOSCALING MADE THE ORIGINAL FAILURE WORSE. Each new pod
   received a share of a queue full of doomed work and pinned
   immediately, so scaling increased the number of instances
   producing zero goodput. Capacity does not help when the
   problem is that capacity is being spent on undeliverable
   work.

3. GOODPUT REACHED ZERO AT 30 SECONDS WHILE EVERY DASHBOARD
   WAS GREEN. Adding goodput as a first-class metric was a
   two-day change and it is the one that would have made the
   original incident a 90-second event, because it is the only
   signal that distinguishes "busy" from "wasting".

One thing that did not work on the first attempt:

The first shedding implementation rejected requests AFTER
dequeue and after the auth check, which involved a Redis
lookup.

Under a 1,000 req/s load test, the shed path itself consumed
enough Redis capacity to slow the auth lookups for the
requests that were NOT being shed, so the system got worse as
shedding engaged.

Moved admission control before auth, using only the criticality
header (which is set at the edge and signed), so a shed costs a
counter increment and a preallocated response.

The lesson recorded: the shed path is the hot path under
overload, so anything it touches becomes a bottleneck.

Production evidence

Google's SRE Book chapter on handling overload specifies the criticality scheme (CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE), its propagation through RPC metadata, and the practice of per-client adaptive throttling, along with the argument that a request rejected after partial work is nearly as expensive as one served.

Netflix's concurrency-limits library implements the gradient and Vegas algorithms, deriving a concurrency limit from measured latency using TCP congestion-control ideas, and its documented purpose is to remove hand-tuned limits that are correct only at the moment they were set.

CoDel (Controlled Delay), from Nichols and Jacobson's work on bufferbloat, is the algorithm behind "if minimum sojourn time exceeds a target for an interval, start dropping," and it was adapted from network queueing to application-level request queueing by Meta, whose published work on their load-shedding infrastructure also documents the LIFO-under-overload argument.

Metastable failures in distributed systems (Bronson et al., HotOS 2021) names the sustaining feedback loops, retries, queue growth and cache misses, that keep a system degraded after the trigger has gone, which is the theoretical account of the 25-minutes-after-recovery behaviour above.

gRPC deadlines and their propagation are a first-class protocol feature specifically so that a hop can discard work whose deadline has expired, and the gRPC documentation states the anti-pattern of setting a fresh timeout per hop explicitly.

AWS's "Exponential Backoff and Jitter" article is the canonical treatment of why unjittered retries re-synchronise into a second spike, and retry budgets appear in Google's SRE material and in Envoy's retry configuration as retry_budget.

stale-while-revalidate and stale-if-error are standardised Cache-Control extensions (RFC 5861), which makes serving stale a configuration rather than a bespoke degradation path.

The debate

Is shedding better than queueing? Above capacity, decisively. Queueing converts a capacity problem into a latency problem and then into a goodput problem, and once queue delay exceeds the client timeout, every unit of work is waste. The honest counter-argument is that shedding is visible and queueing is not, so shedding produces user complaints and support tickets where queueing produces a slow site, and that visibility is politically harder even though it is the better outcome.

Should you always prioritise? Where you can propagate criticality, yes. Without propagation, prioritised shedding is worse than uniform shedding, because a service three hops down will shed a payment while preserving a prefetch. The propagation is the work; the shedding decision is trivial.

Is LIFO fair? Not in the ordinary sense, and it is better on the metric that matters. Under overload FIFO distributes "nobody gets anything" to everyone; LIFO gives full service to as many as capacity allows. The position: FIFO by default, LIFO above a delay threshold, and say the tradeoff out loud because it sounds wrong until the goodput arithmetic is on the table.

Do adaptive limits beat fixed ones? For anything with variable downstream latency, yes, because a fixed limit is correct only at the moment it was tuned. The cost is that an adaptive controller is another thing that can misbehave, notably during a slow start after a deploy, and it needs a floor so it cannot drive the limit to zero on a transient latency spike.

Should autoscaling be part of the answer? It is necessary and it is not the answer at this timescale. Scaling reacts in minutes and overload arrives in milliseconds, and in the worked example scaling actively made the failure worse by adding instances that inherited a hopeless queue. Scale for the trend; shed for the spike.

Is designing degradation tiers worth the effort? It is the highest-return rung, and in the worked example it raised effective capacity by a third before any shedding was involved, mostly by removing things from the critical path that had no business being there. The failure is designing them and never exercising them, which produces a fallback path that runs for the first time during an incident.

Follow-up Q&A

"What are the options when demand exceeds capacity?"

Five, in order of how much the user loses: do the work; degrade to a cheaper answer; apply backpressure so the producer slows and everyone is served later; shed, so some requests get nothing but cheaply and by priority; and fail fast to protect a dependency and yourself. Climb only as far as you must, per request. Most teams implement one rung, usually shedding or nothing, and are surprised by the failures the other four were for. Backpressure in particular does not exist for a public API, because the producer is the internet and it will not slow down.

"Why is queueing worse than rejecting?"

Because once queue delay exceeds the client's timeout, every response is delivered to a client that has left, so throughput stays healthy while goodput is zero. In one incident goodput reached zero at 30 seconds while every dashboard was green, and the queue then held 180,000 doomed requests, which meant the system did not recover when load fell because draining them consumed the capacity needed to catch up. Rejecting in 2 milliseconds costs the user the same thing as a 31-second wait that ends in a timeout, and costs the system about fifteen thousand times less.

"Where should you shed, and why does it matter?"

As early and cheaply as possible: at the edge, or at admission control before any I/O. A request rejected after partial work costs nearly as much as serving it, so late shedding does not relieve anything. This is not theoretical: one implementation rejected after the auth check, which involved a Redis lookup, and under load test the shed path's Redis traffic slowed auth for the requests that were not being shed, so the system degraded further as shedding engaged. The shed path is the hot path under overload, so it must touch nothing and allocate nothing, including its response body.

"Why does LIFO beat FIFO under overload?"

Because the oldest request in the queue is the one whose client is most likely to have given up. Serving oldest-first means every response goes to a departed client and goodput is zero; serving newest-first means the requests you complete still have someone waiting, and the ones that starve were already doomed. FIFO under overload distributes "nobody gets anything" to everyone. It is wrong under normal load, so the practical form is FIFO by default with a switch to LIFO when queue delay exceeds a threshold, which is what CoDel formalises.

"How do adaptive concurrency limits work?"

They apply TCP congestion-control reasoning to a request limit. Track the minimum observed round-trip time as the uncontended latency floor, compare it to current latency, and treat the ratio as a gradient: near one means no queueing so raise the limit, well below one means a queue has formed so lower it. It is AIMD on a latency signal, it needs no tuning, and it shrinks automatically when a downstream dependency slows, which is exactly when a hand-tuned fixed limit is most wrong. It needs a floor so a transient latency spike cannot drive the limit to zero.

"What is the role of deadline propagation?"

It makes shedding recursive and cheap. Every request carries a deadline, every hop passes the remaining budget rather than setting a fresh timeout, and any hop whose deadline has already expired discards the work immediately without doing it. Without it, a service performs a 400-millisecond database query for a request whose client gave up three seconds earlier, and the database's capacity is consumed by undeliverable work. In one incident, per-hop fresh 30-second timeouts meant checkout held pricing connections for departed clients until the pricing service's pool was exhausted for everyone else.

"Why did autoscaling make an overload incident worse?"

Because each new instance joined and received a share of a queue full of doomed requests, pinned its CPU immediately, and added another instance producing zero goodput. Scaling reacts in minutes; overload arrives in milliseconds; and capacity does not help when the problem is that capacity is being spent on work that cannot be delivered. Scale for the trend and shed for the spike, and fix the queue before adding instances to it.

Common misconceptions

"Queue it, we will catch up." Once queue delay exceeds the client timeout, catching up means completing work nobody will receive, and the queue is what prevents recovery.

"Throughput is fine, so we are fine." Throughput cannot distinguish serving from wasting. Goodput can, and in one incident it hit zero at 30 seconds while throughput was normal.

"Shed when we detect a problem." By the time a human detects it, the incident is already decided. Degradation and shedding have to be triggered automatically, on queue delay or latency gradient.

"Just add more instances." Autoscaling reacts in minutes and can make things worse by distributing a hopeless queue across more pods.

"FIFO is fair." Under overload it gives everyone the same outcome: nothing.

"Retries make the system more reliable." Without a budget and jitter they multiply arrival exactly when capacity is shortest, which is how a degradation becomes an outage.

Interview delivery note

Say this verbatim: "Once queue delay exceeds the client timeout, every response you produce goes to a client that has left, so throughput stays flat while goodput is zero. In one incident goodput hit zero at 30 seconds and every dashboard was green for eleven minutes. Rejecting in two milliseconds costs the user the same thing and costs the system fifteen thousand times less." It states the mechanism, the measurement failure and the trade in three sentences.

The senior-versus-staff separator is treating the five responses as one ordered ladder with automatic triggers. A senior engineer adds a circuit breaker or a rate limit. A staff engineer writes down the degradation tiers with the product owner, binds each to a queue-delay threshold, exercises the fallbacks on one percent of traffic continuously so they are tested paths, and only then adds prioritised admission control. In the worked example the degradation rung alone raised effective capacity from 460 to 610 requests per second, mostly by removing an LLM call from the critical path that nobody had noticed because it was fast enough at normal load.

The second signal is knowing that the shed path is the hot path. Saying "our first attempt shed after the auth check, which hit Redis, so under load the shed path slowed auth for the requests we were not shedding and the system got worse as shedding engaged" shows you have implemented this rather than read about it, and it generalises: any overload control that does work proportional to the overload is not a control.

The Staff & Team Lead Interview Reference