Resilience patterns: circuit breaker, bulkhead, timeout, retry

What it is

Four patterns that stop one failing dependency from taking down the caller. They are usually listed together and they defend against different failures, which is the distinction that matters:

TIMEOUT          Bounds how long you wait.
                 Defends against: a dependency that never
                 answers. Without it, threads accumulate
                 until the pool is exhausted.

RETRY            Tries again after a failure.
                 Defends against: transient failures.
                 AMPLIFIES: overload, if not bounded and
                 jittered.

CIRCUIT BREAKER  Stops calling a dependency that is failing.
                 Defends against: wasting resources on calls
                 that will fail, and against your retries
                 keeping a struggling dependency down.

BULKHEAD         Isolates resources per dependency.
                 Defends against: one slow dependency
                 consuming the shared pool that every other
                 dependency needs.

Commonly confused as interchangeable. A circuit breaker does not help with a slow dependency until it trips, and a bulkhead protects you the whole time, which is why the bulkhead is the one people skip and the one that most often would have prevented the outage.

Also commonly confused: retry is a resilience pattern that causes outages. Every one of these four makes things better except retry, which makes things better under transient failure and dramatically worse under overload, and telling those apart is the whole skill.

The problem it solves

The cascade, which is how a single dependency failure becomes a total outage.

1. The recommendations service slows from 20 ms to 5 s.
   Not down. SLOW, which is the dangerous case.

2. Your service calls it synchronously with a 30 s timeout
   (the default in most HTTP clients, which is effectively
   no timeout).

3. Threads that would have returned in 20 ms now hold for
   5 s. By Little's Law, concurrency = throughput x latency,
   so at 200 requests/sec the in-flight count goes from
   200 x 0.02 = 4 to 200 x 5 = 1,000.

4. The thread pool is 200. It is exhausted.

5. *** Every endpoint now fails, including the ones that
   never call recommendations. ***

6. Callers time out and retry, tripling the load.

7. The whole service is down because one optional
   dependency got slow.

Step 5 is the point. The failure was in an optional dependency and the outage was total, because the resource being exhausted was shared. That is what the bulkhead prevents and what none of the other three do.

Mechanics

Timeouts: the one that must exist

THE RULE: every network call has an explicit timeout, and
the default is not one.

  Java HttpClient        no timeout by default
  Python requests        no timeout by default
  Go http.Client         no timeout by default
  JDBC                   often no query timeout

A missing timeout is not "wait a long time", it is "wait
forever", and forever is how thread pools die.

Setting the value is the interesting part, and percentile-based is the answer:

TIMEOUT = p99.9 of the dependency's healthy latency,
          not a round number

  p50   20 ms
  p99   80 ms
  p99.9 200 ms
  -> timeout 250 ms, not 1 s and certainly not 30 s

WHY NOT HIGHER: a timeout above the dependency's real
latency distribution does not save anyone. It just means you
hold a thread for a second before failing, which is the
cascade above at a slower rate.

WHY NOT LOWER: you convert healthy slow requests into
failures and then retry them, which adds load to something
that was working.

Deadline propagation is the version most systems lack:

Client sets a 500 ms budget.
  -> Service A spends 100 ms, calls B with a 400 ms budget.
  -> B spends 200 ms, calls C with a 200 ms budget.
  -> C's own timeout is min(its default, 200 ms).

Without propagation, C happily works for 2 seconds on a
request whose caller gave up 1.5 seconds ago. That work is
pure waste and it consumes capacity during exactly the
incident where capacity matters.

gRPC has deadline propagation in the protocol; HTTP does not, so it is a header convention you implement (X-Request-Deadline or the grpc-timeout equivalent) and enforce in a client wrapper.

Retry: the pattern that causes outages

THE AMPLIFICATION

  Service is at capacity and starts returning errors.
  Every client retries 3 times.
  Load is now 3x on a service that was already over
  capacity.
  More errors, more retries.
  *** The retry policy has converted a degradation into an
      outage. ***

Three rules that make retry safe, and all three are needed:

1. RETRY ONLY WHAT IS RETRYABLE.
   429 and 503 with Retry-After: yes.
   500: maybe, and it might be deterministic, so it will
        fail again.
   400, 404, 422: never. The request is wrong and it will
        be wrong next time.
   Timeouts: only if the operation is IDEMPOTENT, because
        you do not know whether it completed.

2. EXPONENTIAL BACKOFF WITH FULL JITTER.
   Without jitter, retries synchronise: everyone fails at
   t=0, everyone retries at t=1, and you have recreated the
   spike.
     sleep = random(0, min(cap, base * 2**attempt))
   FULL jitter (random from zero) beats equal jitter and
   decorrelated jitter in AWS's published simulations, and
   it is one line.

3. BUDGET THE RETRIES, not just the count.
   A per-request cap of 3 still allows the whole fleet to
   triple its load. A RETRY BUDGET caps retries as a
   fraction of total requests:
     "retries may be at most 10% of successful requests
      over the last 10 seconds"
   Above that, fail immediately without retrying.

The retry budget is the mechanism most implementations lack, and it is the one that actually prevents the cascade, because a per-request limit bounds one client's behaviour and says nothing about the aggregate.

# A token-bucket retry budget: the fleet-level guard.
class RetryBudget:
    def __init__(self, ratio=0.1, min_per_sec=10):
        self.ratio, self.min = ratio, min_per_sec
        self.successes = Counter(window_s=10)
        self.retries = Counter(window_s=10)

    def may_retry(self) -> bool:
        allowed = self.min + self.successes.rate() * self.ratio
        return self.retries.rate() < allowed

Circuit breaker: stop calling a dead thing

THREE STATES

  CLOSED     calls pass through. Failures are counted.
  OPEN       calls fail IMMEDIATELY without attempting.
             After a cooldown, move to half-open.
  HALF-OPEN  a limited number of trial calls. Success closes
             the breaker; failure reopens it.

Two things to get right that most implementations get wrong:

Trip on a failure RATE over a rolling window, not on a consecutive count.

"5 consecutive failures" never trips under a 50% failure
rate, because successes keep resetting the counter. And a
50% failure rate is a serious outage.

  Correct: >50% failures over the last 20 requests in the
  last 10 seconds, with a minimum request threshold so it
  cannot trip on 2 samples.

Trip on LATENCY, not only on errors.

The realistic failure is slow, not dead. A dependency
answering successfully in 8 seconds passes every error-based
breaker and destroys you exactly as effectively as one
returning 500s.

  Trip if p99 over the window exceeds N x the healthy
  baseline, or if the timeout rate exceeds a threshold.

And the half-open state needs care: allowing full traffic through on the first success re-overloads a recovering dependency, so allow a small number of concurrent trial calls and ramp.

What a breaker does not do: help before it trips. During the window it takes to detect the failure, you are still holding threads. That gap is what the bulkhead covers.

Bulkhead: the one that is skipped

Named after ship compartments: a breach floods one compartment rather than the hull.

WITHOUT: one shared thread pool of 200 for all dependencies.
  recommendations slows -> holds 1,000 -> pool exhausted ->
  EVERYTHING fails.

WITH: separate pools per dependency.
  recommendations:  30 threads
  payments:         50 threads
  inventory:        50 threads
  search:           40 threads
  reserve:          30 threads

  recommendations slows -> its 30 threads are held ->
  recommendations calls fail fast -> EVERYTHING ELSE
  CONTINUES.

Sizing from Little's Law rather than by guessing:

threads = throughput x latency, with headroom

  recommendations: 50 req/s x 0.02 s healthy = 1 in flight
  -> 30 threads is generous, and generous is fine because
     the point is the CEILING, not the capacity.

The bulkhead's value is that the number is BOUNDED, not
that it is correct.

Semaphore bulkheads versus thread-pool bulkheads:

THREAD POOL   a separate pool per dependency. Real isolation
              including the ability to time out a blocked
              call. Costs threads and context switches.

SEMAPHORE     a permit count on the calling thread. Much
              cheaper, and it CANNOT interrupt a blocked
              call, so it needs the underlying client to
              have its own timeout.

With virtual threads or async I/O, the thread-cost objection largely disappears, and a semaphore bounding concurrent in-flight calls per dependency is cheap and sufficient. That is the modern default and it is worth saying, because the Hystrix-era thread-pool overhead argument is frequently still repeated.

The order they compose in

request
  |
  +-- BULKHEAD        do I have a permit for this dependency?
  |                   no -> fail fast, do not queue
  +-- CIRCUIT BREAKER is it open?
  |                   yes -> fail fast, do not call
  +-- TIMEOUT         bounded by the remaining deadline
  |
  +-- the call
  |
  +-- on failure: RETRY, if retryable AND the budget allows
  |
  +-- FALLBACK        cached value, default, or degrade

Bulkhead first is deliberate: it is the cheapest check and it is the one that protects everything else. And a fallback at the end is what converts a failure into a degradation, which is the actual goal: none of these patterns make the dependency work, they stop its failure from being yours.

A worked example: the outage and the fix

INCIDENT
  Checkout unavailable for 34 minutes. Root cause: the
  recommendations service (optional, used for an upsell
  module) degraded from 20 ms to 6 s.

WHY IT CASCADED
  1. Shared thread pool of 200 across all dependencies.
  2. Default HTTP client timeout: 30 seconds.
  3. Retry: 3 attempts, fixed 1-second delay, no jitter,
     no budget.

  At 200 req/s with recommendations at 6 s, in-flight goes
  to 1,200 against a 200 pool. Exhausted in about 4 seconds.
  Retries tripled the load on a service already failing.
  The synchronised 1-second retry created a thundering herd
  every second.

THE FIXES, in order of how much each contributed

  a. BULKHEAD. 30 threads for recommendations. Alone, this
     would have contained the entire incident: 30 threads
     held, 170 available, checkout unaffected.
     *** The single highest-value change. ***

  b. TIMEOUT at 250 ms (p99.9 was 200 ms). Failure detected
     in 250 ms rather than 30 s, so even the shared pool
     would have survived longer.

  c. CIRCUIT BREAKER on p99 latency, not just errors. The
     dependency was returning 200s slowly, so an
     error-based breaker would never have tripped.

  d. RETRY: full jitter and a 10% budget. The fixed 1-second
     delay was synchronising retries into a herd.

  e. FALLBACK: render the page without the upsell module.
     Which raises the real question below.

THE QUESTION THAT SHOULD HAVE BEEN ASKED FIRST
  Why was an optional module a synchronous blocking call in
  the checkout path at all? The resilience patterns made the
  failure survivable, and moving the call off the critical
  path made it irrelevant. Both were done, and the second
  mattered more.

That last point generalises: resilience patterns bound the damage from a dependency you must call, and the better question is often whether you must call it.

Production evidence

Michael Nygard's Release It! (2007) is the origin of the modern vocabulary: circuit breaker, bulkhead, timeouts, fail fast, and the stability antipatterns (cascading failure, blocked threads, integration points) that they answer.

Netflix's Hystrix popularised the combination and its documentation is explicit that thread pool isolation exists specifically to prevent one dependency exhausting a shared pool. Hystrix is now in maintenance mode, with Resilience4j and service-mesh implementations preferred, which is worth knowing so you do not recommend it.

AWS's "Exponential Backoff and Jitter" article gives the simulation showing full jitter outperforming equal and decorrelated jitter, which is the basis for the specific recommendation.

Google's SRE book, chapter 22 ("Addressing Cascading Failures"), documents retry amplification and introduces the retry budget as a fleet-level guard, which is the mechanism most implementations omit.

Envoy and Istio implement circuit breaking, outlier detection, retries with budgets and concurrency limits at the mesh layer, which moves these patterns out of application code entirely, and their configuration surface is a good reference for what a complete implementation includes.

Resilience4j is the current library reference on the JVM and its semaphore-based bulkhead is the cheap default that the virtual-thread era makes sufficient.

The debate

The case for implementing all four in the application: the application knows what is retryable, what a sensible fallback is, and what the deadline budget means. A mesh cannot know that a recommendations failure should render an empty module rather than a 500.

The case for the service mesh: it applies uniformly without every team implementing it, it is language-agnostic, and it can be tuned in production without a deploy. For circuit breaking, outlier detection and concurrency limiting specifically, the mesh does it well.

The case against circuit breakers generally: they are a source of incidents themselves. A misconfigured breaker trips on a transient blip and takes a healthy dependency out of service, and the half-open state can re-overload something that was recovering. Some organisations use only timeouts, bulkheads and load shedding for exactly this reason.

My position: timeouts and bulkheads always, in the application; circuit breaking and outlier detection in the mesh; retries with a fleet-level budget; and a fallback that is a product decision.

The one I would prioritise is the bulkhead, because it is the one people skip and the one that would have contained the incident above on its own. A circuit breaker does not help until it trips, and a bulkhead protects you during the detection window, which is exactly when the pool is being consumed. Bounded concurrency per dependency is the single most valuable line of resilience configuration.

On timeouts, the specific discipline is percentile-based rather than round numbers: p99.9 of healthy latency plus margin, not one second because it looks reasonable. And deadline propagation, because without it a downstream service spends two seconds on a request whose caller gave up long ago, which is pure waste during exactly the incident where capacity matters.

On retries, the rule I would insist on is a fleet-level retry budget rather than only a per-request cap. Three attempts per request still lets the whole fleet triple its load on a service that is already failing, which converts a degradation into an outage. Capping retries at ten percent of successful requests bounds the aggregate, which is the thing that actually matters.

And on circuit breakers, two corrections to the common implementation: trip on a failure rate over a rolling window rather than consecutive failures, because a 50 percent failure rate never trips a consecutive counter; and trip on latency, not only errors, because the realistic failure is slow rather than dead and a dependency answering 200s in eight seconds passes every error-based breaker.

Where I would push back on the framing: these patterns bound the damage from a dependency you must call, and the better question is usually whether you must call it. In the worked example the patterns made the failure survivable, and moving an optional upsell module off the synchronous checkout path made it irrelevant. That is the larger fix and it is an architecture question rather than a configuration one.

Follow-up Q&A

"Walk me through a cascading failure." A dependency slows rather than fails, which is the dangerous case. Your calls that returned in 20 milliseconds now hold for 6 seconds, and by Little's Law in-flight concurrency is throughput times latency, so at 200 requests a second that goes from 4 to 1,200 against a pool of 200. The pool is exhausted in seconds, and now every endpoint fails including the ones that never call that dependency. Then callers time out and retry, tripling the load. One optional dependency took the whole service down.

"Which of the four patterns would have prevented it?" The bulkhead, on its own. Thirty threads reserved for recommendations means thirty threads get held and a hundred and seventy remain, so checkout is unaffected. That is why it is the one I would prioritise and it is the one people skip: a circuit breaker does not help until it trips, and during the detection window the pool is being consumed. The bulkhead protects you the whole time.

"How do you choose a timeout value?" From the dependency's healthy latency distribution, not from a round number: p99.9 plus margin, so if p99.9 is 200 milliseconds the timeout is 250, not one second and certainly not the 30-second default that most HTTP clients ship with. Higher does not save anyone, it just means holding a thread longer before failing. Lower converts healthy slow requests into failures which you then retry, adding load to something that was working.

"What is deadline propagation and why does it matter?" The caller's remaining budget travels with the request, so a service three hops down bounds its own timeout by what is left rather than by its default. Without it, that service works for two seconds on a request whose caller gave up 1.5 seconds ago, which is pure waste consuming capacity during exactly the incident where capacity matters. gRPC has it in the protocol; over HTTP it is a header convention you implement in a client wrapper.

"When is retry dangerous?" Under overload, which is when you most want it. A service at capacity starts erroring, every client retries three times, load triples on something already over capacity, and the retry policy has converted a degradation into an outage. Three rules make it safe: retry only what is genuinely retryable, so never a 400 and only a timeout if the operation is idempotent; exponential backoff with full jitter, because without jitter everyone retries at the same instant and recreates the spike; and a fleet-level retry budget.

"What's a retry budget?" A cap on retries as a fraction of successful requests, typically ten percent over a rolling window, above which you fail immediately without retrying. It is the mechanism most implementations lack, and it matters because a per-request cap of three bounds one client's behaviour and says nothing about the aggregate: the whole fleet can still triple its load. Google's SRE book introduces it for exactly that reason.

"What do circuit breaker implementations get wrong?" Two things. Tripping on consecutive failures, which never fires under a 50 percent failure rate because successes keep resetting the counter, and 50 percent failures is a serious outage. And tripping only on errors, when the realistic failure is slow rather than dead: a dependency returning 200s in eight seconds passes every error-based breaker and exhausts your threads exactly as effectively. So: failure rate over a rolling window with a minimum sample threshold, and a latency trip as well.

"Thread-pool bulkhead or semaphore?" Semaphore, in most modern stacks. A thread-pool bulkhead gives real isolation including the ability to interrupt a blocked call, at the cost of threads and context switches, which is the Hystrix-era design. A semaphore is a permit count on the calling thread, much cheaper, and it cannot interrupt a blocked call so it relies on the client's own timeout. With virtual threads or async I/O the cost objection largely disappears, and bounded concurrency per dependency is cheap and sufficient.

"Application or service mesh?" Both, split by what each knows. Circuit breaking, outlier detection and concurrency limiting go in the mesh, because they apply uniformly, work across languages and can be tuned without a deploy. Timeouts, deadline propagation, retry classification and fallbacks stay in the application, because only it knows that a 422 is not retryable and that a recommendations failure should render an empty module rather than a 500.

"Are circuit breakers ever a bad idea?" Yes, and some organisations skip them deliberately. They are a source of incidents themselves: a misconfigured breaker trips on a transient blip and removes a healthy dependency, and a badly designed half-open state re-overloads something that was recovering. If I had to choose a subset I would take timeouts, bulkheads and load shedding, because those three have no failure mode where they make a healthy system worse.

Common misconceptions

"These four are interchangeable resilience patterns." They defend against different failures. A circuit breaker does nothing until it trips; a bulkhead protects during the detection window.

"Retry makes things more reliable." Under transient failure, yes. Under overload it amplifies and converts a degradation into an outage. It is the only one of the four that can make things worse.

"A circuit breaker on error rate is enough." The realistic failure is slow, not dead. A dependency answering successfully in eight seconds passes every error-based breaker.

"The default timeout is fine." Most HTTP clients default to no timeout, which is not "wait a long time", it is "wait forever".

"Bulkheads cost too many threads." That was the Hystrix-era objection. A semaphore bounding concurrent calls is nearly free, and virtual threads remove the argument entirely.

Interview delivery note

Lead with the cascade, because the mechanism is what justifies all four: "The failure to design for is a dependency getting slow rather than failing. Calls that returned in twenty milliseconds now hold for six seconds, and by Little's Law in-flight concurrency is throughput times latency, so at two hundred requests a second that's twelve hundred against a pool of two hundred. The pool is gone in seconds and now every endpoint fails, including the ones that never touched that dependency."

Then name the one that would have prevented it: "The bulkhead, on its own. Thirty threads reserved for that dependency means thirty get held and everything else continues. It's the one people skip, and the reason it matters more than the circuit breaker is that a breaker doesn't help until it trips, and during the detection window the pool is being consumed."

Give the timeout discipline concretely: "Timeouts from the p99.9 of healthy latency, not round numbers, so 250 milliseconds if p99.9 is 200, not the thirty-second default most clients ship. And deadline propagation, because otherwise a service three hops down works for two seconds on a request whose caller gave up 1.5 seconds ago."

Volunteer the retry danger, because it inverts the expectation: "And retry is the only one of the four that can make things worse. Under overload, three retries per client triples load on something already failing. So: full jitter, and a fleet-level retry budget capping retries at about ten percent of successes, because a per-request cap bounds one client and says nothing about the aggregate."

Close with the question that dwarfs the configuration: "Though the question I'd ask first is why an optional upsell module was a synchronous blocking call in the checkout path at all. The patterns made that failure survivable; moving it off the critical path made it irrelevant, and that mattered more."

The Staff & Team Lead Interview Reference