Deadline propagation and cancellation across a call tree

What it is

A deadline is an absolute point in time by which a request must be answered. A timeout is a duration measured from when a call starts. The difference sounds pedantic and is the entire subject of this page:

Timeout:  "give up after 2 seconds"     -> each hop starts its own 2-second clock
Deadline: "give up at 14:32:07.412 UTC" -> every hop shares the same clock

Deadline propagation is passing that absolute time down the call tree, so a service that receives a request with 300 milliseconds remaining knows it has 300 milliseconds, not 2 seconds, and passes its own remaining budget to whatever it calls.

Cancellation is the other half: when the deadline passes, or when the client goes away, the signal propagates down so downstream work stops rather than continuing to consume resources for a response nobody will read.

What this is confused with: retries and timeouts are frequently discussed together and deadline propagation is what makes them compose safely. Without it, a three-level call tree where each level has a 2-second timeout and 3 retries can legitimately take 2 * 3 * 3 * 3 = 54 seconds while the user's browser gave up at 5.

The problem it solves

Timeout multiplication. Consider a gateway calling service A, which calls B, which calls C. Each has a sensible-looking 2-second timeout and 2 retries.

gateway  ──2s──▶  A  ──2s──▶  B  ──2s──▶  C
                  x3          x3          x3

Worst-case work performed by C: 3 attempts. B calls C up to 3 times, so 9. A calls B up to 3 times, so 27 C-attempts for one user request. The gateway gave up after 2 seconds; the other 26 attempts are pure waste, executed against a system that is already struggling, which is why it was slow in the first place. Retry amplification is multiplicative in tree depth, and it is the mechanism behind a large fraction of cascading failures.

Wasted work after the client left. A user hits back, or the browser times out, or the mobile app is backgrounded. The connection closes. Without cancellation propagation, the gateway is still waiting on A, which is still waiting on B, which is running a 4-second query. Every one of those holds a thread, a connection and a database session. Under load, a service can spend the majority of its capacity computing answers to questions nobody is listening to.

Nobody can reason about the total. The most practical problem: when each hop has its own timeout, no single place states what a request's latency budget is. Someone sets a 30-second timeout on an internal call "to be safe," and the user-facing SLO of 1 second is now unenforceable, and nobody notices until an incident.

Mechanics

The rule

On receiving a request:
    remaining = deadline_from_request - now()
    if remaining <= 0: fail fast with DEADLINE_EXCEEDED, do NO work

When making an outbound call:
    child_deadline = min(my_deadline, now() + my_own_max_for_this_call)
    subtract a small budget for local work and network overhead

On deadline expiry:
    cancel in-flight work, propagate cancellation downstream, return the error

Two details make this work in practice. Subtract a margin for local processing and the return trip, otherwise the deepest service uses the entire remainder and the response cannot get back in time. A common approach is to pass down remaining - (expected_local_work + network_margin), often 5 to 10 percent.

Fail fast on arrival. If a request arrives already expired, do nothing. This is where deadline propagation pays for itself under overload: a queue of expired requests is drained instantly rather than executed, which is load shedding you get for free.

gRPC: built in

gRPC is the reference implementation because deadlines are part of the protocol, not a convention. The client sets a deadline; it travels as the grpc-timeout header; the server sees it on the Context.

// Client: a DEADLINE, not a timeout. Absolute from this moment.
InventoryResponse r = stub
    .withDeadlineAfter(300, MILLISECONDS)
    .checkStock(request);

// Server: read the remaining budget and pass a reduced one downstream.
@Override
public void checkStock(StockRequest req, StreamObserver<StockResponse> obs) {
    Context ctx = Context.current();
    Deadline deadline = ctx.getDeadline();

    if (deadline != null && deadline.isExpired()) {
        obs.onError(Status.DEADLINE_EXCEEDED
            .withDescription("expired on arrival").asRuntimeException());
        return;                                   // do NO work
    }

    // gRPC propagates the deadline automatically to outbound stubs made
    // within this Context. This call inherits the REMAINING time.
    WarehouseResponse w = warehouseStub.query(toWarehouseRequest(req));

    obs.onNext(toResponse(w));
    obs.onCompleted();
}

The automatic inheritance is the important part: an outbound gRPC call made inside the server's Context gets the remaining deadline without the developer doing anything. Deadline propagation that requires every developer to remember it does not survive contact with a codebase.

gRPC also propagates cancellation. If the client cancels or the deadline expires, the server's Context is cancelled, and a listener fires:

Context.current().addListener(ctx -> {
    // Client is gone or the deadline passed. Stop work now.
    queryHandle.cancel();
    metrics.increment("work.cancelled");
}, executor);

HTTP: convention, because there is no standard

HTTP has no deadline header in the base specification. Three approaches exist and you should pick one and enforce it:

A custom header carrying either the absolute deadline or the remaining milliseconds. Remaining-milliseconds is more common because it avoids clock skew:

// Outbound
long remaining = deadline.timeRemaining(MILLISECONDS);
long forChild = Math.max(0, remaining - LOCAL_BUDGET_MS);
request.header("X-Request-Deadline-Ms", String.valueOf(forChild));

Envoy's x-envoy-expected-rq-timeout-ms, which a service mesh sets automatically from the route's configured timeout, and which downstream services can read. This is the best option when you already run a mesh, because it is set by infrastructure rather than by every application correctly.

The gRPC-Web / Connect grpc-timeout header, which carries a duration string like 300m (300 milliseconds), if you are using those protocols over HTTP.

The clock-skew point is worth stating: absolute deadlines across machines require synchronised clocks. With NTP-managed clocks and skew in the low tens of milliseconds, absolute deadlines are fine for budgets measured in hundreds of milliseconds. Below that, or across networks where you do not control the clocks, relative remaining-time is safer. gRPC uses relative (grpc-timeout is a duration) for exactly this reason and converts to an absolute deadline locally.

Cancellation in the JVM

Three mechanisms, and the difference matters:

// 1. Reactor: cancellation is built into the Subscription contract.
//    A client disconnect cancels the subscription and the chain unwinds.
return webClient.get().retrieve().bodyToMono(Result.class)
    .timeout(Duration.ofMillis(remaining))
    .doOnCancel(() -> log.info("cancelled, client went away"));

// 2. Virtual threads / structured concurrency (Java 21+):
//    scope shutdown interrupts every subtask.
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
    var inventory = scope.fork(() -> inventoryClient.check(sku));
    var pricing   = scope.fork(() -> pricingClient.get(sku));
    scope.joinUntil(deadline.toInstant());    // deadline, not timeout
    scope.throwIfFailed();
    return combine(inventory.get(), pricing.get());
}   // scope close interrupts anything still running

// 3. Plain Future: cancel(true) interrupts, and the task must cooperate.
future.cancel(true);

scope.joinUntil(Instant) is genuinely deadline-shaped rather than timeout-shaped, which is a small sign that structured concurrency was designed with this in mind.

The critical caveat for all three: cancellation is cooperative below the API boundary. Interrupting a thread blocked in a JDBC query does not stop the database from executing it. Statement.cancel() sends a cancel request to the server, and whether the server honours it promptly depends on the database. So the resource you most want to reclaim (a 4-second query) is the one hardest to cancel, and the honest mitigation is a server-side statement timeout:

-- Postgres: enforce the budget where the work actually happens.
SET LOCAL statement_timeout = '250ms';

Setting the database's own timeout from the remaining deadline is the only way to make cancellation real for query work, and it is the detail that separates a design that works from one that only propagates the header.

Budget allocation across a tree

A user-facing budget must be divided. The allocation is a design decision:

User-facing SLO:                        1000ms p99
  minus client render + network:         -150ms
Gateway budget:                           850ms
  minus gateway overhead:                 -30ms
  ├─ auth check (parallel)                 50ms
  ├─ product service                      400ms
  │    ├─ database                        250ms   (statement_timeout)
  │    └─ cache                            10ms
  └─ recommendations (parallel, optional) 200ms   ← degrade, do not fail

Two things to note. Parallel calls share the budget rather than consuming it in sequence, so the branch cost is the max, not the sum, which is a strong argument for parallelising independent calls. And optional calls should degrade rather than fail: recommendations timing out should return an empty list, not a 500. Marking each downstream as required or optional at design time is what makes graceful degradation implementable rather than aspirational.

A worked example: 54 seconds of work for a 5-second request

A checkout flow. Gateway calls an order service, which calls inventory and payment; inventory calls a warehouse service.

Original configuration, each set independently by the team that owned it:

gateway    -> order:      timeout 10s,  retries 2
order      -> inventory:  timeout  5s,  retries 3
inventory  -> warehouse:  timeout  3s,  retries 3
order      -> payment:    timeout  8s,  retries 2

Every one is defensible in isolation. Together, the worst case for warehouse work generated by one user request:

warehouse attempts = 3 (inventory retries)
                   x 3 (order retries to inventory... wait, 3 retries = 4 attempts)

Being precise, because this is where the arithmetic usually goes wrong: "3 retries" means up to 4 attempts.

inventory -> warehouse:  4 attempts x 3s  = 12s, 4 warehouse calls
order -> inventory:      4 attempts x 12s = 48s, 16 warehouse calls
gateway -> order:        3 attempts x 48s = 144s, 48 warehouse calls

Forty-eight warehouse calls, up to 144 seconds of work, for a request whose user had a 5-second browser timeout. The gateway's own 10-second timeout capped what the gateway waited for, and capped nothing downstream, because nothing downstream knew the gateway had stopped waiting.

The incident. The warehouse service degraded, going from 40 ms to 2.5 seconds.

09:41  warehouse p99: 40ms -> 2.5s
09:42  inventory begins retrying; warehouse load 3.1x normal
09:43  order begins retrying inventory; warehouse load 11x normal
09:44  gateway begins retrying order; warehouse load 34x normal
09:45  warehouse fully saturated, all requests timing out
09:46  order service thread pools exhausted (threads parked on inventory)
09:47  checkout fully unavailable
10:20  warehouse scaled 4x, load subsided, recovery

The warehouse's own degradation was 60x; the retry amplification made it 34x worse. A partial degradation that customers would have experienced as slowness became a total outage of checkout, and the amplification came entirely from the retry tree.

The redesign.

// 1. A single budget, set once at the edge, from the user-facing SLO.
public Mono<CheckoutResult> checkout(CheckoutRequest req) {
    Deadline deadline = Deadline.after(3, SECONDS);       // browser gives up at 5
    return orderClient.place(req, deadline);
}

// 2. Every service reads its remaining budget and passes a reduced one.
public Mono<Order> place(OrderRequest req, Deadline deadline) {
    if (deadline.isExpired()) return Mono.error(new DeadlineExceeded());

    Deadline childDeadline = deadline.minus(LOCAL_OVERHEAD);

    // Parallel: they SHARE the budget rather than consuming it in sequence.
    return Mono.zip(
            inventoryClient.check(req.sku(), childDeadline),
            paymentClient.authorize(req.payment(), childDeadline))
        .timeout(deadline.remaining())
        .map(t -> buildOrder(t.getT1(), t.getT2()));
}

// 3. Retries are budget-aware: retry only if there is time left for another try.
private RetryBackoffSpec retryWithin(Deadline deadline) {
    return Retry.backoff(3, Duration.ofMillis(50))
        .filter(this::isRetryable)
        .filter(e -> deadline.remaining().toMillis() > ESTIMATED_ATTEMPT_MS);
}

Plus a fleet-level retry budget (retries capped at 10 percent of successful request volume, as in resilience patterns), circuit breakers on each client, and statement_timeout set from the remaining deadline on every database call.

Verified with the same fault injected in staging (warehouse at 2.5s):

                                    before        after
warehouse calls per user request    up to 48      up to 3
worst-case total latency            144s          3.0s (hard cap)
warehouse load multiplier at
  onset of degradation              34x           1.4x
checkout availability during
  warehouse degradation             0%            94% (6% fail fast, correctly)
p99 checkout latency                timeout       2.9s
wasted work (responses computed
  after client gave up)             ~40% of load  ~0%

The 6 percent failure rate is the design working. Those requests genuinely could not be served within the budget, and failing them fast at 3 seconds is strictly better than holding resources for 144 seconds to fail anyway. The capacity those 6 percent would have consumed is what kept the other 94 percent working, which is the argument for deadlines in one sentence.

The "wasted work" line is the one that surprised the team most: about 40 percent of the order service's load during the incident was computing responses for connections that had already closed.

Production evidence

gRPC has deadlines in the protocol (grpc-timeout), with automatic propagation through Context and automatic cancellation, and Google's gRPC documentation states that services should always set deadlines and that a missing deadline is a bug. That this is protocol-level rather than convention-level is the strongest available evidence that it needs to be infrastructure rather than discipline.

Google's internal practice, described in the SRE book's chapter on handling overload, is deadline propagation with a shared budget across the RPC tree, plus the explicit practice of checking whether the deadline has already expired on arrival and doing no work if so. The book also documents the retry-amplification arithmetic and their per-server retry budget (retries capped as a fraction of request volume).

Envoy sets x-envoy-expected-rq-timeout-ms from the route timeout, and its retry policy includes per_try_timeout distinct from the overall route timeout, which is the mesh-level acknowledgement that "timeout" is ambiguous between an attempt and a request.

Netflix's Hystrix and its successors were built around the observation that timeouts without a shared budget do not bound total latency, and their bulkhead plus timeout plus circuit breaker combination exists because none of the three alone prevents cascading failure.

Go's context.Context makes this idiomatic at the language level: ctx is the first parameter of essentially every I/O function in the standard library, carrying both a deadline and a cancellation channel. The fact that Go made it a convention strong enough to appear in every function signature, and that this is widely regarded as one of Go's better decisions, is a useful counterpoint to the JVM ecosystem where it remains optional.

The debate

Deadlines or timeouts? Deadlines, for anything crossing a service boundary. A timeout per hop cannot bound the total, and the total is what the user experiences. The argument for timeouts is simplicity: no header to propagate, no clock concerns, and every HTTP client supports them out of the box. My position: a deadline set once at the edge and propagated, with per-hop timeouts as a secondary guard for the case where propagation fails or a call is made outside the propagated context. Both, with the deadline authoritative.

Absolute time or remaining duration on the wire? Remaining duration, because it is immune to clock skew. Convert to an absolute deadline locally on receipt so that local elapsed time is subtracted correctly. This is what gRPC does and the reasoning is sound: you cannot control the clocks of everything in a call tree, especially across organisational boundaries.

Where should the budget be set? At the edge, from the user-facing SLO, and nowhere else. The failure mode to avoid is each service picking its own number, which produces the 54-second situation. A concrete practice: make the deadline a required parameter in your internal client libraries, so a call without one does not compile. Optional propagation degrades to no propagation within about two quarters.

Is retry-with-deadline enough, or do you need retry budgets too? Both. Deadline-aware retries bound the latency of one request tree. They do not bound the load: under widespread degradation every request is retrying within its budget, and the aggregate is still multiplied. A fleet-level retry budget (retries as a percentage of successful requests, typically 10 percent) is what bounds the load, and it is the one that prevents the retry storm from becoming the outage.

When is deadline propagation not worth it? Fire-and-forget work and asynchronous pipelines. A message consumer processing from a queue has no waiting client, so there is no deadline to inherit; what it needs is a processing timeout and a dead letter queue. Trying to apply request deadlines to batch or streaming work is a category error and produces confusing code.

Follow-up Q&A

"Why is a per-hop timeout insufficient?"

Because timeouts compose multiplicatively and deadlines compose by intersection. Three levels at 2 seconds with 2 retries each is up to 54 seconds of work for a request the client abandoned at 5. Each timeout bounds one call; nothing bounds the tree. A deadline set once and propagated means every hop is working against the same absolute instant, so the total is bounded by construction and the deepest service knows it has 80 milliseconds left rather than a fresh 2 seconds.

"How do you handle clock skew?"

Put the remaining duration on the wire, not an absolute timestamp, and convert to an absolute deadline locally on receipt. Then only local elapsed time matters and clock differences between machines are irrelevant. This is what gRPC's grpc-timeout does. If you must send absolute times, NTP-managed skew in the tens of milliseconds is fine for budgets of hundreds of milliseconds and not fine for tight ones.

"What actually happens when you cancel?"

Depends how deep it needs to go, and this is where implementations disappoint. In Reactor, cancellation propagates through the subscription chain and unwinds cleanly. With structured concurrency, closing the scope interrupts the subtasks. Below the API boundary it is cooperative: interrupting a thread blocked in a JDBC query does not stop the database, which needs Statement.cancel() and a server willing to honour it promptly. So the resource you most want to reclaim is the hardest to cancel, and the practical answer is to set the database's own statement_timeout from the remaining deadline, enforcing the budget where the work happens rather than where you are waiting for it.

"How do you allocate a budget across a tree?"

Start from the user-facing SLO, subtract client rendering and network, and that is the edge budget. Then subtract each hop's own overhead as you descend. Parallel calls share the budget rather than consuming it sequentially, so the cost of a parallel branch is the maximum of its children, not the sum, which is a real argument for parallelising independent calls. And mark each downstream as required or optional: an optional one that exceeds its share should degrade to a default rather than failing the request.

"A service receives a request that is already expired. What should it do?"

Return DEADLINE_EXCEEDED immediately and do no work. This is the highest-value line of code in the whole pattern, because under overload the queue is full of requests whose clients have gone, and draining them instantly instead of executing them is load shedding you get for free. In the worked example roughly 40 percent of the order service's load during the incident was work for closed connections.

"Deadlines are propagating and you still have a retry storm. What is missing?"

A fleet-level retry budget. Deadline-aware retries bound one request tree's latency; they do not bound aggregate load, because under widespread degradation every request is independently retrying within its own budget and the total against the failing service still multiplies. Cap retries at a fraction of successful request volume, typically 10 percent, measured per client fleet. Plus circuit breakers, so a sustained failure stops generating attempts at all rather than continuing to retry within budget.

Common misconceptions

"Setting a timeout on every call is enough." It bounds each call and not the tree. The total is the product of the depths and the retry counts, and no per-hop number constrains it.

"The client's timeout protects the backend." It protects the client. When the gateway stops waiting, every downstream service continues working unless cancellation propagates, so the load is unchanged and the results are discarded.

"Cancellation is automatic." It is automatic within a framework that implements it (gRPC contexts, Reactor subscriptions, structured concurrency scopes) and cooperative below that. A thread interrupt does not stop a running SQL query, and a running SQL query is usually the expensive thing.

"3 retries means 3 attempts." It means up to 4. This off-by-one runs through every amplification calculation and consistently makes the real number worse than the estimate.

"Deadlines make the system less reliable, because requests fail that would have succeeded." Some do, and they fail fast instead of slowly, releasing capacity that keeps the majority working. In the example, 6 percent failing at 3 seconds is what allowed 94 percent to succeed rather than 100 percent failing at 144 seconds.

Interview delivery note

Say this verbatim: "A timeout is per hop and a deadline is per request. Three levels of 2-second timeouts with 2 retries each is up to 54 seconds of work for a request the client abandoned at 5 seconds, so I set one deadline at the edge from the user-facing SLO and propagate the remaining budget down, and any service that receives an expired request does no work at all." That is the distinction, the arithmetic and the mechanism in one answer.

The senior-versus-staff separator is the fail-fast-on-arrival check. A senior engineer describes propagating a deadline correctly. A staff engineer points out that the highest-value part is checking expiry on arrival and doing nothing, because under overload that turns a queue of abandoned requests into free load shedding, and that in practice a substantial fraction of a struggling service's load is work for connections that already closed.

The second signal is knowing that deadline-aware retries do not bound load, only latency, so you also need a fleet-level retry budget. Distinguishing "this bounds one request tree" from "this bounds aggregate pressure on the failing service" is the difference between having read about retry storms and having been in one.

The Staff & Team Lead Interview Reference