Throughput engineering: sharing the pipe

One agent calling one model is a client. A fleet calling one model is a crowd, and the concepts chapter's quota (the provider's tokens-per-minute speed limit) becomes the resource the whole platform is really scheduling. This chapter decodes how the limit is enforced, manufactures the classic failure (the 429 storm) in a lab, abolishes it with thirty lines of client-side discipline, and then tours the other levers (provisioned capacity, batch, routing) that change the pipe instead of sharing it better.

The two meters

A Bedrock quota is per model family, per region, per account, and it runs two meters at once:

• RPM, requests per minute: a count of calls, regardless of size.
• TPM, tokens per minute: the token volume those calls carry.

Either meter can trip. A chat product with thousands of small calls hits RPM long before TPM; an agent fleet re-sending 40,000-token transcripts hits TPM at a modest request count. Fleet arithmetic (the lab catalog's ceilings) should always be run against both, with the binding one identified by name, because the remedies differ: RPM pressure wants batching-up of work per call, TPM pressure wants caching and shorter transcripts. One subtlety worth knowing: on some platforms, cached tokens count differently against quota than fresh ones, which can make caching a throughput lever as well as a cost one; check the current accounting before assuming either way.

Enforcement, as the decoder ring would predict, is bucket-shaped: capacity that drains as you consume and refills continuously. The bucket's depth is your burst allowance (how hard you can slam it for a few seconds); the refill is your sustained rate. A request the bucket cannot cover is rejected with HTTP 429, costing you a round trip and, at scale, something worse than a round trip, as the lab is about to show.

Anatomy of a storm

The failure mode is emergent, which is why teams meet it in production rather than in design review. Forty workers finish their first units and fire again, greedily. The bucket empties; the surplus calls get 429s. Every SDK's default response is exponential backoff, individually sensible; collectively, the rejected workers now sleep while quota refills unclaimed, then wake in loose synchrony and slam the bucket again. Attempts multiply, throughput stutters, and, the genuinely expensive part, everything else on the account shares the pain, because the quota is account-wide: the fleet's storm rains on the chat product, the batch scorer, and every other team.

The production bar's gate 7 named the cure: the fleet must be a deliberate citizen of the quota, shaping its own demand instead of discovering the limit by collision. Concretely, run the provider's bucket on your own side: a shared client-side token bucket, sized below the quota, that workers must draw from before calling. No tokens in hand, no request sent; the provider never says no because nobody asks for what is not there.

Lab 2.2: the storm, and its absence

The lab simulates both strategies against an identical workload: 300 work units (1.2M tokens) across 40 workers, a 200,000 TPM quota, and, because the account-wide blast radius is the real lesson, a bystander: another team's small workload on the same account, trying one 2,000-token request every 10 seconds. The governed strategy sizes its bucket at 90% of quota, deliberately leaving headroom. Everything is seeded and deterministic.

python3 token_bucket.py

quota 200,000 tokens/min | 40 workers | 300 units, 1,214,260 tokens total
quota-bound lower bound: 364s (no strategy can beat this)

strategy 'greedy'
  attempts  800   429s  500   completed 300/300 in 383s
  bystander (2,000 tok every 10s): 27/39 requests served (69%)
strategy 'governed'
  attempts  300   429s    0   completed 300/300 in 405s
  bystander (2,000 tok every 10s): 41/41 requests served (100%)

Read the three comparisons separately, because they carry different lessons:

• Attempts: 800 versus 300. The greedy fleet made 2.7 requests per unit of work; five hundred of them were pure waste, each a network round trip, a log line, and a nudge to the provider that this account cannot pace itself. The governed fleet asked exactly once per unit.
• The bystander: 69% versus 100%. This is the number that gets escalated. The greedy fleet ran the bucket dry continuously, so the other team's little requests found empty quota one time in three, and nothing in that team's dashboards explains why. The governed fleet's 10% headroom served the bystander perfectly. Quota storms are how one team's batch job becomes another team's incident.
• Time: 383 versus 405 seconds. The greedy fleet was actually faster, and honesty about that matters: with quota saturated either way, brute force loses little wall clock. The governed fleet paid 22 seconds (6%) for zero waste and an intact account. That is the citizenship trade, and at 3 a.m., when the question is "why is the chat product timing out," it is obviously correct. Note the floor, too: 364 seconds is the quota-bound minimum from the lab catalog's arithmetic, and both strategies orbit it, because no client cleverness outruns the refill rate.

One production note the simulation glosses: its governed bucket is a shared variable, which works inside one process. A fleet of hundreds of workers across machines needs the bucket to be genuinely shared (a fast central counter, or per-worker slices with rebalancing), and naive per-worker division (quota over N) wastes the burst capacity the bucket exists to provide. Part 6 builds the distributed version into the swarm scheduler, alongside its cousin, admission control: the governor paces requests already in flight, admission decides at the front door whether new work should enter at all.

Changing the pipe instead

Sharing the pipe better has a ceiling: the pipe. Four levers change the pipe itself, each with a distinct shape:

The composition is the design: interactive agents governed on the on-demand quota, offline phases exiled to batch, a provisioned floor under anything with an SLA, and the portfolio shrinking the demand before any of it. The reference architecture's model plane box, drawn with a governor inside it, was this chapter in miniature.

Don't be confused: quota versus capacity. A quota increase raises what you are allowed to consume; it manufactures no silicon. In a constrained region, a granted increase can still meet elevated error rates at peak, which is what cross-region and global inference profiles (Chapter 11) exist to absorb, and what provisioned throughput exists to make contractual. Allowed and available are different words on purpose.

Full source

"""Lab 2.2: the token bucket. Reproduce a 429 storm, then remove it.

A fleet of workers shares one model quota (tokens per minute). The
provider enforces the quota with something bucket-shaped on its side:
requests that would overdraw it are rejected with HTTP 429.

Two client strategies against the identical quota and workload:
  greedy   : every idle worker fires immediately; on a 429 it backs off
             exponentially and retries (the SDK-default behavior).
  governed : workers draw tokens from a shared client-side bucket sized
             at 90% of the quota, and only fire once tokens are in hand,
             so the server never says no and 10% headroom remains.

The quota is account-wide, so the simulation also runs a bystander: a
small interactive workload (another team's app on the same account)
that tries a 2,000-token request every 10 seconds. Whether the fleet
leaves the bystander any quota is the difference that matters.

Simulated clock, seeded sizes: deterministic, same run, same numbers.
"""

from __future__ import annotations

import random

QUOTA_TPM = 200_000                 # the account's tokens-per-minute quota
REFILL_PER_S = QUOTA_TPM / 60       # ~3,333 tokens/second
GOVERNOR_SHARE = 0.90               # the fleet self-limits to 90% of quota
WORKERS = 40
UNITS = 300                         # work units the fleet must finish
BYSTANDER_TOKENS = 2_000            # another team's request, every 10s
HORIZON_S = 3_600                   # give up after a simulated hour


def unit_sizes() -> list[int]:
    rng = random.Random(0)
    return [rng.randint(2_500, 5_500) for _ in range(UNITS)]


def service_secs(tokens: int) -> int:
    """Wall-clock seconds the model spends serving a granted request."""
    return 3 + tokens // 2_000


def simulate(strategy: str) -> dict:
    sizes = unit_sizes()
    next_unit = 0
    server_bucket = REFILL_PER_S                   # starts with 1s of quota
    client_bucket = 0.0                            # governed strategy only
    workers = [{"busy_until": 0, "backoff": 0, "retry_at": 0, "unit": None}
               for _ in range(WORKERS)]
    attempts = rejections = done = 0
    by_ok = by_fail = 0

    for now in range(HORIZON_S):
        server_bucket = min(server_bucket + REFILL_PER_S, QUOTA_TPM)
        client_bucket = min(client_bucket + REFILL_PER_S * GOVERNOR_SHARE,
                            QUOTA_TPM * GOVERNOR_SHARE)

        for w in workers:                          # the fleet acts first
            if now < w["busy_until"] or now < w["retry_at"]:
                continue
            if w["unit"] is None:
                if next_unit >= UNITS:
                    continue
                w["unit"] = next_unit
                next_unit += 1
            tokens = sizes[w["unit"]]

            if strategy == "governed":
                if client_bucket < tokens:         # wait; no attempt made
                    continue
                client_bucket -= tokens

            attempts += 1
            if server_bucket >= tokens:            # the provider grants it
                server_bucket -= tokens
                w["busy_until"] = now + service_secs(tokens)
                w["backoff"] = 0
                w["unit"] = None
                done += 1
            else:                                  # HTTP 429
                rejections += 1
                w["backoff"] = min(w["backoff"] + 1, 6)
                w["retry_at"] = now + 2 ** w["backoff"]

        if done < UNITS and now % 10 == 0:         # the bystander tries
            if server_bucket >= BYSTANDER_TOKENS:
                server_bucket -= BYSTANDER_TOKENS
                by_ok += 1
            else:
                by_fail += 1

        if done == UNITS:
            elapsed = now + 1
            break
    else:
        elapsed = HORIZON_S

    return {"strategy": strategy, "attempts": attempts,
            "rejections": rejections, "done": done, "elapsed": elapsed,
            "by_ok": by_ok, "by_fail": by_fail}


if __name__ == "__main__":
    total = sum(unit_sizes())
    print(f"quota {QUOTA_TPM:,} tokens/min | {WORKERS} workers | "
          f"{UNITS} units, {total:,} tokens total")
    print(f"quota-bound lower bound: {total / REFILL_PER_S:,.0f}s "
          f"(no strategy can beat this)\n")
    for strategy in ("greedy", "governed"):
        r = simulate(strategy)
        served = r["by_ok"] + r["by_fail"]
        print(f"strategy '{r['strategy']}'")
        print(f"  attempts {r['attempts']:>4}   429s {r['rejections']:>4}   "
              f"completed {r['done']}/{UNITS} in {r['elapsed']:,}s")
        print(f"  bystander (2,000 tok every 10s): "
              f"{r['by_ok']}/{served} requests served "
              f"({100 * r['by_ok'] // max(served, 1)}%)")

👉 Next: model operations, the last chapter of the model plane: models as configuration, upgrades behind eval gates, and the portfolio discipline that spends Opus tokens only where Opus judgment is the product.