The swarm scheduler: four pieces of plain code
Strip any topology from the previous chapter to its execution layer and the same machine appears: a queue of work units, a fleet of workers, a governor sharing the token quota, and a ledger enforcing the budget. That machine is the swarm scheduler, it is deliberately plain code (the architecture's tenet: judgment in workers, determinism in the platform), and this chapter builds it as a simulator that the next three chapters keep extending. The simulator is the book's second-most reused artifact after Part 1's event store, and like everything T0, it is seeded: every number below reproduces on your machine.
The four pieces
The queue holds units (from the orchestrator's decomposition) and gives the fleet its elasticity: arrival bursts become queue depth, not worker panic. Priorities live here when tenants or urgency classes exist; the simulator keeps FIFO and lets Part 4's SQS mapping carry the fairness discussion.
The workers are slots, not identities: a worker takes a unit, runs the agent loop for it, reports, takes another. Everything an individual worker knows lives in the unit's own event ledger, which is what makes workers disposable, and disposability is what makes Chapter 25's crash story boring.
The governor is Chapter 14's client-side token bucket, promoted to fleet infrastructure: workers draw tokens before calling the model, waiting rather than colliding. In the simulator it is four lines; the AWS-scale distributed version is next chapter's problem.
The ledger makes budgets real at two moments. Admission: a unit is dispatched only if the remaining budget covers its projected cost, so over-demand parks in the queue instead of overspending. Settlement: every token actually spent is decremented, including tokens spent on attempts that failed, because the provider billed them whether or not the work survived. That honesty has consequences the resume lab will make vivid.
Exhaustion is a first-class outcome, not an error: the scheduler
stops admitting, drains what is in flight, and closes the run as
run.exhausted with its partial results intact and its remainder
enumerated. Gate 3 of the production bar,
implemented in an if statement.
Lab 6.1: the fleet, simulated
Four hundred units (8,000 to 16,000 tokens each, about five minutes of worker wall clock), a 300k TPM quota governed at 95%, a 4% per-attempt failure rate with retries, and a 6M token budget:
python3 swarm_sim.py
400 units, 4,797,958 tokens of first-attempt work | quota 300,000 TPM | governor at 95%
--- baseline: 100 workers, budget 6.0M tokens ---
completed 400/400 in 1,746s | spent 4,905,337 tokens | retries 9 | dead-lettered 0
--- the worker-scaling sweep (same quota, same work) ---
workers elapsed note
25 5,091s worker-bound
50 2,815s worker-bound
100 1,746s worker-bound
200 1,604s quota-bound (81 workers surplus)
Little's law crossover: N* = 119 workers; beyond it, the quota is the fleet
--- budget-capped run: same fleet, budget 2.5M tokens ---
run.exhausted at 1,031s: 206/400 units complete, 2,498,987 tokens spent
partial results are usable and the run is resumable (Lab 6.2) once the budget is topped up
Three readings:
- The baseline is what healthy looks like. Every number in that line maps to a dashboard the ops chapters will build: completion, elapsed, spend against budget, retry count, dead letters. A fleet that cannot emit this line is not a fleet; it is a hope.
- The sweep is the chapter's thesis in four rows. Doubling workers from 25 to 50 nearly halves the wall clock; doubling from 100 to 200 buys 8%, because the crossover sits at N* = 119, exactly where the lab catalog's envelope math said to look: quota rate times average service time. Past N*, additional workers are idle capacity waiting politely behind the governor, and the honest responses are a quota increase, cache-shrinking the units, or acceptance. "Run more agents" is an argument about N*, or it is not an argument.
- Exhaustion reads like a landing, not a crash. The capped run stopped admitting at the right moment, drained cleanly, and reported exactly where it stood: 206 units of usable output, 194 enumerated for a future run. Compare the alternative that gate 3 exists to prevent: the same fleet discovering its budget in the invoice.
What the simulator deliberately omits
Naming the simplifications is part of the lab's honesty, and each is a pointer: real fleets have heterogeneous unit costs known only after execution (the admission check uses estimates, so ledgers need small reserves); real workers hold leases that expire on crash (the simulator's workers cannot die until Chapter 25 kills them); real queues carry priorities and tenant fairness (Part 4's skeleton); and the governor here is one variable in one process, which is exactly the gap the next chapter closes at AWS scale.
Don't be confused: the scheduler vs an orchestrator agent. The orchestrator (when one exists) is judgment: a model deciding how to split work and what it means. The scheduler is arithmetic: queues, buckets, and a ledger, deterministic and cheap, running the same whether the units came from a brilliant decomposition or a dumb one. Blending them (a model deciding, mid-loop, what runs next and what it may spend) produces a system whose throughput and spending are as reproducible as its prompts, which is to say not. Every fleet in this book keeps the split absolute.
Full source
"""Lab 6.1: the swarm scheduler. A fleet in one file, no cloud.
Simulates the machinery every agent fleet reduces to, whatever its
topology: a queue of work units, a fleet of workers, a shared token
quota behind a governor, and a budget ledger doing admission control.
The scheduler is plain code; all judgment stays inside the (simulated)
worker agents.
The demo runs three experiments:
1. a baseline fleet run, with the full stats a real run would emit,
2. the worker-scaling sweep: same quota, more workers, and the wall
where Little's law says extra workers stop mattering,
3. a budget-capped run: exhaustion as a clean, resumable partial
result instead of a surprise.
Also importable by Lab 6.2 (kill and resume). Standard library only.
Deterministic: same run, same numbers.
"""
from __future__ import annotations
import math
import random
QUOTA_TPM = 300_000 # account tokens-per-minute quota
REFILL = QUOTA_TPM / 60 # 5,000 tokens/second
GOVERNOR_SHARE = 0.95 # the fleet self-limits to 95% of quota
UNITS = 400
MAX_ATTEMPTS = 3 # then the unit is dead-lettered
SEED = 0
def unit_tokens(unit: int) -> int:
"""Token appetite of one work unit (model calls, all turns)."""
return random.Random(f"{SEED}:{unit}").randint(8_000, 16_000)
def attempt_fails(unit: int, attempt: int) -> bool:
"""Deterministic per (unit, attempt): ~4% of attempts fail."""
return random.Random(f"{SEED}:{unit}:{attempt}").random() < 0.04
def service_secs(tokens: int) -> int:
"""Wall-clock seconds one worker spends on the unit (~5 min avg)."""
return 180 + tokens // 100
def run_fleet(workers: int, budget: int,
kill_at: int | None = None,
resume: dict | None = None) -> dict:
done = set(resume["done"]) if resume else set()
dead = set(resume["dead"]) if resume else set()
spent = resume["spent"] if resume else 0
queue = [u for u in range(UNITS) if u not in done and u not in dead]
attempt = {u: 0 for u in range(UNITS)}
in_flight: list[tuple[int, int, int]] = [] # (unit, finish_t, attempt)
bucket = 0.0
retries = exhausted = 0
t = 0
while (queue or in_flight) and t < 50_000:
t += 1
if kill_at is not None and t >= kill_at:
return {"killed_at": t, "done": sorted(done),
"dead": sorted(dead), "spent": spent,
"in_flight": len(in_flight)}
bucket = min(bucket + REFILL * GOVERNOR_SHARE, QUOTA_TPM)
still = []
for unit, finish, att in in_flight: # collect finished units
if t < finish:
still.append((unit, finish, att))
elif attempt_fails(unit, att): # tokens spent, work lost
if att + 1 >= MAX_ATTEMPTS:
dead.add(unit)
else:
queue.append(unit)
retries += 1
else:
done.add(unit)
in_flight = still
while queue and len(in_flight) < workers:
tokens = unit_tokens(queue[0])
if spent + tokens > budget: # admission control
exhausted, queue = 1, []
break
if bucket < tokens: # governor: wait, not 429
break
unit = queue.pop(0)
bucket -= tokens
spent += tokens
in_flight.append((unit, t + service_secs(tokens),
attempt[unit]))
attempt[unit] += 1
return {"elapsed": t, "done": sorted(done), "dead": sorted(dead),
"spent": spent, "retries": retries, "exhausted": exhausted}
def crossover_workers() -> int:
"""Little's law: workers the quota can actually feed."""
avg_tok = sum(unit_tokens(u) for u in range(UNITS)) / UNITS
avg_secs = sum(service_secs(unit_tokens(u))
for u in range(UNITS)) / UNITS
units_per_min = REFILL * GOVERNOR_SHARE * 60 / avg_tok
return math.ceil(units_per_min * avg_secs / 60)
if __name__ == "__main__":
need = sum(unit_tokens(u) for u in range(UNITS))
print(f"{UNITS} units, {need:,} tokens of first-attempt work | "
f"quota {QUOTA_TPM:,} TPM | governor at "
f"{int(GOVERNOR_SHARE * 100)}%")
r = run_fleet(workers=100, budget=6_000_000)
print(f"\n--- baseline: 100 workers, budget 6.0M tokens ---")
print(f"completed {len(r['done'])}/{UNITS} in {r['elapsed']:,}s | "
f"spent {r['spent']:,} tokens | retries {r['retries']} | "
f"dead-lettered {len(r['dead'])}")
print(f"\n--- the worker-scaling sweep (same quota, same work) ---")
print(f"{'workers':>8}{'elapsed':>10} note")
n_star = crossover_workers()
for n in (25, 50, 100, 200):
r = run_fleet(workers=n, budget=6_000_000)
note = "worker-bound" if n < n_star else \
f"quota-bound ({n - n_star} workers surplus)"
print(f"{n:>8}{r['elapsed']:>9,}s {note}")
print(f"Little's law crossover: N* = {n_star} workers; "
f"beyond it, the quota is the fleet")
r = run_fleet(workers=100, budget=2_500_000)
print(f"\n--- budget-capped run: same fleet, budget 2.5M tokens ---")
print(f"run.exhausted at {r['elapsed']:,}s: {len(r['done'])}/{UNITS} "
f"units complete, {r['spent']:,} tokens spent")
print("partial results are usable and the run is resumable "
"(Lab 6.2) once the budget is topped up")
👉 Next: fan-out on AWS, where the same four pieces map onto distributed map, queue-fed fleets, and managed agent sessions, and the one-variable governor learns to be shared by five hundred workers on different machines.