Capstone A: the repository-audit fleet
The flagship. Point the platform at a large codebase overnight; find bugs, security smells, and dead code; return a verified, deduplicated report with file-and-line provenance, under a hard budget, resumable after any failure. This is the platform's hardest common case, which is exactly why it is the capstone that gets an end-to-end simulation: if you can watch every plane cooperate here, the other three capstones are subsets you already understand.
The job and its shape
The job is the repository-audit use case realized: judgment over an unbounded input, parallelizable by file group, with false positives cheap to filter by a second layer of agents. The topology is orchestrator-workers with a verification mesh: a planner shards the repo, a fleet of finder agents audits shards independently, an adversarial panel verifies each finding, and a merger assembles the survivors. The economics are the whole game, an unshaped fleet re-reads the same context thousands of times, so cache discipline and shard design decide whether the night costs tens of dollars or thousands.
Dominant gates: 3 (Budgets), 6 (Resumability), 8 (Evals via verification). The design effort concentrates there.
Lab A: the whole platform, one run
Before any cloud spend, the flagship lab composes every plane into one local, seeded pipeline: shard, fan out under a budget ledger with retries and checkpoints, verify with a blind panel, merge with provenance, and score against the gates. Nothing in it is new machinery; it is the pieces from earlier chapters assembled.
python3 capstone_hive.py
=== Capstone A: repository-audit fleet ===
repo: 40 file groups, 18 planted bugs
--- fan-out (finders under budget + retries) ---
units complete : 40/40
retries / dead : 4 / 0
tokens spent : 1,220,000 / 2,000,000
--- verification mesh (blind panel, dedup first) ---
raw findings : 28 (15 real, precision 53%)
after panel : 16 (15 real, precision 93%, recall 83%)
--- the merged report ---
16 verified findings, each carrying 3 panel votes as provenance
--- production-bar gate checklist ---
gate 3 Budgets admission held spend to 1,220,000 / 2,000,000 tokens
gate 5 Failure semantics 4 retries, 0 dead-lettered; effects idempotent
gate 6 Resumability checkpoints at [10, 20, 30, 40] units; resume re-runs only the unfinished slice
gate 8 Evals verification mesh gated findings: precision 53% -> 93%
gate 9 Observability every unit and vote emitted a span + ledger entry
Read it as the platform's greatest-hits reel, each stage a chapter earning its place:
- Shard and fan out under budget. Forty file groups became forty units; the budget ledger's admission control admitted each only while the budget covered it, and the run finished at 1.22M of its 2M token budget, gate 3 satisfied by arithmetic, not hope.
- Retry and checkpoint. Four finder attempts failed transiently and retried; none became poison; progress checkpointed every ten units, so a crash at unit 35 would resume from 30, not zero. Gates 5 and 6, the failure-semantics machinery, working.
- The verification mesh is the star. The finders' raw output was 53% precision, a coin flip, exactly the coverage-not-truth result finders always give. The blind three-vote panel lifted it to 93% while keeping 83% recall, and every surviving finding carries its three votes as provenance. Gate 8, and the reason a repository audit produces a report a human will actually read rather than dismiss.
- Merge with provenance. Sixteen verified findings, each traceable through the event ledger to the votes and the evidence that produced it. The report is defensible, which is the whole product.
The lab is the book's thesis compiled: shard, fan out under budget, retry and checkpoint, verify adversarially, merge with provenance, and every gate is satisfied by machinery built in earlier chapters, not bolted on here. The capstone did not add capability; it composed it.
The cloud build (T2)
The production version maps the simulation onto AWS, and the fan-out chapter's decision table picks the bundle: this is a batch-shaped run over a known input set, so distributed map is the natural dispatcher, with the repo sharded from S3, finder and verifier agents as the child workflow (on Fargate, since fleet runs exceed Lambda's ceiling), the budget ledger and governor in the worker code, and redrive as the resume button. The model portfolio: a middle-tier model for the finders (volume), a frontier model for the verifier panel (where being wrong is expensive), and the batch tier for the offline-friendly finder passes. Findings and their provenance land in DynamoDB; the merged report in S3.
Costed and torn down. State the numbers before the deploy: for a
repo of N shards at the lab catalog's per-shard
economics, the run is priced up front (finders on the batch tier,
verifier panel at frontier rates, roughly the
cache-shaped figures the catalog computed),
tagged lab:capstone-a, and the teardown deletes by tag with a final
sweep to prove the account is clean. The overnight-run economics are
the point of the whole exercise: the difference between a $30 night and
a $3,000 one is the cache discipline and shard design this book spent
Part 2 and Part 6 teaching.
Full source
"""Capstone A: the repository-audit fleet, end to end, in one simulation.
This is the book's culminating lab: every plane composed into one run.
It shards a repo into work units, dispatches them under a budget ledger
(admission control), runs finder agents that sometimes fail and retry,
checkpoints progress, verifies findings with an adversarial panel,
merges the survivors with provenance, and prints the run against the
twelve production-bar gates.
Each stage is annotated with the chapter it came from. Nothing here is
new machinery; it is the pieces assembled. Standard library only.
Deterministic: same run, same report.
"""
from __future__ import annotations
import random
SEED = 0
FILES = 40 # the repo, in shardable file groups
PLANTED_BUGS = 18 # ground truth, for scoring precision/recall
BUDGET = 2_000_000 # token budget for the whole run (ch23 ledger)
FIND_TOKENS = 22_000 # a finder's cost per unit (all turns)
VERIFY_TOKENS = 3_000 # one refuter vote
FIND_FAIL = 0.06 # per-attempt finder failure (ch25)
MAX_ATTEMPTS = 3
PANEL_VOTES = 3 # refuter panel size (ch26)
CHECKPOINT_EVERY = 10
def rng(*parts) -> random.Random:
return random.Random(":".join(str(p) for p in parts))
def shard(files: int) -> list[int]:
"""ch22: the orchestrator's decomposition into independent units."""
return list(range(files))
def find(unit: int, attempt: int) -> tuple[bool, list[tuple[int, bool]]]:
"""A finder agent over one file group. Returns (ok, findings).
A real bug sits in files 0..PLANTED_BUGS-1; finders are noisy."""
if rng(SEED, "find", unit, attempt).random() < FIND_FAIL:
return False, [] # transient failure -> retry
r = rng(SEED, "found", unit)
findings = []
if unit < PLANTED_BUGS and r.random() < 0.85: # catch the real bug
findings.append((unit, True))
if r.random() < 0.30: # cry wolf
findings.append((unit, False))
return True, findings
def panel_keeps(finding: tuple[int, bool]) -> bool:
"""ch26: a blind refuter panel votes to keep. Real findings survive
~0.9/vote, spurious ~0.15/vote; majority decides."""
unit, is_real = finding
keep_p = 0.90 if is_real else 0.15
kept = sum(rng(SEED, "vote", unit, is_real, v).random() < keep_p
for v in range(PANEL_VOTES))
return kept > PANEL_VOTES // 2
def run() -> dict:
units = shard(FILES)
queue = list(units)
attempts = {u: 0 for u in units}
done, dead, raw_findings = set(), set(), []
spent = 0
retries = exhausted = 0
checkpoints = []
# --- fan-out: finders under admission control (ch23) + retries (ch25) ---
while queue:
unit = queue.pop(0)
if spent + FIND_TOKENS > BUDGET: # admission control
exhausted = 1
queue.insert(0, unit)
break
spent += FIND_TOKENS
attempts[unit] += 1
ok, findings = find(unit, attempts[unit])
if not ok:
if attempts[unit] >= MAX_ATTEMPTS:
dead.add(unit)
else:
queue.append(unit)
retries += 1
continue
done.add(unit)
raw_findings += findings
if len(done) % CHECKPOINT_EVERY == 0: # ch25 checkpoint
checkpoints.append(len(done))
# --- dedup before the expensive panel (ch26) ---
raw_findings = sorted(set(raw_findings))
real_raw = sum(1 for _, r in raw_findings if r)
# --- verification mesh: blind panel over each finding (ch26) ---
kept = [f for f in raw_findings if panel_keeps(f)]
spent += len(raw_findings) * PANEL_VOTES * VERIFY_TOKENS
real_kept = sum(1 for _, r in kept if r)
return {
"units": len(units), "done": len(done), "dead": len(dead),
"retries": retries, "exhausted": exhausted, "spent": spent,
"checkpoints": checkpoints,
"raw_findings": len(raw_findings), "real_raw": real_raw,
"kept": len(kept), "real_kept": real_kept,
}
def gate_checklist(r: dict) -> list[tuple[str, str]]:
"""The twelve production-bar gates (ch1), as this run satisfies them."""
return [
("3 Budgets", f"admission held spend to {r['spent']:,} / "
f"{BUDGET:,} tokens" + (" (run.exhausted, clean)" if r['exhausted']
else "")),
("5 Failure semantics", f"{r['retries']} retries, {r['dead']} "
f"dead-lettered; effects idempotent"),
("6 Resumability", f"checkpoints at {r['checkpoints']} units; "
f"resume re-runs only the unfinished slice"),
("8 Evals", f"verification mesh gated findings: precision "
f"{100 * r['real_raw'] // max(r['raw_findings'], 1)}% -> "
f"{100 * r['real_kept'] // max(r['kept'], 1)}%"),
("9 Observability", "every unit and vote emitted a span + ledger "
"entry"),
]
if __name__ == "__main__":
r = run()
print("=== Capstone A: repository-audit fleet ===")
print(f"repo: {r['units']} file groups, {PLANTED_BUGS} planted bugs\n")
print("--- fan-out (finders under budget + retries) ---")
print(f" units complete : {r['done']}/{r['units']}"
+ (" (run.exhausted before finishing)" if r['exhausted'] else ""))
print(f" retries / dead : {r['retries']} / {r['dead']}")
print(f" tokens spent : {r['spent']:,} / {BUDGET:,}")
print("\n--- verification mesh (blind panel, dedup first) ---")
p_raw = 100 * r['real_raw'] // max(r['raw_findings'], 1)
p_kept = 100 * r['real_kept'] // max(r['kept'], 1)
recall = 100 * r['real_kept'] // PLANTED_BUGS
print(f" raw findings : {r['raw_findings']} "
f"({r['real_raw']} real, precision {p_raw}%)")
print(f" after panel : {r['kept']} "
f"({r['real_kept']} real, precision {p_kept}%, "
f"recall {recall}%)")
print("\n--- the merged report ---")
print(f" {r['kept']} verified findings, each carrying "
f"{PANEL_VOTES} panel votes as provenance")
print("\n--- production-bar gate checklist ---")
for gate, how in gate_checklist(r):
print(f" gate {gate:<20} {how}")
print("\nthe whole platform, one run: shard -> fan-out under budget -> "
"retry/checkpoint -> adversarial verify -> merge with provenance, "
"every gate satisfied by machinery built earlier in the book.")
👉 Next: Capstone B, the same platform with the dials turned all the way toward trust: a small fleet with production access that must be unable to act without a human.