Failure semantics: retries, poison, and resume

A fleet that runs long enough meets every failure its design ignored. This chapter designs them in advance: the taxonomy of ways agent work fails, the mechanics that make each one boring, and the lab that kills the simulator mid-run and itemizes, token by token, what the crash cost. Nothing here is speculative; it is the production bar's gates 5 and 6 becoming code.

The taxonomy

Five ways a unit of agent work fails, each with its own correct response, and mixing them up is the classic fleet bug:

  • Transient failure (a model timeout, a flaky tool): retry, with jitter, through the governor so a bad minute does not become a retry storm. The tokens spent are gone; the ledger records them; the attempt counter increments.
  • Poison work (a unit that fails every attempt, on content, not luck: the malformed PDF, the repository shard that crashes the parser): after N attempts, stop. The unit parks in the dead-letter queue with its attempt ledgers attached, a human looks, and the fleet moves on. The alternative, infinite retry, converts one bad input into unbounded spend, and it is astonishing how many fleets ship it by default.
  • Worker death (the process, container, or spot instance dies): the unit's lease expires and the dispatcher re-issues it. The lease is the mechanism that distinguishes "worker is slow" from "worker is gone" without asking the worker, implemented as a conditional write with a TTL in the state plane. Losing a worker is designed to be indistinguishable from never having assigned the unit, which is what makes fleets of spot instances economical.
  • Partial completion (the agent did real work, then failed): with per-unit event ledgers, the work up to the last checkpoint survives, and re-dispatch resumes rather than restarts. This is Part 1's replay machinery earning its keep at fleet scale.
  • Budget exhaustion: not a failure of a unit but of the run's premise, handled by the scheduler as a clean landing: stop admitting, drain, enumerate the remainder.

Beneath all five sits the delivery reality the concepts chapter installed: at-least-once delivery means any unit may run twice, so every effect carries an idempotency key (tool design) and the ledger's writes are conditional. Attempts are at-least-once; effects are exactly-once; the gap between those sentences is where duplicate tickets come from.

Lab 6.2: kill it and count

The lab murders the baseline fleet at t=700s, resumes from what the scheduler checkpointed, and audits the damage:

python3 swarm_resume.py

--- run A: killed at t=700s ---
at the moment of death: 136 units complete, 100 in flight, 2,829,190 tokens spent
the checkpoint (completed set + ledger) survives; the 100 in-flight units do not

--- run B: resumed from the checkpoint ---
resumed the remaining 264 units; after 1,087s more: 389/400 done, dead-lettered 0
and then run.exhausted: the ledger (correctly) still counts the crash's wasted tokens, so the budget ran out 11 units short

--- the cost of the crash, itemized ---
re-executed work : only the 100 in-flight units (25% of the run), never the 136 completed ones
budget damage    : ~1,217,219 tokens of in-flight work were spent and lost, which is exactly the shortfall above
naive restart    : would have re-run all 400 units and re-spent 2,829,190 tokens
with per-unit event checkpoints (Part 1's ledger inside each unit), in-flight work resumes mid-turn and both numbers round toward zero

The first reading is the expected one: resume re-executed only the in-flight units, never the 136 completed, while a naive restart would have re-run everything and re-spent 2.8M tokens. Gate 6, demonstrated.

The second reading is the one designs miss, and the lab was arranged so you could not: the crash spent budget. The 100 in-flight units had consumed 1.2M tokens of model work that died with them, the ledger (correctly, because the provider billed it) still counts that spend, and so the resumed run exhausted its budget 11 units short of done. Crashes cost twice: once in wall clock, once in budget, and a run's budget must carry a reserve proportional to (in-flight cost x expected crashes) or resumes will land short exactly like this one. That reserve is a line in Hive's admission arithmetic now, and it came from this lab.

The closing line points at the deeper fix: coarse, unit-level checkpointing loses whole units; the per-unit event ledger from Part 1 checkpoints inside units, so a resumed unit replays its recorded turns and continues from its last event, and both loss numbers collapse toward zero. The simulator models the coarse version because the fine one was already built, four chapters into the book, which is why it was built first.

The managed-service note: distributed map's redrive is this lab's resume as a product feature (re-run failed children only), and it is good; what it does not know about is your ledger, so the budget- damage lesson survives every bundle choice in the previous chapter.

Don't be confused: retry vs resume. Retry re-attempts a unit that failed, from its start, spending fresh tokens on old work. Resume continues a run (or, with event ledgers, a unit) from its checkpoint, spending tokens only on what never finished. Retry is a failure-semantics decision (how many, with what backoff, before poison); resume is a state-design consequence (you can only resume what was checkpointed). Fleets need a policy for the first and an architecture for the second, and no amount of the first substitutes for the second.

Full source

"""Lab 6.2: kill the fleet mid-run, resume from the checkpoint.

Uses the Lab 6.1 simulator. Run A is killed at t=700s; whatever the
scheduler had checkpointed (completed units, dead letters, budget
spent) becomes the resume state, and run B picks up from there. The
measurements that matter: how much work the crash actually cost, and
how that compares to the naive alternative (restart everything).

Standard library only. Deterministic: same run, same numbers.
"""

from __future__ import annotations

from swarm_sim import UNITS, run_fleet, unit_tokens

BUDGET = 6_000_000

if __name__ == "__main__":
    a = run_fleet(workers=100, budget=BUDGET, kill_at=700)
    print(f"--- run A: killed at t={a['killed_at']}s ---")
    print(f"at the moment of death: {len(a['done'])} units complete, "
          f"{a['in_flight']} in flight, {a['spent']:,} tokens spent")
    print(f"the checkpoint (completed set + ledger) survives; "
          f"the {a['in_flight']} in-flight units do not")

    resume_state = {"done": a["done"], "dead": a["dead"],
                    "spent": a["spent"]}
    b = run_fleet(workers=100, budget=BUDGET, resume=resume_state)
    print(f"\n--- run B: resumed from the checkpoint ---")
    print(f"resumed the remaining {UNITS - len(a['done'])} units; "
          f"after {b['elapsed']:,}s more: {len(b['done'])}/{UNITS} done, "
          f"dead-lettered {len(b['dead'])}")
    if b["exhausted"]:
        print(f"and then run.exhausted: the ledger (correctly) still "
              f"counts the crash's wasted tokens, so the budget ran out "
              f"{UNITS - len(b['done']) - len(b['dead'])} units short")

    lost_tokens = a["spent"] - sum(unit_tokens(u) for u in a["done"])
    print(f"\n--- the cost of the crash, itemized ---")
    print(f"re-executed work : only the {a['in_flight']} in-flight units "
          f"({100 * a['in_flight'] // UNITS}% of the run), never the "
          f"{len(a['done'])} completed ones")
    print(f"budget damage    : ~{lost_tokens:,} tokens of in-flight work "
          f"were spent and lost, which is exactly the shortfall above")
    print(f"naive restart    : would have re-run all {UNITS} units and "
          f"re-spent {a['spent']:,} tokens")
    print(f"with per-unit event checkpoints (Part 1's ledger inside each "
          f"unit), in-flight work resumes mid-turn and both numbers "
          f"round toward zero")

👉 Next: the verification mesh, where the fleet stops merely surviving and starts being right: adversarial panels, majority gates, and precision purchased at a measured price per false alarm.