Replay: time-travel debugging
Picture the incident this chapter exists for. An agent did something strange in production last night: investigated the wrong service, say, or confidently blamed an innocent deploy. The transcript is right there in the ledger, and reading it tells you what happened but not why, and certainly not what would have happened differently if one tool had returned something else. With most software you would reach for a debugger and re-run the failing input. With an agent, "re-run" seems impossible: the model is a remote service, its outputs vary, the moment is gone.
Except we spent two chapters quietly buying back that moment. The
event store recorded every model response and
tool result with an address. The model interface
lets anything that can produce a ModelTurn sit in the model's chair.
Put them together: feed the recorded responses back through the loop,
and last night runs again on your laptop. That is replay, and this
chapter builds it, verifies it is exact, and then commits the act that
makes it a debugger rather than a screening room: changing history to
see what depends on it.
A stand-in that can react
First, an upgrade forced by honesty. Chapter 7's ScriptedModel is a
tape, and a tape cannot support the experiment we are planning. If we
patch a tool result and the "model" just plays its next scripted line
regardless, the experiment proves nothing; the interesting question is
whether the run diverges in response.
So this lab introduces RuleModel: a stand-in that reads the
conversation and reacts, deterministically. Its judgment is a handful of
rules that mirror how the investigation ought to branch: no metrics yet,
get metrics; metrics show a spike, look for a deploy just before it; a
deploy lands in that window, blame it and stop; no deploy fits, search
the logs and reason from those. It is a pure function of the messages:
same conversation in, same turn out, every time. Real reasoning it is
not, and it does not need to be. It needs exactly two properties, it
reacts and it repeats, because those are what the experiment consumes.
Where does that leave real models, which react but do not repeat? With a
recorded run, their past behavior is pinned: the ledger holds the
actual responses, so replay is exact up to the point where you change
something. After the patch point, a real system either re-samples live
(new judgment, honestly labeled) or stops at the divergence and shows
you what changed in the model's inputs. The lab's RuleModel gives us
the luxury of exactness on both sides of the patch, which is the right
setting for learning the machinery.
Three experiments
Record. Run RuleModel live against the unpatched world, recording
through the Chapter 9 instrumented loop. Nine events; the verdict blames
deploy v841, the pool-shrinking config change.
Replay, and verify it is exact. Build a RecordedModel whose
respond pops the next model.responded event off run A's ledger, run
the loop again with it, and compare ledgers:
class RecordedModel:
def __init__(self, events):
self._turns = [e["data"] for e in events
if e["type"] == "model.responded"]
def respond(self, messages):
d = self._turns.pop(0)
return ModelTurn(d["stop_reason"], d["text"],
[ToolCall(**c) for c in d["tool_calls"]])
Note what is and is not replayed: model turns come from the tape, but the tools execute for real. The replay therefore re-derives the tool results and the comparison checks them too, which makes it a genuine end-to-end test of the loop and tools, not a photocopy of the file. The demo asserts the two ledgers are byte-identical.
Inject a fault. Now the time travel. Same RuleModel, same task,
one change: list_deploys is patched to return a world in which the
v841 deploy record is missing, as if the deploy had bypassed the
pipeline that records deploys. Everything else is untouched. Because the
model stand-in reacts, the investigation must now take its other branch,
and because everything is deterministic, whatever happens is exactly
attributable to the patch.
python3 replay_lab.py
--- run A: live, unpatched world (9 events) ---
answer: Likely cause: deploy v841 at 01:55 (config: payments-db pool size 50 -> 10), minutes before the 02:00 spike. Recommend rollback.
--- replay of run A from its own ledger ---
replayed ledger identical to the original? True
--- run B: same model, list_deploys result patched (12 events) ---
answer: No deploy correlates with the 02:00 spike, but the logs show payments-db pool exhaustion at 02:03:14. Suspect a change made outside the deploy pipeline; escalate to the database team.
--- diff: run A vs run B ---
events 1..6 identical; first divergence at seq 7:
A: tool.returned list_deploys -> deploys: [v840 @18:20, v841 @01:55]
B: tool.returned list_deploys -> deploys: [v840 @18:20]
run A finished in 9 events; run B took 12 (it kept investigating).
Read the diff like a causal chain, because that is what it is. The runs are identical through seq 6: same metrics call, same spike found, same decision to check deploys. At seq 7 history changes: one tool result, one missing record. Everything after is consequence: run B finds no deploy near the spike, falls to its log-searching branch, spends three more events, and lands on a different diagnosis with different evidence (the 02:03:14 pool-exhaustion line) and a different recommendation (escalate to the database team rather than roll back v841). One patched input, two futures, and the exact fork point on screen.
Notice something else the experiment quietly demonstrated: run B's conclusion is reasonable. A real incident where the deploy record is missing would genuinely send a competent investigator down the database-team path. Fault injection does not just find bugs in agents; it maps how their conclusions depend on their evidence, which is the kind of understanding you want before trusting a fleet of them.
What replay buys at scale
This machinery, built on an eleven-event toy, is load-bearing for the rest of the book in at least four places:
- Production bugs become local experiments. Gate 4 of the production bar demanded exactly what run A's replay demonstrated: yesterday's weirdest run, re-executed deterministically, with the freedom to patch turn 17 and watch.
- Incidents become regression tests. A ledger that exhibits a failure is a test fixture forever: replay it against every prompt and tool change, assert the failure stays fixed. Part 9's eval suites are substantially this, industrialized.
- Resume is replay's little sibling. Recovering a crashed run means replaying its ledger to reconstruct state, then continuing live from the frontier. Part 6's kill-and-resume lab for whole fleets is this chapter's code plus bookkeeping.
- Verification uses the diff. When Part 6's verifier agents dispute a finder's claim, the dispute is settled by evidence at addresses in a ledger, and "what would the finder have concluded without this evidence" is precisely a fault-injection question.
The discipline that keeps all of it possible deserves its one-sentence
form: record at the edges, stay pure in the middle. Everything
nondeterministic (model responses, tool outputs, time, randomness)
enters the system at a boundary; record it there, and the loop between
boundaries stays a deterministic function you can re-run at will. Break
the rule once (a datetime.now() inside the loop, a tool that secretly
reads global state) and replay quietly degrades from proof to
approximation. The lab kept its events timestamp-free for exactly this
reason; production systems record timestamps at the boundary and
exclude them from replay comparison.
Don't be confused: replay vs retry. Retry runs a piece of work again, live, hoping the world treats it better this time; it is a reliability mechanism, and it may legitimately produce a different outcome. Replay re-executes recorded history and must produce the same outcome, or something is broken. Retry looks forward; replay looks backward. Fleets need both, and confusing them (retrying when you meant to replay, and especially replaying effects as if they were retries) is how duplicate tickets get filed at 3 a.m.
Part 1, closed
Count what a few hundred lines of standard-library Python now own: a loop that runs an investigation against any model-shaped thing (Chapter 7); tool definitions that measurably steer behavior, fail helpfully, and carry idempotency where it matters (Chapter 8); a ledger from which state is derived rather than stored, with checkpoints proven equal to full history (Chapter 9); and now exact replay with fault injection. Small as it is, this stack already passes gates 4 and 5 of the production bar in miniature, and every remaining part of the book is one of these components growing into its production form.
Full source
"""Lab 1.4: replay and time-travel debugging.
Three experiments on top of the Lab 1.3 event store:
1. record a live run into a ledger,
2. replay it: re-execute the loop with model turns served from the
ledger instead of a model, and verify the copy is byte-identical,
3. inject a fault: patch one tool's result and re-run, then diff the
two ledgers to see exactly where and how the run diverged.
The model here is a RuleModel: unlike Lab 1.1's fixed tape, it is a pure
function of the conversation, so it *reacts* to the patched result the
way a real model would, while staying fully deterministic.
Standard library only. Deterministic: same run, same ledgers.
"""
from __future__ import annotations
import json
from loop_agent import DEPLOYS, TOOLS, ModelTurn, Tool, ToolCall
from event_store import EventStore, run_agent_logged
TASK = "Checkout p99 latency spiked overnight. Find the likely cause."
# ---------------------------------------------------------------------------
# A deterministic model stand-in with just enough judgment to react
# ---------------------------------------------------------------------------
class RuleModel:
def respond(self, messages: list[dict]) -> ModelTurn:
seen = self._tool_results(messages)
if "query_metrics" not in seen:
return ModelTurn(
"tool_use", "Checking the latency metrics first.",
[ToolCall("r1", "query_metrics",
{"service": "checkout", "window": "24h"})])
series = seen["query_metrics"]["p99_ms"]
spike = max(series, key=lambda h: series[h])
if "list_deploys" not in seen:
return ModelTurn(
"tool_use",
f"p99 spikes at {spike}. Looking for a deploy just before it.",
[ToolCall("r2", "list_deploys", {"service": "checkout"})])
culprit = self._deploy_near(seen["list_deploys"]["deploys"], spike)
if culprit is not None:
return ModelTurn(
"end_turn",
text=(f"Likely cause: deploy {culprit['version']} at "
f"{culprit['at']} ({culprit['note']}), minutes before "
f"the {spike} spike. Recommend rollback."))
if "search_logs" not in seen:
return ModelTurn(
"tool_use",
"No deploy lands near the spike. Searching the logs instead.",
[ToolCall("r3", "search_logs", {"query": "ERROR"})])
pool = [h for h in seen["search_logs"]["hits"] if "pool exhausted" in h]
if pool:
return ModelTurn(
"end_turn",
text=(f"No deploy correlates with the {spike} spike, but the "
f"logs show payments-db pool exhaustion at "
f"{pool[0].split()[0]}. Suspect a change made outside "
f"the deploy pipeline; escalate to the database team."))
return ModelTurn("end_turn",
text="No deploy or log evidence found; needs a human.")
@staticmethod
def _tool_results(messages: list[dict]) -> dict:
id_to_name, out = {}, {}
for m in messages:
if m.get("role") == "assistant":
for call in m.get("tool_calls", []):
id_to_name[call["id"]] = call["name"]
if m.get("role") == "tool":
for r in m["results"]:
out[id_to_name[r["tool_call_id"]]] = json.loads(r["content"])
return out
@staticmethod
def _deploy_near(deploys: list[dict], spike: str) -> dict | None:
hour = int(spike.split(":")[0])
for d in deploys:
if int(d["at"].split(":")[0]) in (hour - 1, hour):
return d
return None
# ---------------------------------------------------------------------------
# Replay: serve model turns from a ledger instead of a model
# ---------------------------------------------------------------------------
class RecordedModel:
def __init__(self, events: list[dict]):
self._turns = [e["data"] for e in events
if e["type"] == "model.responded"]
def respond(self, messages: list[dict]) -> ModelTurn:
d = self._turns.pop(0)
return ModelTurn(d["stop_reason"], d["text"],
[ToolCall(**c) for c in d["tool_calls"]])
# ---------------------------------------------------------------------------
# Fault injection: the same toolset, with one tool's result patched
# ---------------------------------------------------------------------------
def patched_tools() -> list[Tool]:
def list_deploys_missing_v841(service: str) -> str:
rows = [d for d in DEPLOYS
if d["service"] == service and d["version"] != "v841"]
return json.dumps({"service": service, "deploys": rows})
out = []
for tool in TOOLS:
if tool.name == "list_deploys":
tool = Tool(tool.name, tool.description, tool.params,
list_deploys_missing_v841)
out.append(tool)
return out
def first_divergence(a: list[dict], b: list[dict]) -> int | None:
for ea, eb in zip(a, b):
if (ea["type"], ea["data"]) != (eb["type"], eb["data"]):
return ea["seq"]
return None
if __name__ == "__main__":
store_a = EventStore("run_a.jsonl")
answer_a = run_agent_logged(RuleModel(), TOOLS, TASK, store_a)
print(f"--- run A: live, unpatched world "
f"({len(store_a.events())} events) ---")
print(f" answer: {answer_a}")
store_r = EventStore("run_replay.jsonl")
run_agent_logged(RecordedModel(store_a.events()), TOOLS, TASK, store_r)
print("\n--- replay of run A from its own ledger ---")
print(f" replayed ledger identical to the original? "
f"{store_r.events() == store_a.events()}")
store_b = EventStore("run_b.jsonl")
answer_b = run_agent_logged(RuleModel(), patched_tools(), TASK, store_b)
print(f"\n--- run B: same model, list_deploys result patched "
f"({len(store_b.events())} events) ---")
print(f" answer: {answer_b}")
print("\n--- diff: run A vs run B ---")
seq = first_divergence(store_a.events(), store_b.events())
print(f" events 1..{seq - 1} identical; first divergence at seq {seq}:")
for label, store in (("A", store_a), ("B", store_b)):
event = store.events()[seq - 1]
payload = json.loads(event["data"]["output"])
rows = [f"{d['version']} @{d['at']}" for d in payload["deploys"]]
print(f" {label}: {event['type']} {event['data']['name']} "
f"-> deploys: [{', '.join(rows)}]")
print(f" run A finished in {len(store_a.events())} events; "
f"run B took {len(store_b.events())} (it kept investigating).")
👉 Next, Part 2 takes the model's chair seriously: Bedrock as the model plane, where the tokens come from, what they cost, and how caching and quotas bend a fleet's economics. Until those chapters land, the map shows what is coming.