Tracing, replay and checkpointing

What it is

Three related capabilities that together make an agent debuggable, and they are frequently conflated:

Tracing records what happened: every model call with its full input and output, every tool call with arguments and result, timings, token counts and costs, linked into a tree by a shared trace ID.

Replay re-executes a recorded run, either deterministically (returning the recorded responses, to test changed orchestration code) or live (re-issuing the calls, to test a changed prompt against a real input).

Checkpointing persists the agent's state at step boundaries so a run can be resumed, rewound, or forked after a crash or a human interruption.

What they are confused with: logging. A log line saying "called lookup_order, got 200 OK" tells you nothing about why the model chose that tool, what it was shown, or what it did with the result. The unit of an agent trace is the full model input, and a logging system that truncates or samples is not a tracing system for this purpose.

The distinction that matters practically: tracing is for diagnosis, replay is for iteration, checkpointing is for recovery and human-in-the-loop. You can have any one without the others and they compose usefully.

The problem it solves

An agent failure has no stack trace. A conventional service that fails gives you an exception, a line number and a state. An agent that produces a wrong answer after nine tool calls gives you a wrong answer. The information you need (what was in the context at step 6, why the model picked that tool, what the tool returned) exists only if you recorded it, and it is gone by the time anyone asks.

Three specific failures follow:

Unreproducible bugs. A user reports a bad answer. Without a trace you cannot see the input, and the model is not deterministic even at temperature 0 (see sampling strategies), so re-running the same request may not reproduce it. Without the recorded context, the investigation is guesswork.

Regression testing that requires live calls. Changing orchestration code means re-running the whole agent against real model calls to know if you broke anything, which is slow and costs money, so it is done rarely and changes ship untested.

Long runs that cannot survive anything. An agent doing 40 steps over 20 minutes fails at step 37 and starts again. Worse, if the first 36 steps had side effects, restarting repeats them.

Mechanics

The trace tree

trace: req_8841 (task: "why did deploy 4471 fail?")     total 41.2s  $0.089
├── llm: plan                          1.8s   2,140 tok in / 310 out
├── tool: get_deploy_status            0.3s
├── llm: decide next                   1.2s   2,890 tok in / 84 out
├── span: subagent "read logs"         18.4s  $0.041
│   ├── llm: plan                      1.1s
│   ├── tool: search_logs              6.2s   (returned 14,200 tokens)
│   ├── llm: analyse                   9.8s   16,400 tok in / 420 out
│   └── result: 180 tokens returned to parent
├── llm: decide next                   1.4s   3,410 tok in / 96 out
├── tool: get_deploy_diff              0.9s
└── llm: synthesise                    2.1s   5,120 tok in / 640 out

That tree answers questions a log cannot: where the time went (the sub-agent's analyse call), where the money went, what the sub-agent returned to the parent, and how much context each step carried.

What every span must record:

@dataclass
class Span:
    trace_id: str
    span_id: str
    parent_span_id: str | None
    kind: Literal["llm", "tool", "retrieval", "agent", "chain"]
    name: str
    started_at: float
    ended_at: float

    # For LLM spans: the FULL input, not a summary.
    model: str | None = None
    messages: list[dict] | None = None      # complete, including system
    params: dict | None = None              # temperature, top_p, tools, seed
    output: str | None = None
    tokens_in: int | None = None
    tokens_out: int | None = None
    cost_usd: float | None = None

    # For tool spans:
    tool_name: str | None = None
    tool_args: dict | None = None
    tool_result: Any = None
    tool_error: str | None = None

    # Correlation with the outside world:
    prompt_version: str | None = None
    code_version: str | None = None
    user_id: str | None = None

messages being complete is the non-negotiable part, and it is what teams cut first because it is the bulk of the storage. A trace with a truncated prompt cannot answer "what did the model actually see," which is the question you have (see context engineering, where the system prompt reached the model 78 percent of the time and nobody could see it).

prompt_version and code_version on every span are what let you correlate a quality change with a deploy. Without them, "quality dropped on Tuesday" is unattributable.

Storage, since full traces are large

mean trace size (9-step agent):     ~180 KB of JSON
at 100k requests/day:               18 GB/day, 6.5 TB/year

The tiering that works:

Hot (7 days):     everything, full fidelity, queryable.
Warm (90 days):   full traces for FAILURES and a sample of successes
                  (1-5%), plus metadata for everything.
Cold (2 years):   metadata only: timings, token counts, costs, outcome.
Always:           any trace referenced by a bug report or an eval case,
                  pinned indefinitely.

Sampling successes and keeping all failures is the right asymmetry, because the failures are what you investigate and the successes are what you need for volume statistics. The pinning rule matters more than it looks: a trace referenced by an eval case must outlive the retention window or the eval case becomes unexplainable.

Replay: two modes, two purposes

async def replay(trace_id: str, mode: Literal["deterministic", "live"]):
    trace = store.load(trace_id)

    if mode == "deterministic":
        # Return recorded responses. Tests ORCHESTRATION changes:
        # routing, budgets, error handling, assembly order.
        model_fn = RecordedModel(trace)      # matches on call index
        tool_fn  = RecordedTools(trace)
    else:
        # Re-issue calls for real. Tests PROMPT and MODEL changes
        # against a real input. Tools are still recorded, so no side effects.
        model_fn = LiveModel()
        tool_fn  = RecordedTools(trace)

    return await run_agent(trace.initial_input, model=model_fn, tools=tool_fn)

Tools are recorded in both modes, which is the safety property. Replaying an agent that issues refunds must not issue refunds, and the only reliable way to guarantee that is to never call the real tool during replay.

Deterministic replay is where the value is for engineering. It runs in milliseconds, costs nothing, and turns "did I break the orchestration" into a unit test:

@pytest.mark.parametrize("trace_id", GOLDEN_TRACES)
def test_orchestration_unchanged(trace_id):
    original = store.load(trace_id)
    replayed = replay(trace_id, mode="deterministic")
    assert replayed.tool_call_sequence == original.tool_call_sequence
    assert replayed.final_output == original.final_output

A library of 50 recorded traces becomes a regression suite that costs nothing to run. That is the single highest-value thing tracing enables, and it is under-built because the recording has to be complete for the replay to work.

The subtlety: deterministic replay matches recorded responses by call index, so a code change that adds or removes a model call desynchronises the replay. Match on a hash of the request instead, and treat a miss as a signal rather than an error:

class RecordedModel:
    def __call__(self, messages, **params):
        key = hash_request(messages, params)
        if key in self.recorded:
            return self.recorded[key]
        # The code now makes a DIFFERENT call than it did. That is the finding.
        raise ReplayDivergence(f"unrecorded call at step {self.index}: {key}")

Checkpointing

@dataclass
class Checkpoint:
    run_id: str
    step: int
    state: AgentState             # messages, scratchpad, plan, whatever the agent holds
    pending: ToolCall | None      # a call issued but not yet resolved
    created_at: float

def run_with_checkpoints(task, store):
    state = store.latest(run_id) or AgentState.initial(task)
    while not state.done:
        state = step(state)
        store.save(Checkpoint(run_id, state.step, state))    # AFTER each step
        if state.needs_approval:
            return Suspended(run_id, state.pending)          # wait for a human
    return state.result

Three things checkpointing enables, and only the first is obvious:

Crash recovery. Resume from step 36 instead of step 0.

Human-in-the-loop. The agent suspends before a consequential action, a human approves or edits, and the run resumes. This is the main production use and it needs checkpointing because the wait may be hours and the process will not stay alive.

Time travel. Rewind to step 4, change something, and fork a new branch. That is a debugging tool and it is also how you build "undo" into an agent product.

The hard part is side effects. A checkpoint taken after a model call but before the tool executes leaves an ambiguity on resume: did the tool run? The resolutions, in order of preference:

1. Idempotency keys on every tool that mutates. Resume re-issues; the tool
   deduplicates. This is the only fully correct answer.
2. Checkpoint the INTENT before executing and the RESULT after, so resume
   can see a pending call with no result and decide.
3. Mark tools as replay-unsafe and require human confirmation on resume.

Without idempotency keys, resumption is a correctness problem and not merely an engineering one. See idempotency.

A worked example: three days to diagnose one bad answer

A research assistant agent: searches internal documents, reads them, synthesises a briefing. About 4,000 runs a day, 8 to 20 steps each.

The incident. A user reported that a briefing had confidently cited a policy that did not exist. High-visibility, because the briefing had gone to an executive.

What existed: application logs.

2026-08-03T14:22:01Z INFO  agent run=r_8841 step=1 action=plan
2026-08-03T14:22:04Z INFO  agent run=r_8841 step=2 action=tool tool=search_docs
2026-08-03T14:22:09Z INFO  agent run=r_8841 step=3 action=tool tool=read_doc
...
2026-08-03T14:23:44Z INFO  agent run=r_8841 done tokens=48210 cost=0.31

Nothing about what the model was shown or what the tools returned. Three days of investigation followed: re-running the query (which produced a correct answer, so the bug did not reproduce), reading the search index, and eventually finding by hand that a document in the corpus contained a quoted example of a fictional policy, and the agent had read the quote as fact.

The tracing they built:

@trace_span(kind="llm")
async def call_model(messages, **params):
    span = current_span()
    span.messages = messages                 # COMPLETE
    span.params = params
    span.prompt_version = PROMPT_VERSION
    span.code_version = GIT_SHA
    response = await client.create(messages=messages, **params)
    span.output = response.content
    span.tokens_in = response.usage.input_tokens
    span.tokens_out = response.usage.output_tokens
    span.cost_usd = price(response.usage, params["model"])
    return response

storage:            18 GB/day hot, 7-day retention
mean trace size:    194 KB
cost:               ~$340/month (object storage + index)

The same class of incident two months later took 11 minutes, because the trace showed the exact document text the model had been given, with the fictional policy in a block quote, and the model's reasoning span saying "the policy document states...".

The fix that came out of it was a retrieval change (excluding block-quoted content from indexed passages), which nobody would have found without seeing the retrieved text.

Change 2: the replay suite. Once traces existed, they became test fixtures.

GOLDEN_TRACES = [
    "r_8841",       # the fictional-policy incident
    "r_9102",       # a run where the agent looped on a failing tool
    "r_7734",       # a correct multi-hop run, as a positive case
    # ... 47 more, curated from real runs
]

before:  orchestration changes tested by running the agent live
         ~$14 and 20 minutes per full test pass
         run: roughly weekly

after:   deterministic replay of 50 traces
         $0 and 8 seconds
         run: on every commit

orchestration bugs reaching production:  ~3/month -> 0.2/month

Free, instant regression testing was worth more than the diagnostic capability, which was the surprise. The traces were built to explain one incident and their durable value was as a test suite.

Change 3: checkpointing, for a different reason than expected. Long research runs occasionally failed at step 15 of 18 and restarted from zero, wasting the work. But the requirement that actually justified it was human approval: the agent had been given the ability to file tickets, and that needed a human gate.

async def run(task: str, run_id: str):
    state = checkpoints.latest(run_id) or AgentState.initial(task)
    while not state.done:
        state = await step(state)
        checkpoints.save(run_id, state)
        if state.pending and state.pending.tool in REQUIRES_APPROVAL:
            await notify_for_approval(run_id, state.pending)
            return Suspended(run_id)          # process can now exit
    return state.result

# Hours later, from an approval webhook:
async def on_approved(run_id: str, edited_args: dict | None):
    state = checkpoints.latest(run_id)
    state.pending.args = edited_args or state.pending.args
    await run(state.task, run_id)             # resumes from the checkpoint

The bug they hit immediately: resumption re-executed the last tool call, because the checkpoint was taken before execution and the resume path did not know whether it had run. Two runs filed duplicate tickets before anyone noticed.

# The fix: an idempotency key derived from the run and step.
async def execute_tool(state, call):
    key = f"{state.run_id}:{state.step}:{call.tool}"
    return await tools[call.tool].run(**call.args, idempotency_key=key)

Final:

                                  before        after
mean time to diagnose a bad
  answer                          3 days        11 minutes
orchestration regression tests    weekly, $14   every commit, $0
orchestration bugs to production  ~3/month      0.2/month
long-run failures losing all work common        none (resume)
human approval gate               impossible    supported
duplicate side effects on resume  n/a           0 (idempotency keys)
tracing cost                      $0            ~$340/month

The tracing cost is the number to hold against all of that, and it is the objection teams raise: 18 GB a day feels like a lot. Against three days of engineer time per incident and weekly rather than per-commit regression testing, it is not close.

The transferable finding: traces built for debugging turned out to be most valuable as test fixtures. The team justified the work on diagnosis and got the larger return from replay, which they had not planned. If you are arguing for tracing, argue for the regression suite, because it is a recurring saving rather than an insurance policy.

Production evidence

LangSmith, Langfuse, W&B Weave, Braintrust and Arize Phoenix all implement the trace-tree model with full prompt capture, and all treat the recorded run as convertible into a test case. That every product in the category built the same two features (trace and promote-to-eval) is convergent evidence that the replay-as-fixture pattern is the durable value.

OpenTelemetry's GenAI semantic conventions define span attributes for LLM calls (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens) so agent traces fit into standard observability infrastructure rather than requiring a separate stack. Adoption of these conventions is the current direction and it matters because it means agent traces can share a backend with everything else.

LangGraph's checkpointer is a first-class abstraction with persistence backends, and its documented motivations are exactly the three above: crash recovery, human-in-the-loop interrupts, and time-travel debugging. Temporal is the general-purpose durable execution engine several teams use for the same purpose, where the agent loop is a workflow and each step is an activity with automatic retry and replay.

Anthropic's Claude Code and similar tools persist session state so a conversation survives a restart, which is the consumer-facing version of checkpointing.

The durable-execution framing (Temporal, Restate, DBOS) is worth knowing because it solves the side-effect problem properly: activities are recorded, replay is deterministic by construction, and idempotency is part of the model rather than something you bolt on.

The debate

Should you capture full prompts? Yes, and it is the thing most likely to be cut for cost or privacy. Without the complete input you cannot answer the central question, which is what the model actually saw, and the most common production bugs are things silently absent from the context. The privacy objection is legitimate and the answer is redaction at capture, not truncation: strip PII by pattern before storing, keep the structure. Truncation destroys the artifact; redaction preserves it.

How long should you keep traces? Tiered: everything for a week, failures plus a small sample for 90 days, metadata for two years, and anything referenced by an eval case pinned forever. The pinning rule is the one people forget, and an eval case whose originating trace has expired becomes an unexplainable assertion.

Deterministic or live replay? Both, for different questions. Deterministic replay tests orchestration changes (routing, budgets, error handling, assembly) at zero cost in milliseconds, and it is the one that becomes a CI suite. Live replay tests prompt and model changes against real inputs and costs real money, so it runs on prompt changes only. Tools must be recorded in both, or replay has side effects.

Is checkpointing worth the complexity? For short agent runs with no side effects, no: just retry. It earns its place when runs are long enough that losing the work matters, when a human gate is required (which is most consequential agent products), or when side effects make blind retry unsafe. The complexity is not the checkpoint, it is the side-effect semantics, and if your tools are not idempotent you have to solve that first regardless.

Should you build this or buy it? Buy, almost certainly. The trace-capture problem is well-solved by several products and by OpenTelemetry conventions, and the differentiated work is in what you do with the traces (curating a golden set, building the replay suite, wiring evals). Teams that build their own tracing typically produce something that captures less than a product would and takes a quarter.

What is the honest cost objection? 18 GB a day at 100k requests is real storage and a real bill. The response is that the alternative is measured in engineer-days per incident, and the worked example moved diagnosis from three days to eleven minutes. Where the volume genuinely does not justify it, sample: full traces for all failures and 1 percent of successes still gives you the diagnostic capability and most of the fixture library.

Follow-up Q&A

"Why is logging insufficient for agents?"

Because the question you need to answer is what the model was shown, and a log line saying "called tool X" does not contain it. Agent failures have no stack trace: a wrong answer after nine steps gives you the wrong answer and nothing about why. The unit of an agent trace is the complete model input, including the system prompt, the assembled context and the tool results, and any system that truncates or samples that is not tracing for this purpose.

"What does a trace need to record?"

Per span: kind, name, parent, timings. For model calls, the complete message list, the parameters, the output, token counts and cost. For tool calls, the arguments, the result and any error. And on every span, the prompt version and the code version, which is what lets you attribute a quality change to a deploy. The complete message list is the part teams cut and the part that answers the question.

"What is deterministic replay and why does it matter?"

Re-running a recorded trace with the recorded model responses returned instead of live calls, which tests orchestration changes (routing, budget policy, error handling, assembly order) in milliseconds at zero cost. A library of 50 curated traces becomes a regression suite you can run on every commit. In one case that replaced a weekly $14 live test pass and took orchestration bugs reaching production from about 3 a month to 0.2.

"How do you replay without repeating side effects?"

Record the tools and return recorded results in both replay modes. Replaying an agent that issues refunds must not issue refunds, and the only reliable guarantee is never calling the real tool during replay. For resumption after a checkpoint, which does execute for real, the answer is idempotency keys derived from the run and step, so a re-issued call deduplicates at the tool.

"What is checkpointing for?"

Three things, and only the first is obvious. Crash recovery, so a 40-step run failing at step 37 resumes rather than restarting. Human-in-the-loop, which is the main production use: the agent suspends before a consequential action, the process exits, a human approves hours later, and the run resumes from the checkpoint. And time travel, rewinding to an earlier step to fork a different branch, which is a debugging tool and also how you build undo into an agent product.

"What is the hard part of checkpointing?"

Side effects. A checkpoint taken after the model decides to call a tool but before the tool executes leaves an ambiguity on resume: did it run? In one system that filed duplicate tickets. The correct fix is idempotency keys on every mutating tool so resumption deduplicates; the partial fixes are checkpointing intent and result separately so resume can see a pending call with no result, or marking tools replay-unsafe and requiring confirmation.

Common misconceptions

"Logs are traces." A log records that something happened; a trace records what the model was shown and what came back. The complete model input is the artifact, and it is what answers the questions you actually have.

"Traces are for debugging." That is the justification and the larger return is usually the regression suite: recorded traces replayed deterministically test orchestration changes for free on every commit. Argue for tracing on that basis, because it is recurring rather than insurance.

"Truncate large prompts to save storage." That removes exactly the content that answers "what did the model see." Redact by pattern for privacy; do not truncate for size. Sample whole traces instead if volume is the problem.

"Replay tests everything." Deterministic replay tests orchestration, not prompts or models, because it returns the recorded responses. Prompt changes need live replay against recorded inputs, and model changes need an eval set.

"Checkpointing is just for crash recovery." The main production use is human-in-the-loop approval, where the process must be able to exit and resume hours later. Crash recovery is the easy case.

Interview delivery note

Say this verbatim: "Agent failures have no stack trace, so the trace has to capture the complete model input, not a summary. And the durable value is not diagnosis, it is that 50 recorded traces replayed deterministically become a regression suite that runs on every commit for nothing, which in one case took orchestration bugs reaching production from three a month to one every five months." The necessity plus the return, and the return is the part that funds the work.

The senior-versus-staff separator is the side-effect problem in resumption. A senior engineer builds tracing and checkpointing correctly. A staff engineer notices that a checkpoint between the model's decision and the tool's execution creates an ambiguity on resume, that this filed duplicate tickets in production, and that the only fully correct fix is idempotency keys derived from run and step rather than a heuristic about whether the call completed. Connecting agent checkpointing to distributed-systems idempotency is the move.

The second signal is redaction rather than truncation for privacy. Truncating a prompt to save space or satisfy a privacy review destroys the one thing the trace exists to answer; pattern-based redaction preserves the structure and the context assembly, which is where the bugs are.

The Staff & Team Lead Interview Reference