The guided lab: one feature, before and after

TL;DR. This is the chapter where you run the experiment yourself, with every command spelled out. First, a step-by-step guide to caching in Claude Code: what the harness does for you automatically, the exact steps to verify it is working in your session right now, the do/don't list of what you control, and how to diagnose a bust. Then the experiment: the same real feature ("add a --json flag to the usage ledger") was implemented twice by Claude Code on this machine, once from a naive one-line prompt and once from a prompt that applies this book's practices. Both runs produced working, verified code. The naive run took 12 turns, 355,085 prompt tokens, 5,373 output tokens, $1.06, 99 seconds; the optimized run took 5 turns, 138,575 prompt tokens, 1,300 output tokens, $0.53, 35 seconds: half the cost and a third of the time, for the identical outcome, quantified turn by turn with a new per-session audit tool you can point at any session you have ever run.

Contents

• Part 1: caching in Claude Code, step by step
• The instrument: session_audit.py
• Part 2: the experiment, step by step
• The results, turn by turn
• Reading the deltas
• A bonus lesson the lab did not plan
• Run it on your own work
• Further reading
• Takeaways

The earlier chapters gave you the theory (Chapter 24), the receipts (Chapter 23), and the machine-wide evidence (Chapter 25). What they did not give you is a recipe: the exact sequence of commands that shows caching working in your session, and a worked before/after where a context-engineering practice is applied to a real coding task and the difference is a number. This chapter is that recipe, and every output in it is real: the two experiment sessions were run on this machine while writing this page, and their transcripts are quoted verbatim.

Part 1: caching in Claude Code, step by step

The first thing to understand is what you do not do. In the raw API you place cache_control breakpoints yourself (Chapter 6); in Claude Code you never see that parameter. The harness manages caching automatically: it keeps the prefix byte-stable (Chapter 28), moves the breakpoint down the conversation as it grows, and pays for the 1-hour TTL (visible in the ephemeral_1h_input_tokens field of every usage block, Chapter 23). Your job is not to enable caching. Your job is to verify it is working and avoid the behaviors that break it. Step by step:

Step 1: generate two turns of evidence. Open a session in any project and ask two ordinary questions (read a file, ask a follow-up). Caching is per-request, so you need at least two billed turns to see a hit.

Step 2: find the session's transcript. Your session writes to ~/.claude/projects/<munged-cwd>/<session-id>.jsonl (Chapter 25). The newest file in that directory is the session you are in:

ls -t ~/.claude/projects/-Users-you-src-yourproject/*.jsonl | head -1

Step 3: audit it. Run the per-session instrument from this chapter (below) against that file:

python3 code/session_audit.py ~/.claude/projects/<project>/<session>.jsonl

Healthy caching has an unmistakable signature, the accretion pattern from Chapter 23: turn 1 shows a large write and a read covering the static prefix; every later turn shows a small write (just what was appended) and a read equal to everything before it, with the prefix column saying ok. If instead every turn shows large writes and small reads, caching is broken for your session and you are paying roughly 10x more per prompt token than your neighbor.

Step 4: keep it healthy. The behaviors that decide the hit, with the reason in parentheses:

Step 5: diagnose a bust. When the audit shows a RESET row, look at what happened just before that turn in the session: a /compact, a /model, a settings change, an MCP toggle. The reset itself prints its price (the write column of that row is the re-bill). One real example appears later in this chapter: a session on this machine that reset at turn 6 and paid 35,206 write-tokens for it in one turn.

The instrument: session_audit.py

usage_ledger.py totals a machine; the experiment needs a per-session, per-turn view. This tool prints one row per billed request with the four usage fields, marks whether each turn extended the cache intact (ok) or broke it (RESET), and totals cost against the uncached counterfactual:

"""Audit ONE Claude Code session: the before/after instrument for the guided lab.

Where usage_ledger.py totals a whole machine, this drills into a single
session transcript and prints the per-turn accounting you need to compare a
baseline run against an optimized run of the same task:

  - one row per billed API turn: input / cache write / cache read / output,
    whether the turn EXTENDED the cached prefix (the accretion signature
    from the usage-anatomy chapter), and its cost at API prices
  - session totals: turns, full prompt volume, cache hit rate, output,
    cost, and the counterfactual cost with caching off

Point it at a session file, or at a project directory to list sessions:

    python3 session_audit.py ~/.claude/projects/<project>/<session>.jsonl
    python3 session_audit.py ~/.claude/projects/<project>/

Standard library only.
"""

import json
import sys
from pathlib import Path

PRICES = {  # $/Mtok (input, output); write 1.25x/2x by TTL, read 0.1x
    "claude-opus-4-8": (5.00, 25.00), "claude-opus-4-7": (5.00, 25.00),
    "claude-sonnet-4-6": (3.00, 15.00), "claude-sonnet-5": (3.00, 15.00),
    "claude-haiku-4-5": (1.00, 5.00), "claude-fable-5": (10.00, 50.00),
}


def price(model):
    for k, v in PRICES.items():
        if model.startswith(k):
            return v
    return (5.00, 25.00)


def cost(model, u):
    inp, out = price(model)
    inp, out = inp / 1e6, out / 1e6
    w = u.get("cache_creation") or {}
    w5 = w.get("ephemeral_5m_input_tokens", 0)
    w1 = w.get("ephemeral_1h_input_tokens", 0)
    if w5 == 0 and w1 == 0:
        w5 = u.get("cache_creation_input_tokens", 0)
    return (u.get("input_tokens", 0) * inp + w5 * inp * 1.25 + w1 * inp * 2.0
            + u.get("cache_read_input_tokens", 0) * inp * 0.10
            + u.get("output_tokens", 0) * out)


def uncached(model, u):
    inp, out = price(model)
    prompt = (u.get("input_tokens", 0) + u.get("cache_creation_input_tokens", 0)
              + u.get("cache_read_input_tokens", 0))
    return prompt * inp / 1e6 + u.get("output_tokens", 0) * out / 1e6


def turns_of(path):
    """One (usage, model) per billed request; assistant lines share a
    requestId per response, so dedupe on it. Sidechains are excluded."""
    seen, turns = set(), []
    for line in path.open(errors="replace"):
        try:
            d = json.loads(line)
        except json.JSONDecodeError:
            continue
        if d.get("type") != "assistant" or d.get("isSidechain"):
            continue
        u = (d.get("message") or {}).get("usage")
        rid = d.get("requestId")
        if u and rid and rid not in seen:
            seen.add(rid)
            turns.append((u, (d.get("message") or {}).get("model", "?")))
    return turns


def audit(path):
    turns = turns_of(path)
    if not turns:
        print(f"{path.name}: no billed turns found")
        return
    print(f"=== {path.name} ===")
    hdr = f"{'turn':>4}{'input':>8}{'write':>9}{'read':>10}{'out':>7}{'prefix':>8}{'cost':>9}"
    print(hdr)
    print("-" * len(hdr))
    tot = {"input_tokens": 0, "cache_creation_input_tokens": 0,
           "cache_read_input_tokens": 0, "output_tokens": 0}
    c = base = 0.0
    prev_total = None
    for i, (u, model) in enumerate(turns, 1):
        read = u.get("cache_read_input_tokens", 0)
        write = u.get("cache_creation_input_tokens", 0)
        if prev_total is None:
            mark = "start"
        elif read >= 0.95 * prev_total:
            mark = "ok"        # cache extended intact
        else:
            mark = "RESET"     # the prefix broke this turn
        prev_total = read + write
        for k in tot:
            tot[k] += u.get(k, 0)
        tc = cost(model, u)
        c += tc
        base += uncached(model, u)
        print(f"{i:>4}{u.get('input_tokens',0):>8,}{write:>9,}{read:>10,}"
              f"{u.get('output_tokens',0):>7,}{mark:>8}{tc:>9.4f}")
    prompt = (tot["input_tokens"] + tot["cache_creation_input_tokens"]
              + tot["cache_read_input_tokens"])
    print("-" * len(hdr))
    print(f"turns: {len(turns)}   full prompt volume: {prompt:,} tok   "
          f"output: {tot['output_tokens']:,} tok")
    print(f"cache hit rate: {tot['cache_read_input_tokens']/max(prompt,1)*100:.1f}%   "
          f"cost: ${c:.4f}   (uncached would be ${base:.4f}, {base/max(c,1e-9):.1f}x)")


def main():
    target = Path(sys.argv[1]) if len(sys.argv) > 1 else None
    if target is None or not target.exists():
        sys.exit("usage: python3 session_audit.py <session.jsonl | project dir>")
    if target.is_dir():
        files = sorted(target.glob("*.jsonl"),
                       key=lambda f: f.stat().st_mtime, reverse=True)
        print(f"sessions in {target} (newest first):")
        for f in files[:10]:
            n = len(turns_of(f))
            if n:
                print(f"  {f.name}   {n} turns")
        return
    audit(target)


if __name__ == "__main__":
    main()

This is the before/after instrument for everything that follows, and for any A/B you run yourself under the Chapter 27 protocol.

Part 2: the experiment, step by step

The task. Add a real feature to a real codebase: a --json flag for Chapter 25's usage_ledger.py, so the summary can be consumed by scripts. Concrete, verifiable (the output must parse), and small enough to run twice.

The setup. Two identical copies of this book's project (code/ and src/, 53 files) in two directories, so each run gets a fresh, isolated session and transcript. Both runs used the same model, the same permissions (--permission-mode acceptEdits, python3 allowed so the agent can verify its work), and Claude Code's print mode so the runs are scriptable; in an interactive session, /clear before each run gives you the same fresh start.

Run A, the naive prompt. One vague sentence, the way most people prompt on day one. No file named, no requirements, no constraints:

claude -p "Add a JSON output option to the usage ledger tool in this project, so
the same summary can be printed as machine-readable JSON. Make sure it works." \
  --output-format json --permission-mode acceptEdits --allowed-tools "Bash(python3:*)"

Run B, the engineered prompt. The same task, specified the way this book teaches: the exact file named (no exploration needed), the output contract spelled out (no design wandering), reads explicitly bounded (Chapter 5), a verification command supplied, and terse output requested (Chapter 4):

Add a --json flag to code/usage_ledger.py so the summary prints as machine-readable JSON.

Requirements:
- When invoked as `python3 code/usage_ledger.py --json [dir]`, print ONE JSON object
  with keys: sessions, turns, per_model (the four token fields per model), totals
  (prompt_tokens, cache_read, cache_write, input, output, hit_rate), cost_usd,
  uncached_cost_usd, top_sessions (list of {id, cost_usd}, max 5). No table output
  in this mode.
- Default (no flag) behavior must stay byte-identical.
- Keep the diff minimal; no refactors.

Constraints:
- Read ONLY code/usage_ledger.py. Do not explore, list, or read any other file.
- Verify with: python3 code/usage_ledger.py --json | python3 -m json.tool > /dev/null && echo OK
- Reply with one line: what changed and the verification result. No recap, no plan narration.

The measurement. After each run: the CLI's own JSON result (total_cost_usd, duration_ms), then session_audit.py on the session's transcript for the turn-by-turn view, then an independent check that the feature actually works (python3 code/usage_ledger.py --json | python3 -m json.tool). Both runs passed that check; this is a comparison of two successes.

The results, turn by turn

Run A, audited from its real transcript:

=== d7f9dda0-fa44-4bae-acd6-ea95a2a9cd85.jsonl ===
turn   input    write      read    out  prefix     cost
-------------------------------------------------------
   1   5,932    3,716    14,837    184   start   0.1577
   2       2    6,299    18,553    169      ok   0.1530
   3       2      811    24,852    127      ok   0.0474
   4       2    2,874    25,663    193      ok   0.0928
   5       2      375    28,537    526      ok   0.0624
   6       2      662    28,912    300      ok   0.0572
   7       2      436    29,574    659      ok   0.0713
   8       2      771    30,010    326      ok   0.0618
   9       2      438    30,781    999      ok   0.0895
  10     124    1,029    31,219    790      ok   0.0925
  11       2    1,288    32,248    471      ok   0.0816
  12       2    1,588    33,536    629      ok   0.0968
-------------------------------------------------------
turns: 12   full prompt volume: 355,085 tok   output: 5,373 tok
cache hit rate: 92.6%   cost: $1.0639   (uncached would be $3.8195, 3.6x)

Run B, same instrument:

=== 36c48aab-06eb-495b-99ef-4a5293125ba2.jsonl ===
turn   input    write      read    out  prefix     cost
-------------------------------------------------------
   1   5,899    3,989    14,837    141   start   0.1607
   2       2    8,785    18,826    300      ok   0.2095
   3     197      436    27,611    580      ok   0.0673
   4       2      887    28,047    186      ok   0.0551
   5       2      121    28,934     93      ok   0.0360
-------------------------------------------------------
turns: 5   full prompt volume: 138,575 tok   output: 1,300 tok
cache hit rate: 85.3%   cost: $0.5286   (uncached would be $1.4507, 2.7x)

Side by side, same task, same model, both features verified working:

Reading the deltas

Each row of the comparison is one of this book's levers, showing up in the accounting exactly where the theory said it would:

• Turns fell 12 to 5 because exploration was engineered away. Run A had to find the ledger tool (its early turns are searches and reads across the project) and then decide what the JSON should contain. Run B was handed the file path and the output contract, so its five turns are: read the file, edit, verify, fix nothing, report. Every eliminated turn removes a whole re-send of the growing window, which is why prompt volume fell 2.6x, faster than turns alone would suggest.
• Prompt volume is the compounding cost. 355k versus 138k full prompt tokens for the same feature is Chapter 17's re-sent-window arithmetic in a lab jar. Note both runs were cache-healthy (92.6 and 85.3 percent hit rates, every turn ok); the naive run did not waste money on cache misses, it wasted money on volume the cache still had to serve. Caching discounts the window; only a shorter session shrinks it.
• Output fell 4.1x because it was asked to. Run A narrated: plans, progress, a closing essay (5,373 output tokens at the 5x rate is $0.13 of pure narration). Run B's "one line, no recap" bought the Chapter 4 saving with one sentence of prompt.
• The hit-rate paradox is worth a beat: run A has the higher hit rate (92.6 vs 85.3) while costing double. Longer sessions always look better on this metric, because ever more of each prompt is history served from cache. Hit rate measures cache health, not efficiency; never optimize for it directly. Cost, turns, and volume are the objective.
• What did not change is the floor. Both runs paid the same turn-1 setup (about 19k tokens of prefix write+read) and the same per-turn tool schemas. Prompt engineering cannot touch the harness floor; it owns everything above it.

A bonus lesson the lab did not plan

The first attempt at run A was launched without allowing python3, and the agent spent the session fighting the permission gate: trying the command, being denied, trying via a subagent, being denied again, and finally reporting honestly that it could not verify its work. That blocked session cost 22 turns, 734,123 prompt tokens, and $1.81 audited ($2.10 including its subagent), against $1.06 for the identical prompt with the permission granted:

turns: 22   full prompt volume: 734,123 tok   output: 10,819 tok
cache hit rate: 95.7%   cost: $1.8099   (uncached would be $7.8822, 4.4x)

That is the most expensive single finding in this chapter: a misconfigured environment cost 70 percent more than the worst prompt. An agent that cannot run its verification loop burns turns on workarounds, and turns are the compounding unit. Before you tune a single word of a prompt, make sure the agent can execute the task's feedback loop (tests, build, linter) without hitting a wall; Chapter 19's permission configuration is upstream of every optimization in this book.

Run it on your own work

The replication recipe, compressed:

1. Pick one recurring task and write two prompts for it: your current one, and one that names the files, states the output contract, bounds the reads, supplies the verification command, and requests terse output.
2. Run both from a clean start (/clear, or claude -p in a scratch copy), with identical permissions that allow the verification loop.
3. Audit both transcripts with session_audit.py; confirm both runs actually succeeded before comparing anything.
4. Compare cost, turns, prompt volume, output, in that order, and check the prefix column stayed ok (a RESET in one arm contaminates the comparison).
5. Fold what won into permanence: the winning prompt shape becomes a slash command or a CLAUDE.md convention, and the environment fix (allowed tools) becomes project settings, so the saving repeats without anyone remembering this experiment.

Don't be confused. claude -p starts a fresh session per invocation, which is what makes it a clean instrument for A/B runs, and also what makes it the wrong way to do a long multi-step task day to day: each -p call rebuilds conversation context from nothing. The experiment uses print mode to isolate a variable; your working sessions should stay continuous for exactly the caching reasons in Part 1.

Further reading

• Chapter 27: the general A/B protocol this experiment instantiates, including the per-command/realized/net savings distinction.
• Chapter 23 and Chapter 24: the usage fields and cache rules the audit tool reads.
• Claude Code, "Manage costs effectively" (code.claude.com/docs/en/costs) and the CLI reference for claude -p, --output-format json, and --allowed-tools.

Takeaways

• Caching in Claude Code is automatic; your five steps are: generate two turns, find the transcript, audit it, keep the session continuous and the config stable mid-task, and treat every RESET row as a purchase you should recognize.
• The measured experiment: the same verified feature cost $1.06, 12 turns, and 355k prompt tokens from a naive prompt, versus $0.53, 5 turns, and 139k from an engineered one. Half the cost, 2.8x faster, identical outcome.
• The deltas map to the levers: named files kill exploration turns, a stated contract kills design wandering, bounded reads shrink the window, and one terseness sentence cut output 4.1x at the 5x rate.
• Hit rate is health, not efficiency: the expensive run had the better hit rate. Optimize cost, turns, and volume; just keep the prefix column ok.
• The accidental headline: a permission gate the agent could not pass cost more than bad prompting (22 turns, $1.81 for the same task). Fix the environment before the prompt.
• Fold wins into permanence: slash commands, CLAUDE.md, and project settings are where a measured improvement stops depending on memory.

👉 You have now run the whole loop on a real task: engineer the context, run the work, audit the receipts. The closing chapter of this part is the reference that holds it all: every measurement surface Claude Code offers, layer by layer, and the ratios that turn readings into decisions. Continue to The measurement compendium.