The optimization lab: one session, every lever, measured

TL;DR. This is a worked lab: take one autopilot coding session and optimize it phase by phase, measuring the cost at each step, until every lever is stacked. A from-scratch cost model of a representative 30-turn session is run for real, and it lands the fully optimized session at 87% cheaper, 7.4x, versus the naive baseline. The journey also ranks the levers honestly: bounding the re-sent history (compaction) and cutting per-turn reads do most of the work, while a small CLAUDE.md and tool-output compression barely move the number on their own. The point is not the exact figure; it is seeing which moves matter and in what order.

Contents

Every chapter so far gave you one lever and a measurement. This one puts them in a line and runs a single workload through all of them, so you can see the compounding, and the order of leverage, in one table. Think of it as the capstone experiment: the professional workflow was the plan; this is the plan, measured.

The scenario

The workload is an autopilot session: an agent building a small feature on its own over about 30 turns, the kind of run you start and let work. The naive baseline is the version most people run on day one, with every lever off:

  • A 15,000-token stable prefix (system prompt, tool schemas, and a large CLAUDE.md).
  • Whole-file reads, about 4,000 tokens per turn.
  • Verbose tool output (tests, git, logs), about 3,000 tokens per turn.
  • Verbose narration, about 800 output tokens per turn.
  • No caching, no compaction: the whole conversation is re-sent and grows every turn.

Those numbers are deliberately ordinary, and the prices are the real ones from Chapter 2 (opus-4-8 at $5 input and $25 output per million tokens, haiku at $1 and $5) with the caching multipliers from Chapter 6 (write 1.25x, read 0.1x). The compression ratios are the ones measured earlier in the book.

Don't be confused. This is a cost model, not a live agent run. It computes the dollar cost of a described session from real prices and measured ratios, so the numbers are reproducible and the shape is faithful, but they are a simulation, not a bill from a specific run. The verified on-box output below is the model's output; your real session will differ in the constants and agree in the structure. Reproduce the real version with /usage before and after, as the last section explains.

The harness, and the benchmark it produces

"""The optimization lab: one autopilot session, every lever, measured.

This is a COST MODEL, not a live agent run. It simulates a representative 30-turn
autopilot coding session (the kind that builds a small feature on its own) and
computes the dollar cost under each cumulative optimization, using real published
prices and the compression ratios measured earlier in this book. The point is the
SHAPE of the journey: which lever moves the number, and by how much, and where the
honest costs are (the tool-compression turn penalty, the subagent's own bill).

Prices, US dollars per million tokens (from chapter 2):
  opus-4-8   input  $5    output $25
  haiku-4-5  input  $1    output $5
Prompt caching (chapter 6): cache write 1.25x input, cache read 0.1x input.

Standard library only. Run:  python3 optimization_lab.py
"""

OPUS_IN, OPUS_OUT = 5.0, 25.0          # $/Mtok
HAIKU_IN, HAIKU_OUT = 1.0, 5.0
CACHE_WRITE, CACHE_READ = 1.25, 0.10   # multipliers on input price

M = 1_000_000


def simulate(p):
    """Cost of a session described by params p. Returns input tokens billed,
    output tokens, and total dollars (accounting for caching, compaction,
    delegation, and the semantic-cache skip)."""
    T = p["turns"]
    # Semantic cache: a fraction of turns are near-duplicates served from a
    # stored answer, so the model is never called for them.
    active = round(T * (1 - p["sem_hit"]))
    # Tool compression can cause occasional re-reads, adding a few turns.
    total_turns = active + p["extra_turns"]

    # What each turn appends to the conversation and re-sends forever after.
    tool_in_window = 200 if p["subagent"] else p["tool_window"]
    chunk = p["reads"] + tool_in_window + p["out"]

    prefix_cost = 0.0
    conv_tokens = 0          # the growing conversation, always full input price
    out_tokens = 0
    cumulative = 0
    for t in range(1, total_turns + 1):
        # The stable prefix: written once, then cache-read each later turn.
        if p["caching"]:
            mult = CACHE_WRITE if t == 1 else CACHE_READ
        else:
            mult = 1.0
        prefix_cost += p["prefix"] * mult * OPUS_IN / M

        cumulative += chunk
        ctx = cumulative
        if p["compact_budget"]:           # bound the re-sent history
            ctx = min(cumulative, p["compact_budget"])
        conv_tokens += ctx
        out_tokens += p["out"]

    conv_cost = conv_tokens * OPUS_IN / M
    out_cost = out_tokens * OPUS_OUT / M

    # Delegated verbose work runs in a cheap subagent, off the main window.
    sub_cost = 0.0
    if p["subagent"]:
        sub_in, sub_out = p["tool_window"] + 500, 200   # reads raw output, returns a summary
        sub_cost = total_turns * (sub_in * HAIKU_IN / M + sub_out * HAIKU_OUT / M)

    in_tokens = round(p["prefix"] * total_turns + conv_tokens)
    cost = prefix_cost + conv_cost + out_cost + sub_cost
    return dict(turns=total_turns, in_tok=in_tokens, out_tok=out_tokens, cost=cost)


# Baseline: a naive autopilot session. No caching, a big CLAUDE.md, whole-file
# reads, verbose tool output, verbose narration, unbounded history.
base = dict(turns=30, prefix=15000, reads=4000, tool_window=3000, out=800,
            caching=False, compact_budget=None, subagent=False, sem_hit=0.0,
            extra_turns=0)

# Each phase ADDS one lever on top of the previous (cumulative).
phases = [
    ("0  Baseline (naive autopilot)",        {}),
    ("1  + Prompt caching (stable prefix)",  dict(caching=True)),
    ("2  + Trim CLAUDE.md (1345->400 tok)",  dict(prefix=14055)),
    ("3  + Code-aware reads (4000->1200)",   dict(reads=1200)),
    ("4  + Tool-output compression (RTK)",   dict(tool_window=1740, extra_turns=2)),
    ("5  + Output reduction (effort/terse)", dict(out=280)),
    ("6  + Subagent delegation (haiku)",     dict(subagent=True, extra_turns=0)),
    ("7  + Compaction (bound history 30k)",  dict(compact_budget=30000)),
    ("8  + Semantic cache (20% hits)",       dict(sem_hit=0.20)),
]

# Apply cumulatively.
p = dict(base)
rows = []
for name, delta in phases:
    p = {**p, **delta}
    rows.append((name, simulate(p)))

base_cost = rows[0][1]["cost"]
print("=== One 30-turn autopilot session, optimized phase by phase ===")
print(f"{'Phase':<38}{'turns':>6}{'in tok':>12}{'out tok':>9}{'cost $':>9}{'vs base':>9}")
print("-" * 83)
for name, r in rows:
    red = (1 - r["cost"] / base_cost) * 100
    print(f"{name:<38}{r['turns']:>6}{r['in_tok']:>12,}{r['out_tok']:>9,}"
          f"{r['cost']:>9.2f}{red:>8.0f}%")
print("-" * 83)
fin = rows[-1][1]
print(f"\nFully optimized vs baseline: ${base_cost:.2f} -> ${fin['cost']:.2f} "
      f"= {(1-fin['cost']/base_cost)*100:.0f}% cheaper, "
      f"{base_cost/fin['cost']:.1f}x.")

# Standalone effect: what does each single lever do to the BASELINE alone?
print("\n=== Each lever ALONE on the baseline (to rank by raw leverage) ===")
solo = [
    ("Compaction (bound history 30k)", dict(compact_budget=30000)),
    ("Code-aware reads (4000->1200)",  dict(reads=1200)),
    ("Output reduction (800->280)",    dict(out=280)),
    ("Tool compression (3000->1740)",  dict(tool_window=1740, extra_turns=2)),
    ("Prompt caching",                 dict(caching=True)),
    ("Semantic cache (20% hits)",      dict(sem_hit=0.20)),
    ("Trim CLAUDE.md (1345->400)",     dict(prefix=14055)),
]
ranked = []
for name, delta in solo:
    r = simulate({**base, **delta})
    ranked.append((name, (1 - r["cost"] / base_cost) * 100))
for name, red in sorted(ranked, key=lambda x: -x[1]):
    print(f"  {name:<34} {red:>5.0f}% cheaper alone")
print("\nLesson: on a long session the biggest single lever is bounding the")
print("re-sent history (compaction), then cutting per-turn reads. Caching and a")
print("small CLAUDE.md help but cannot fix a conversation that grows unbounded.")

Running it produces the journey as one table:

=== One 30-turn autopilot session, optimized phase by phase ===
Phase                                  turns      in tok  out tok   cost $  vs base
-----------------------------------------------------------------------------------
0  Baseline (naive autopilot)             30   4,077,000   24,000    20.99       0%
1  + Prompt caching (stable prefix)       30   4,077,000   24,000    19.05       9%
2  + Trim CLAUDE.md (1345->400 tok)       30   4,048,650   24,000    19.03       9%
3  + Code-aware reads (4000->1200)        30   2,746,650   24,000    12.52      40%
4  + Tool-output compression (RTK)        32   2,424,480   25,600    10.82      48%
5  + Output reduction (effort/terse)      32   2,149,920    8,960     9.03      57%
6  + Subagent delegation (haiku)          30   1,202,850    8,400     4.50      79%
7  + Compaction (bound history 30k)       30   1,068,690    8,400     3.83      82%
8  + Semantic cache (20% hits)            24     804,360    6,720     2.83      87%
-----------------------------------------------------------------------------------

Fully optimized vs baseline: $20.99 -> $2.83 = 87% cheaper, 7.4x.

=== Each lever ALONE on the baseline (to rank by raw leverage) ===
  Compaction (bound history 30k)        66% cheaper alone
  Semantic cache (20% hits)             33% cheaper alone
  Code-aware reads (4000->1200)         31% cheaper alone
  Prompt caching                         9% cheaper alone
  Output reduction (800->280)            8% cheaper alone
  Tool compression (3000->1740)          3% cheaper alone
  Trim CLAUDE.md (1345->400)             1% cheaper alone

Lesson: on a long session the biggest single lever is bounding the
re-sent history (compaction), then cutting per-turn reads. Caching and a
small CLAUDE.md help but cannot fix a conversation that grows unbounded.

The journey, phase by phase

Read the table top to bottom as the lab progresses:

  • Phase 1, caching (9%). Turning on prompt caching saves the stable prefix's cost on every turn after the first, but the prefix is only 15k of a session whose conversation grows past four million re-sent input tokens. So caching is real money but a small share. The lesson lands early: caching helps, but it cannot rescue a session whose cost is the growing conversation.
  • Phase 2, trim CLAUDE.md (9%, no change). Shrinking CLAUDE.md from ~1,345 to 400 tokens barely moves the total here, because that prefix is already cached and tiny next to the conversation. It is worth doing for adherence and for the uncached case, but it is not where the money is in this scenario.
  • Phase 3, code-aware reads (40%). Reading only the relevant code (4,000 to 1,200 tokens per turn, Chapter 5) is the first big jump, because those reads land in the conversation and are re-sent on every later turn. Cutting per-turn input cuts it many times over.
  • Phase 4, tool-output compression (48%). Compressing noisy command output (Chapter 3) helps, but note the honest cost: the model adds 2 turns (30 to 32) to stand in for the occasional re-read when compression drops something the agent needed, exactly the instability from Chapter 20. The net is still positive, but the gain is smaller than the per-command headline.
  • Phase 5, output reduction (57%). Cutting narration from 800 to 280 output tokens (Chapter 4) is cheap to do and matters because output is billed at five times input.
  • Phase 6, subagent delegation (79%). The largest single jump. Moving the verbose tool work into a cheap haiku subagent (Chapter 13) takes thousands of tokens per turn out of the main window (so they stop being re-sent) and prices the detail work at a fifth. This is the lever the gauges chapter flagged from the real /usage readout.
  • Phase 7, compaction (82%). Bounding the re-sent history (Chapter 11) caps the conversation's growth. Its marginal effect here is modest only because the earlier phases already shrank each turn's chunk; on the naive baseline it is the single biggest lever (next section).
  • Phase 8, semantic cache (87%). Serving the 20% of turns that are near-duplicates from a stored answer (Chapter 7) skips the model entirely for them, taking the final step to 87% cheaper, a 7.4x reduction.

Which lever actually moves the needle

The second table is the more useful one for prioritizing, because it strips out the order. It runs each lever alone on the naive baseline and ranks them:

  1. Compaction, 66% alone. On a long session, the re-sent conversation is the cost, so bounding it is the single biggest win. This is why the real /usage panel warns that usage over 150k context is expensive even when cached.
  2. Semantic cache, 33%, and code-aware reads, 31%. Skipping duplicate turns and cutting per-turn reads are the next tier.
  3. Caching 9%, output reduction 8%, tool compression 3%, CLAUDE.md trim 1%. Real but secondary on their own. Tool compression at 3% alone, with its turn penalty, is the clearest illustration of why the field notes say to measure RTK's realized effect rather than trust the headline.

Remember. The order of leverage is not the order people reach for. Most start with caching and a trimmed CLAUDE.md (the smallest levers here) and never bound the conversation (the biggest). On a long session, compact and clear first, cut per-turn reads second, and delegate verbose work to cheap subagents. The prefix tuning is real but it is the finishing touch, not the foundation.

The proven stack

The fully optimized session that won the lab stacks, in priority order: bound the history (compact and clear between tasks), cut per-turn reads (code-aware, targeted), delegate verbose work to a cheap subagent, shorten the output (low effort, terse), cache the stable prefix (and keep CLAUDE.md small and stable), compress noisy tool output selectively (measured, not blanket), and serve near-duplicate turns from a semantic cache. Together they took the modeled session from $20.99 to $2.83, a 7.4x reduction, with no loss of what the task needed, because every cut removed tokens the work did not use.

That is the thesis of the whole book in one number. None of these levers is exotic, and no single one is a silver bullet; the win is in stacking them, biggest-leverage-first, and measuring as you go.

Reproducing this on a real session

The model is honest about being a model. To run the real version on your own work:

  1. Start a task and note the baseline with /usage and /context (Chapter 21).
  2. Apply the levers one at a time, in the priority order above, and re-check /usage after each on a comparable task. Add rtk gain and rtk cc-economics when you test tool compression so you catch the net effect, not the per-command savings.
  3. Keep what moved your realized number and drop what only looked good in theory. Your constants differ from the model's, so your ranking may reorder the middle of the pack, but the ends hold: bounding the conversation and delegating verbose work are almost always near the top, and prefix tuning near the bottom.

That loop, measure, apply the biggest lever, re-measure, is the practice the model only approximates. The lab tells you where to start; your own /usage tells you when you are done.

Further reading

  • Chapter 16 (the professional workflow) and Chapter 21 (reading the gauges): the plan and the measurement this lab sits between.
  • Chapter 20: the honest, measured view of the individual tools, including why tool compression underperforms its headline.
  • Claude Code, "Manage costs effectively" (code.claude.com/docs/en/costs): the real per-developer cost baselines and the /usage attribution that grounds the priority order here.

Takeaways

  • Stacking the levers on one modeled 30-turn autopilot session took it from $20.99 to $2.83, 87% cheaper, a 7.4x reduction, with no loss of needed context.
  • The order of leverage, measured alone on the baseline: compaction (66%), then semantic cache (33%) and code-aware reads (31%), then caching, output reduction, tool compression, and CLAUDE.md trim (all single digits).
  • Caching and a small CLAUDE.md are real but secondary; they cannot fix a conversation that grows unbounded, which is why bounding history and cutting per-turn reads come first.
  • Tool compression carries a turn penalty (here 30 to 32 turns) and only 3% alone, the clearest reason to measure its realized effect rather than trust the headline.
  • The model tells you where to start; reproduce the real ranking on your own work with /usage, /context, rtk gain, and rtk cc-economics, applying the biggest lever first and re-measuring.

👉 That is the journey end to end: one session, every lever, measured, with a proven order of attack. Every lever so far has worked the input side of the window. The next part turns to the output side and the tools that ground a model in real code: how the decoder is steered one logit at a time, how a language server becomes an agent capability, and how the two meet in Monitor-Guided Decoding. Continue to The decoder's dials: logits and how to control them.