The benchmark bench: proving a tool before you adopt it

TL;DR. A tool's headline ratio is a claim; your session's usage fields are the verdict. This chapter defines a repeatable A/B protocol for benchmarking any context tool inside Claude Code (same task, tool off then on, judged on the four usage fields from Chapter 23), then runs the bench on the tools installed on this machine, with real numbers: RTK compresses this repo's own command traffic 60 to 90 percent per command (79 percent on the mix), and its own meter reports 30.5 percent realized across everything it proxied; Headroom's LogCompressor hits 98 percent on git log but kept 14 of 1,329 lines, the chapter's central lesson that a ratio without a loss inspection is meaningless; three repo packers land within 1 percent of each other until tree-sitter compression opens a 39 percent gap; and the output shapers (caveman, claude-token-efficient) get a break-even formula instead of a slogan. A closing shelf maps every remaining relevant tool to its lever and chapter so nothing in the ecosystem is left unplaced.

Contents

• The protocol: benchmark inside Claude Code
• The bench
• RTK: the command-output proxy, proven and bounded
• Headroom: the compression library, and the loss lesson
• The packers, head to head
• Output shapers: caveman and the break-even
• The wider shelf: nothing left unplaced
• Further reading
• Takeaways

Chapter 26 drove seven tools end to end. This chapter is about the step that has to happen before any of them earns a permanent place in your setup: the benchmark. Every number here follows the discipline Chapter 20 established, because the field notes found the gap the hard way: a tool that saves 88 percent on one command saved 6.5 percent on a real session, and a tool can save tokens per command while costing money per task if it makes the agent re-read what compression dropped (Chapter 22 measured that turn penalty).

The protocol: benchmark inside Claude Code

Claude Code is more than the place these tools run; it is the measuring rig. Every instrument you need ships with it (Chapter 25), so the protocol costs nothing but a repeated task:

1. Pick a repeatable task. Something your real work does often and a tool claims to improve: "summarize the last 40 commits and what they changed", "find and fix the failing test", "map this module's dependencies". Write the prompt down; you will run it twice.
2. Baseline run. /clear, run the task, and record the evidence: the /context panel (Messages size), the session's line in ccusage session or your usage ledger (the four usage fields and cost), and the turn count from the transcript.
3. Enable the tool. Hook, MCP server, or CLAUDE.md edit, whatever the tool's integration is (each section below says exactly which). Note that a CLAUDE.md or tool-list change invalidates the cached prefix (Chapter 24), so the first turn after the change pays a write; judge from the second turn on.
4. Treatment run. /clear, same prompt, record the same numbers.
5. Judge on the deltas that matter, in this order: total session cost (from the ledger, with the multipliers), then prompt tokens per turn, then turn count (a tool that added turns may have lost even if per-turn tokens fell), then output tokens if the tool is output-side.
6. One variable at a time. Two tools enabled together tell you nothing about either.

Don't be confused. Three different numbers get called "savings" and they shrink in order. The per-command ratio (this chapter's bench) is the tool's ceiling on its best input. The realized share (RTK's gain meter, your ledger delta) is what it saved across everything it actually touched. The net effect is realized savings minus what instability cost you in extra turns and re-reads. Vendors quote the first; your bill only feels the third. The protocol above measures the third.

The bench

The lab benchmarks the two compression tools installed on this machine (RTK 0.43.0, Headroom 0.28.0) against this repository's real command traffic, plus a packer face-off. Sizes are exact characters with the chars/4 token estimate from Chapter 2; the ratios, which are what we are after, do not depend on the estimator.

"""Benchmarking the Claude Code companion tools on this repository. Real data.

Two of the tools this book keeps citing are installed on the build machine
(RTK 0.43.0 via Homebrew, Headroom 0.28.0 via pip), so instead of quoting
their READMEs we benchmark them against this repository's real command
output: the exact `git log`, `git diff`, `find`, and `ls` traffic a Claude
Code session doing book work generates.

Methodology, stated up front so the numbers are honest:

  - Sizes are measured in CHARACTERS (exact) and converted to an estimated
    token count with the chars/4 rule of thumb from Chapter 2. Tool-output
    text is ordinary ASCII, where that estimate is decent, but it is an
    ESTIMATE: for billing-grade numbers, re-count with the provider's
    count_tokens. Ratios (the thing we care about) are robust to this.
  - Every command runs live in this script; nothing is copied from a vendor
    benchmark. Your repo will produce different absolute numbers and similar
    shapes.
  - A compression ratio is not a session saving. Chapter 20 measured why:
    the realized saving depends on how much of YOUR session's traffic is
    compressible command output at all.

Requires: rtk on PATH, headroom-ai installed. Standard library otherwise.
"""

import subprocess

# Every command runs from the repository root, so the script measures the
# same traffic no matter where you invoke it from.
ROOT = subprocess.run(["git", "rev-parse", "--show-toplevel"],
                      capture_output=True, text=True).stdout.strip()


def sh(cmd):
    """Run a shell command at the repo root, return stdout (stderr dropped)."""
    return subprocess.run(cmd, shell=True, capture_output=True,
                          text=True, cwd=ROOT).stdout


def est_tokens(s):
    return len(s) // 4


ROW = "{:<34}{:>10,}{:>10,}{:>8}"


def bench_rtk():
    print("=== 1. RTK: the same command, native vs proxied ===")
    print("Characters of output an agent would ingest (est. tokens = chars/4).\n")
    print(f"{'command':<34}{'native':>10}{'rtk':>10}{'saved':>8}")
    cases = [
        ("git log --stat -n 40", "rtk git log --stat -n 40"),
        ("git diff HEAD~2", "rtk git diff HEAD~2"),
        ("find . -name '*.md' -not -path './.git/*'",
         "rtk find . -name '*.md'"),
        ("ls -la books/context-engineering/src",
         "rtk ls books/context-engineering/src"),
    ]
    tot_a = tot_b = 0
    for native, proxied in cases:
        a, b = len(sh(native)), len(sh(proxied))
        tot_a += a
        tot_b += b
        label = native if len(native) <= 33 else native[:30] + "..."
        print(ROW.format(label, a, b, f"{(1 - b / a) * 100:.0f}%"))
    print(ROW.format("TOTAL (est. tokens)", est_tokens_total(tot_a),
                     est_tokens_total(tot_b), f"{(1 - tot_b / tot_a) * 100:.0f}%"))
    print()


def est_tokens_total(chars):
    return chars // 4


def bench_headroom():
    print("=== 2. Headroom: transform-level compression of the same traffic ===")
    from headroom.transforms import LogCompressor, DiffCompressor

    log = sh("git log --stat -n 40")
    diff = sh("git diff HEAD~2")

    r1 = LogCompressor().compress(log)
    r2 = DiffCompressor().compress(diff)
    print(f"{'input':<34}{'chars in':>10}{'chars out':>11}{'saved':>8}")
    for label, before, res in (("git log --stat -n 40 (LogCompr.)", log, r1),
                               ("git diff HEAD~2 (DiffCompr.)", diff, r2)):
        out = res.compressed
        print(f"{label:<34}{len(before):>10,}{len(out):>11,}"
              f"{(1 - len(out) / len(before)) * 100:>7.0f}%")
    print(f"\nLogCompressor detected format '{r1.format_detected}', kept "
          f"{r1.compressed_line_count} of {r1.original_line_count} lines.")
    print("First lines of the compressed log, so you can judge the loss:")
    for line in r1.compressed.splitlines()[:6]:
        print("  " + line[:76])
    print()


def bench_packers():
    print("=== 3. Repo packers: three ways to flatten code for one context ===")
    target = "books/context-engineering/code"
    ftp = sh(f"files-to-prompt {target}")
    ftp_c = sh(f"files-to-prompt --cxml {target}")
    print(f"{'packer':<34}{'chars':>10}{'~tokens':>10}")
    print(f"{'files-to-prompt (plain)':<34}{len(ftp):>10,}{est_tokens(ftp):>10,}")
    print(f"{'files-to-prompt --cxml':<34}{len(ftp_c):>10,}{est_tokens(ftp_c):>10,}")
    print(f"{'repomix (measured in Ch 26)':<34}{'':>10}{37906:>10,}")
    print(f"{'repomix --compress (Ch 26)':<34}{'':>10}{23093:>10,}")
    print("\nSame 18 files every time. The spread between a plain concatenation")
    print("and a tree-sitter-compressed pack is the Chapter 5 lesson in tool")
    print("form: structure-aware selection is worth ~40% before any model runs.")


if __name__ == "__main__":
    bench_rtk()
    bench_headroom()
    bench_packers()

Running it:

=== 1. RTK: the same command, native vs proxied ===
Characters of output an agent would ingest (est. tokens = chars/4).

command                               native       rtk   saved
git log --stat -n 40                  74,578    11,994     84%
git diff HEAD~2                      105,726    26,567     75%
find . -name '*.md' -not -path...     12,799     1,247     90%
ls -la books/context-engineeri...      2,125       844     60%
TOTAL (est. tokens)                   48,807    10,163     79%

=== 2. Headroom: transform-level compression of the same traffic ===
input                               chars in  chars out   saved
git log --stat -n 40 (LogCompr.)      74,578      1,227     98%
git diff HEAD~2 (DiffCompr.)         105,726    103,713      2%

LogCompressor detected format 'LogFormat.GENERIC', kept 14 of 1329 lines.
First lines of the compressed log, so you can judge the loss:
      - /usage dissected: the plan-limit bars (5hr session 97%, weekly 61% on 
        a dollar bill), and the "what's contributing" lines (90% >150k context
        subagent-heavy, 34% general-purpose subagent) each mapped to what it m
        what to do. Captures the critical reading caveat printed on the panel:
        independent characteristics, not a breakdown, and are approximate/loca
      - Observations and findings: this session is the expensive archetype (lo

=== 3. Repo packers: three ways to flatten code for one context ===
packer                                 chars   ~tokens
files-to-prompt (plain)              151,215    37,803
files-to-prompt --cxml               152,770    38,192
repomix (measured in Ch 26)                     37,906
repomix --compress (Ch 26)                      23,093

Same 18 files every time. The spread between a plain concatenation
and a tree-sitter-compressed pack is the Chapter 5 lesson in tool
form: structure-aware selection is worth ~40% before any model runs.

The rest of the chapter reads these results tool by tool, and for each one answers the three questions the user of a coding agent actually has: how is it implemented, how does it plug into Claude Code, and what did the benchmark prove and bound.

RTK: the command-output proxy, proven and bounded

How it is implemented. RTK (github.com/rtk-ai/rtk, brew install rtk) is a Rust CLI proxy: rtk git log runs the native command and rewrites its output with per-command filters (deduplicate, strip decoration, collapse repetition) before the text ever exists in the terminal. It is Chapter 3's tool-output compression at the source, which is the right place: output compressed before the agent reads it never enters the window, so it also never gets re-sent on every later turn.

How it plugs into Claude Code. rtk init -g writes a hook into Claude Code's settings that intercepts Bash tool calls and routes supported commands through the proxy, so the agent's own git, find, and grep traffic is compressed without the model doing anything differently. The two built-in meters are the benchmark half of the tool: rtk gain reports realized savings across every proxied command, and rtk cc-economics tries to reconcile that against your Claude Code spend (on this box that reconciliation currently fails against the latest ccusage JSON format, a useful reminder that glue between fast-moving tools is the first thing to break; the ledger from Chapter 25 answers the same question from the transcripts directly).

What the bench proved. Per command, on this repo's real traffic: 84 percent on git log --stat, 75 percent on a large diff, 90 percent on find, 60 percent on ls, 79 percent across the mix. And RTK's own meter, across the 25 commands it has proxied on this machine, reports the realized number:

Total commands:    25
Input tokens:      548.8K
Output tokens:     381.4K
Tokens saved:      167.4K (30.5%)

 1.  rtk git diff HEAD~2           4  81.4K   75.4%
 2.  rtk git log --stat -n 40      4  56.0K   82.4%
 3.  rtk find                      7  10.8K   62.0%
 ...
 9.  rtk read                      2      0    0.0%

What bounds it. Three honest limits, all visible in the numbers. First, the realized share falls as your mix shifts toward file reads: rtk read saved exactly 0 percent here, and Chapter 20 measured a source-heavy session at 6.5 percent realized against 60-to-90 headline ratios, because one incompressible file read dominated the session. Second, compression is lossy by design, and the failure mode is silent: if the filter drops the one line the agent needed, the agent pays a turn to re-fetch it, the instability the optimization lab priced at two extra turns. Scope RTK to noisy, skimmable commands (logs, finds, test output) and keep it away from anything the agent must see verbatim. Third, judge it end to end: run the protocol above and compare session cost, not rtk gain, because the meter cannot see re-reads.

Headroom: the compression library, and the loss lesson

How it is implemented. Headroom (github.com/chopratejas/headroom, pip install headroom-ai) is a Python context-compression layer: a pipeline of content-typed transforms (LogCompressor, DiffCompressor, SearchCompressor, TabularCompressor, a tree-sitter CodeAwareCompressor), a CacheAligner that reorders blocks stable-first (the Chapter 24 rules as code), a SemanticCache, a Memory store, and headroom learn for mining failed sessions into rule edits (Chapter 12). A HeadroomClient wraps your Anthropic client so the pipeline runs on every request; it defaults to an audit mode that measures what it would save before you let it change anything, which is exactly the right instinct.

How it plugs into Claude Code. Three ways, increasing in commitment: as a library inside your own agents and MCP servers (compress a noisy result before returning it to the model); as an MCP server registered with claude mcp add, exposing compression as tools the agent can call on demand; or as a proxy in front of the API for harnesses you control. For Claude Code itself, the MCP route is the practical one, and audit mode plus the bench below is how you decide whether to bother.

What the bench proved, and the lesson. The transform-level numbers on the same real traffic RTK saw are the most instructive in this chapter, because they bracket the failure modes from both sides:

• LogCompressor on git log --stat: 98 percent saved, and it kept 14 of 1,329 lines. It classified the git log as a generic log and kept a handful of body lines; every commit hash, author, date, and file stat is gone. For "what changed lately?" the answer is unusable. The ratio is spectacular because the information is gone.
• DiffCompressor on the same diff RTK cut 75 percent: 2 percent saved. It is conservative where the diff is genuinely dense, which is safe and honest, and also means no free lunch.

Neither number is a defect report; both are the same lesson from opposite directions. A compression ratio means nothing until you have read what survived. That is why the lab prints the first lines of the compressed output instead of only the ratio, why Headroom's own audit mode exists, and why the protocol at the top of this chapter ends with session cost rather than per-transform percentages. When you evaluate any compressor, yours or a vendor's, put the loss inspection in the loop: ratio, then surviving text, then the task-level A/B.

The packers, head to head

How they are implemented. Three open-source ways to flatten a codebase into one document: files-to-prompt (github.com/simonw/files-to-prompt, pip install files-to-prompt), a minimal concatenator with Claude-friendly --cxml output; Repomix (Chapter 26's walkthrough) with its token census and tree-sitter --compress; and gitingest (github.com/coderamp-labs/gitingest), the same idea aimed at remote URLs (swap github.com for gitingest.com on any repo URL). All are Chapter 5 in tool form.

How they plug into Claude Code. A pack is a context you hand to a fresh session: files-to-prompt books/context-engineering/code --cxml > pack.xml then claude "Read pack.xml and map the dependencies", or paste into claude.ai for a model with no filesystem. Benchmark with /context after the read: the pack's token count lands in Messages, and the question is whether one packed read beats the agent crawling files itself, which the transcript turn count answers.

What the bench proved. On the same 18 files: plain concatenation costs the same no matter who does it (files-to-prompt 37,803 estimated tokens, Repomix 37,906 exact, a rounding error apart), the --cxml framing adds about 1 percent, and the only lever that moves the number is structural: tree-sitter compression at 23,093 tokens, 39 percent below every plain pack. Pick a packer for its workflow (Repomix for the census and compression, files-to-prompt for zero-dep scripting, gitingest for repos you have not cloned); pick compression for the tokens.

Output shapers: caveman and the break-even

How they are implemented. caveman (github.com/JuliusBrussee/caveman) and claude-token-efficient (github.com/drona23/claude-token-efficient) are not programs; they are instructions: a skill and a drop-in CLAUDE.md block that force a terse output style (fragments, no preamble, no recap), attacking the 5x-priced half of the bill (Chapter 4).

How they plug into Claude Code. Paste the block into CLAUDE.md or install the skill; that is the entire integration. Which is also the cost: the instruction text itself is prefix tokens paid on every turn, and editing CLAUDE.md invalidates the cached prefix once.

What to measure, since this box cannot. Style effects need live generations to benchmark, so here the honest bench is the formula plus Chapter 20's field measurements (roughly 65 percent output reduction claimed for caveman-style prompts, netting only at high output volume). The break-even, with output at 5x input (Chapter 2): an instruction of $I$ tokens rides in the prefix each turn at the cached-read rate, so it pays for itself when

$$\text{output tokens cut per turn} ;>; \frac{I \times 0.1}{5} ;=; 0.02,I$$

A 300-token terse-style block breaks even by cutting just 6 output tokens per turn, which is why these nearly always net positive on chatty sessions, and why the real question is quality: run the protocol, and read the terse answers the way we read Headroom's surviving lines. If you have to ask a follow-up to decode a fragment, the turn you added cost more than the style saved. Measure output tokens per turn before and after with the ledger (the output_tokens column divided by turns), not by impression.

The wider shelf: nothing left unplaced

The bench covered what is installable here. The rest of the prominent, actively maintained ecosystem, placed by lever so you can find the chapter that explains it and the tour or map entry that runs it:

Serena deserves the one extra sentence because it is the most Claude-Code-native entry not yet benchmarked in this book: registered with claude mcp add serena ..., it replaces "read the whole file" with language-server operations (find_symbol, references, targeted edits), which attacks the per-turn read size the optimization lab ranked as the second-biggest lever. Benchmark it with exactly the protocol above: same refactor task, /clear, with and without, judged on prompt tokens per turn and turn count.

Remember. The shelf will be stale before the print dries; the protocol will not. Any new tool that claims to save context reduces to one of this book's levers, plugs into Claude Code through one of three doors (hook, MCP server, CLAUDE.md), and submits to the same two-run A/B on your own task. If a tool cannot survive that bench, its README numbers do not matter.

Further reading

• RTK (github.com/rtk-ai/rtk): the hook installation (rtk init -g) and the meters (rtk gain, rtk cc-economics); Chapter 20 for the realized-vs- headline field measurements and stability notes.
• Headroom (github.com/chopratejas/headroom): the transform and audit-mode docs; its CacheAligner against Chapter 24's rules.
• files-to-prompt (github.com/simonw/files-to-prompt), gitingest (github.com/coderamp-labs/gitingest), code2prompt (github.com/mufeedvh/code2prompt), Serena (github.com/oraios/serena): the packers and the symbol-level alternative.
• Claude Code hooks and MCP (code.claude.com/docs): the two integration doors every tool in this chapter walks through, covered in Chapter 19.

Takeaways

• Benchmark inside Claude Code with a two-run protocol: same task, /clear both times, tool off then on, judged on session cost from the ledger, then prompt tokens per turn, then turn count. One variable at a time, and remember a config change busts the cached prefix for one turn.
• RTK, measured on this repo: 60 to 90 percent per command (79 percent on the mix), 30.5 percent realized across everything it proxied, 0 percent on file reads. Scope it to noisy commands and judge it end to end, because its meter cannot see the re-reads its losses cause.
• Headroom's bench is the chapter's lesson: 98 percent on a git log by discarding 1,315 of 1,329 lines, and 2 percent on a dense diff. A ratio without a loss inspection is meaningless; always read what survived.
• Packers converge (files-to-prompt and Repomix within 1 percent plain); only structure-aware compression moves the number (39 percent). Choose the packer by workflow and the tokens by tree-sitter.
• Output shapers break even at about 2 percent of their instruction length in output tokens cut per turn; the risk is quality, so benchmark the answers alongside the counts.
• Every remaining relevant tool maps onto the shelf by lever (Serena, gitingest, code2prompt, LMCache, DSPy, Helicone, Phoenix, OpenLLMetry, and the rest), enters Claude Code through a hook, an MCP server, or CLAUDE.md, and faces the same protocol.

👉 That is the bench: a protocol that outlives any tool list, and real numbers for the tools on this machine. Next, we take the tool that scored best on this machine's own traffic and put it under the microscope: RTK's actual source, stage by stage, down to the four lines that compute every number its meter reports.