The cache-shape lab: how the prompt decides the hit

TL;DR. Prompt caching is a prefix match on the exact bytes of the rendered request, and that one rule decides every hit and every silent miss. This chapter turns Chapter 6's mechanism into a lab: a from-scratch model of the provider cache (breakpoints, TTLs, per-model minimums) and six measured experiments. A timestamp inside the system prompt takes the hit rate from 4/5 to 0/5; moving the same timestamp after the breakpoint restores it; reordering tools kills everything; a 3,000-token prefix on claude-opus-4-8 silently caches nothing because the minimum there is 4,096; and the 5-minute versus 1-hour TTL choice flips winner purely on the gap between your calls, with the 5m cache costing more than no cache once gaps pass its TTL.

Contents

• The rules the provider actually enforces
• The lab
• Reading the experiments
• The TTL decision, measured
• The rules beyond the lab
• The silent-invalidator audit
• Further reading
• Takeaways

Chapter 6 built the KV cache itself and priced the economics. What it could not do in one chapter is show you how fragile the hit is, and how completely the shape of your request controls it. That fragility is where real money leaks: a cache that misses does not error, does not warn, and does not even look wrong in the response. It just quietly bills you 10x more per prompt token, forever, until someone reads the usage fields from Chapter 23 and notices cache_read_input_tokens is zero.

The rules the provider actually enforces

Five rules, all of which the lab implements faithfully:

1. Prefix match on exact bytes. The cache key is a hash of the rendered prompt up to each breakpoint, in render order: tools, then system, then messages. One changed byte at position $N$ invalidates every cache entry at positions $\geq N$.

2. Breakpoints, at most 4. A cache_control: {"type": "ephemeral"} marker on a content block says "cache everything up to and including me". More than four is rejected.

3. A per-model minimum. Below the minimum prefix length, a breakpoint is accepted and nothing is cached: no error, just cache_creation_input_tokens: 0.

   Model	Minimum cacheable prefix
   claude-opus-4-8, claude-haiku-4-5	4,096 tokens
   claude-fable-5, claude-sonnet-4-6	2,048 tokens
   claude-sonnet-4-5 and older Sonnets	1,024 tokens

   The same 3,000-token prompt caches on Sonnet 4.5 and silently does not on Opus 4.8. When you change models, re-check this table before trusting your hit rate.

4. A TTL, refreshed by use. Entries live 5 minutes by default, or 1 hour with "ttl": "1h". Every read refreshes the timer at no cost, so steady traffic keeps a 5-minute entry alive indefinitely. Writes cost 1.25x (5m) or 2x (1h) the input rate; reads cost 0.1x.

5. Everything is scoped. Caches are isolated per organization and workspace, and keyed to the model: switching models mid-conversation is a full miss even with identical text.

The lab

The model below is small enough to read in one sitting and honest about which rules it encodes: prefix hashing over segment content, up to four breakpoints, per-model minimums, TTL expiry with refresh-on-use, and the three usage fields as output, so every experiment reports exactly what response.usage would.

"""How prompt SHAPE decides the cache hit, measured on a from-scratch model.

Prompt caching is a prefix match: the cache key is the exact bytes of the
rendered prompt up to each breakpoint. That one rule explains every hit and
every silent miss. This lab builds a faithful little model of the provider's
cache (prefix hashing, breakpoints, TTL expiry, per-model minimums) and then
runs the classic experiments against it:

  A. an identical, stable prefix        -> one write, then reads forever
  B. a timestamp in the system prompt   -> every request misses, silently
  C. the same timestamp moved AFTER the breakpoint -> hits restored
  D. reordering the tool list           -> full miss (tools render first)
  E. a prefix below the model minimum   -> marker present, nothing cached
  F. 5-minute vs 1-hour TTL economics   -> break-even depends on call gaps

Every experiment reports the same three usage fields the real API reports
(input_tokens, cache_creation_input_tokens, cache_read_input_tokens), so you
can rehearse here exactly what you will later read off `response.usage`.

Run with the standard library only.
"""

import hashlib

# Per-model minimum cacheable prefix. Below this, a breakpoint is accepted
# but NOTHING is cached: no error, just cache_creation_input_tokens == 0.
CACHE_MINIMUM = {
    "claude-opus-4-8": 4096,
    "claude-sonnet-4-6": 2048,
    "claude-haiku-4-5": 4096,
}

IN_RATE = 5.00 / 1_000_000        # opus-4-8 input, per token
READ_RATE = IN_RATE * 0.10
WRITE_RATE = {"5m": IN_RATE * 1.25, "1h": IN_RATE * 2.00}
TTL_SECONDS = {"5m": 300, "1h": 3600}


class PrefixCache:
    """The provider's prompt cache, reduced to its load-bearing rules."""

    def __init__(self, model="claude-opus-4-8"):
        self.model = model
        self.entries = {}   # prefix-hash -> (token_count, expires_at, ttl)
        self.minimum = CACHE_MINIMUM[model]

    def request(self, segments, breakpoints, now, ttl="5m"):
        """Process one request.

        segments:    ordered list of (name, token_count, content) exactly as
                     the prompt renders: tools first, then system, then messages.
        breakpoints: indexes of segments that carry cache_control (max 4).
        Returns a usage dict with the three input-side fields.
        """
        assert len(breakpoints) <= 4, "the API allows at most 4 breakpoints"
        # Hash the exact content of the prefix up to each breakpoint. One
        # changed byte anywhere -> a different hash -> a different cache line.
        read = 0
        write = 0
        covered = 0     # tokens accounted for by cache read/write so far
        prefix_hash = hashlib.sha256()
        tokens_so_far = 0
        for i, (name, count, content) in enumerate(segments):
            prefix_hash.update(f"{name}|{content}|".encode())
            tokens_so_far += count
            if i in breakpoints:
                key = prefix_hash.hexdigest()
                entry = self.entries.get(key)
                if entry and entry[1] > now:
                    # HIT: this whole prefix is served from cache. Using an
                    # entry also refreshes its TTL at no extra cost.
                    read = tokens_so_far
                    write = 0
                    self.entries[key] = (tokens_so_far, now + TTL_SECONDS[entry[2]], entry[2])
                elif tokens_so_far >= self.minimum:
                    # MISS above the minimum: pay the write premium to store it
                    # (minus whatever a shorter breakpoint already covered).
                    write = tokens_so_far - read
                    self.entries[key] = (tokens_so_far, now + TTL_SECONDS[ttl], ttl)
                # Below the minimum: silently no cache at all.
                covered = read + write
        plain = sum(c for _, c, _ in segments) - covered
        return {"input_tokens": plain,
                "cache_creation_input_tokens": write,
                "cache_read_input_tokens": read}


def show(label, usages):
    print(f"--- {label} ---")
    print(f"{'req':>4}{'input':>9}{'write':>9}{'read':>9}")
    for i, u in enumerate(usages, 1):
        print(f"{i:>4}{u['input_tokens']:>9,}"
              f"{u['cache_creation_input_tokens']:>9,}"
              f"{u['cache_read_input_tokens']:>9,}")
    hits = sum(1 for u in usages if u["cache_read_input_tokens"] > 0)
    print(f"     cache hits: {hits}/{len(usages)}\n")


def experiment_stable_vs_timestamp():
    tools = ("tools", 3000, "read,edit,bash,grep sorted deterministically")
    system = ("system", 9000, "You are a careful coding agent. <frozen>")

    # A: stable prefix, breakpoint on the system block. 5 identical requests.
    cache = PrefixCache()
    usages = []
    for t in range(5):
        question = ("user", 200, f"question number {t}")
        usages.append(cache.request([tools, system, question],
                                    breakpoints={1}, now=t * 30))
    show("A. stable prefix, 5 requests 30s apart", usages)

    # B: same requests, but the system prompt interpolates the clock.
    cache = PrefixCache()
    usages = []
    for t in range(5):
        stamped = ("system", 9000, f"You are a careful agent. now={t * 30}s")
        question = ("user", 200, f"question number {t}")
        usages.append(cache.request([tools, stamped, question],
                                    breakpoints={1}, now=t * 30))
    show("B. timestamp INSIDE the system prompt (silent invalidator)", usages)

    # C: the timestamp moved into the user turn, AFTER the breakpoint.
    cache = PrefixCache()
    usages = []
    for t in range(5):
        question = ("user", 200, f"now={t * 30}s question number {t}")
        usages.append(cache.request([tools, system, question],
                                    breakpoints={1}, now=t * 30))
    show("C. same timestamp moved AFTER the breakpoint", usages)

    # D: request 2 reorders the tool list. Tools render at position 0.
    cache = PrefixCache()
    usages = [cache.request([tools, system, ("user", 200, "q1")],
                            breakpoints={1}, now=0)]
    shuffled = ("tools", 3000, "bash,edit,grep,read shuffled this time")
    usages.append(cache.request([shuffled, system, ("user", 200, "q2")],
                                breakpoints={1}, now=30))
    show("D. request 2 reorders the tools (position 0)", usages)

    # E: a prefix below the model minimum. Marker present, nothing cached.
    cache = PrefixCache("claude-opus-4-8")
    small = ("system", 3000, "short prompt")           # < 4096 minimum
    usages = [cache.request([small, ("user", 100, f"q{t}")],
                            breakpoints={0}, now=t * 30) for t in range(3)]
    show("E. 3,000-token prefix on opus-4-8 (minimum 4,096): silently uncached",
         usages)


def experiment_ttl_economics():
    print("--- F. 5-minute vs 1-hour TTL, by traffic pattern ---")
    prefix_tokens = 20_000
    print(f"A {prefix_tokens:,}-token prefix, 20 calls, at three call gaps.\n")
    print(f"{'gap between calls':>20}{'uncached':>10}{'5m TTL':>9}{'1h TTL':>9}")
    for gap, label in ((60, "60 seconds"), (600, "10 minutes"),
                       (3000, "50 minutes"), (5400, "90 minutes")):
        uncached = prefix_tokens * IN_RATE * 20
        costs = {}
        for ttl in ("5m", "1h"):
            cache = PrefixCache()
            total = 0.0
            seg = [("system", prefix_tokens, "frozen"), ("user", 100, "q")]
            for t in range(20):
                u = cache.request(seg, breakpoints={0}, now=t * gap, ttl=ttl)
                total += (u["input_tokens"] * IN_RATE
                          + u["cache_creation_input_tokens"] * WRITE_RATE[ttl]
                          + u["cache_read_input_tokens"] * READ_RATE)
            costs[ttl] = total
        print(f"{label:>20}{uncached:>10.2f}{costs['5m']:>9.2f}{costs['1h']:>9.2f}")
    print("\nRead the rows against each other. Steady traffic: 5m wins (cheaper")
    print("writes, and every USE refreshes the timer for free, so it never")
    print("expires). 10-minute gaps: the 5m entry is always dead on arrival, so")
    print("every call pays a fresh 1.25x write and the '5m cache' costs MORE")
    print("than no cache at all; the 1h entry survives and keeps paying. Even")
    print("50-minute gaps stay warm on 1h thanks to refresh-on-use. Past the")
    print("1-hour TTL both always expire: every call is a pure write premium,")
    print("and the honest move is to turn caching off for that traffic.")


if __name__ == "__main__":
    experiment_stable_vs_timestamp()
    experiment_ttl_economics()

Running it:

--- A. stable prefix, 5 requests 30s apart ---
 req    input    write     read
   1      200   12,000        0
   2      200        0   12,000
   3      200        0   12,000
   4      200        0   12,000
   5      200        0   12,000
     cache hits: 4/5

--- B. timestamp INSIDE the system prompt (silent invalidator) ---
 req    input    write     read
   1      200   12,000        0
   2      200   12,000        0
   3      200   12,000        0
   4      200   12,000        0
   5      200   12,000        0
     cache hits: 0/5

--- C. same timestamp moved AFTER the breakpoint ---
 req    input    write     read
   1      200   12,000        0
   2      200        0   12,000
   3      200        0   12,000
   4      200        0   12,000
   5      200        0   12,000
     cache hits: 4/5

--- D. request 2 reorders the tools (position 0) ---
 req    input    write     read
   1      200   12,000        0
   2      200   12,000        0
     cache hits: 0/2

--- E. 3,000-token prefix on opus-4-8 (minimum 4,096): silently uncached ---
 req    input    write     read
   1    3,100        0        0
   2    3,100        0        0
   3    3,100        0        0
     cache hits: 0/3

--- F. 5-minute vs 1-hour TTL, by traffic pattern ---
A 20,000-token prefix, 20 calls, at three call gaps.

   gap between calls  uncached   5m TTL   1h TTL
          60 seconds      2.00     0.33     0.40
          10 minutes      2.00     2.51     0.40
          50 minutes      2.00     2.51     0.40
          90 minutes      2.00     2.51     4.01

Read the rows against each other. Steady traffic: 5m wins (cheaper
writes, and every USE refreshes the timer for free, so it never
expires). 10-minute gaps: the 5m entry is always dead on arrival, so
every call pays a fresh 1.25x write and the '5m cache' costs MORE
than no cache at all; the 1h entry survives and keeps paying. Even
50-minute gaps stay warm on 1h thanks to refresh-on-use. Past the
1-hour TTL both always expire: every call is a pure write premium,
and the honest move is to turn caching off for that traffic.

Reading the experiments

A is the baseline and the shape you want everywhere: one write, then reads. The healthy signature in production usage blocks is exactly this: a cache_creation spike on the first request, then cache_read covering the prefix on every request after.

B is the most expensive bug in prompt engineering, and it is invisible. The requests in B are semantically identical to A; the only difference is a clock interpolated into the system prompt. Because the cache matches bytes, not meaning, every request writes a brand-new entry that nothing will ever read. Note what the usage columns show: the write column full every time, the read column zero forever. B does not merely lose the discount; it pays the 1.25x write premium on all 12,000 tokens on every single call, so the "cached" system is 25 percent more expensive than never caching at all.

C is the fix, and it costs nothing. The same volatile timestamp, moved after the breakpoint into the user turn, restores 4/5 hits. The rule generalizes: stable content before the marker, volatile content after it. You rarely need to delete dynamic context; you need to relocate it.

D shows why tools are the highest-stakes segment. Tools render at position 0, ahead of the system prompt and everything else, so an unstable tool list (a set iterated in random order, a per-user tool subset, JSON serialized without sorted keys) invalidates the entire cache including segments that did not change. Serialize tools deterministically and never vary the set mid-conversation.

E is the miss that no experiment upstream would catch, because nothing is wrong with the request. The marker is present and valid; the prefix is simply below the model's minimum, so the provider declines to cache and says so only by omission (write and read both zero, all 3,100 tokens billed as plain input). This is also a migration hazard: a 3,000-token prompt that cached fine on an older Sonnet stops caching when you upgrade to a model with a 4,096 minimum, and the only symptom is a slightly larger bill.

The TTL decision, measured

Experiment F is the one to internalize, because it turns "which TTL should I use?" from a style preference into arithmetic on one variable: the gap between calls that share the prefix.

• Gaps under 5 minutes: the 5m cache wins outright. Writes are cheaper (1.25x vs 2x), and since every read refreshes the timer for free, steady traffic never lets the entry expire. This is why interactive chat and busy services default to 5m.
• Gaps between 5 minutes and 1 hour: the 5m entry is dead on arrival every time, so each call pays a fresh write premium and the 5m column lands above uncached ($2.51 vs $2.00). The 1h entry survives the gap and keeps the discount ($0.40). This is the coding-agent regime: you read a file, think, test, come back 12 minutes later. It is exactly why Claude Code's own usage blocks show ephemeral_1h_input_tokens (Chapter 23): the harness buys the doubled write to survive your coffee breaks.
• Gaps beyond 1 hour: both TTLs always expire, every call is a pure write premium, and caching is a net loss. The honest configuration for genuinely sporadic traffic is no cache_control at all, or a scheduled pre-warm if first-token latency matters (below).

The break-even math from first principles: with the 5m TTL, a write plus one read costs $1.25 + 0.1 = 1.35$ input-equivalents against $2.0$ uncached, so it pays for itself on the second request. The 1h TTL costs $2.0 + 0.1 = 2.1$ against $2.0$ after one reuse and $2.2$ against $3.0$ after two, so it needs at least two reads within the hour to win.

The rules beyond the lab

Three production rules did not fit a six-experiment lab but bite often enough to know cold:

• The invalidation hierarchy is tiered, not total. Changing tool definitions or the model invalidates everything. Changing the system prompt invalidates system and messages but leaves a tools-only cache tier intact. Changing tool_choice or toggling thinking invalidates only the messages tier. So you can vary tool_choice per request without losing the tools+system cache; you cannot touch the tool list without losing it all.
• The lookback window is 20 blocks. A breakpoint searches backward at most 20 content blocks for a prior cache entry. An agent turn that appends more than 20 blocks (a burst of parallel tool_use/tool_result pairs does this easily) pushes the previous entry out of reach, and the next request misses silently. The fix is an intermediate breakpoint every 15 blocks or so in long turns.
• Parallel first requests all miss. An entry becomes readable only after the first response begins streaming, so $N$ concurrent requests with the same cold prefix all pay full price. For fan-out, send one request, wait for its first streamed token, then fire the rest; they read the entry the first one just wrote. To hide even the first miss, pre-warm at startup with a max_tokens: 0 request carrying the breakpoint: the API runs prefill, writes the cache, and returns immediately with no output billed.

The silent-invalidator audit

When cache_read_input_tokens is zero across requests that should share a prefix, one of these is almost always the culprit. Grep for them in anything that feeds the prompt:

The diagnostic that settles any doubt costs two API calls: send the identical request twice and read the usage block. Write-then-read means healthy; write-then-write means one of the rows above is live in your prompt path.

Remember. The cache never tells you it missed. The only witnesses are cache_creation_input_tokens and cache_read_input_tokens, which is why Chapter 23 insisted you read them per turn and the next chapter aggregates them across every session on the machine. Shape the prompt for the cache first (stable prefix, volatile tail, deterministic serialization), then verify with the fields, then stop thinking about it.

Further reading

• The Anthropic prompt-caching documentation (platform.claude.com/docs), the authoritative source for breakpoints, TTLs, per-model minimums, and the invalidation tiers modeled here.
• Chapter 6 for the KV-cache mechanism underneath and the base economics; Chapter 8 for what the serving engine does with the same prefixes; Chapter 7 for the different cache that matches meaning instead of bytes.
• Chapter 20 for the field measurements of TTL refresh behavior inside Claude Code sessions.

Takeaways

• One rule explains everything: the cache matches exact prefix bytes up to each breakpoint, in tools-system-messages render order. Semantics never matter; bytes always do.
• The lab's numbers: a stable prefix hits 4/5; a timestamp in the system prompt hits 0/5 and costs 25 percent more than no caching; the same timestamp after the breakpoint restores 4/5; a tool reorder kills everything; a below-minimum prefix caches nothing, silently.
• Minimums are per model (4,096 on claude-opus-4-8 and Haiku 4.5, 2,048 on Fable 5 and Sonnet 4.6, 1,024 on older Sonnets), so model upgrades can silently un-cache prompts that used to hit.
• Pick TTL by call gap: under 5 minutes, 5m wins and refresh-on-use keeps it alive; between 5 and 60 minutes, only 1h pays (Claude Code's own choice); past an hour, caching is a net loss.
• Know the production rules: tiered invalidation (tools worst, tool_choice mildest), the 20-block lookback, cold parallel fan-out, and max_tokens: 0 pre-warming.
• Audit with the usage fields, not with intuition: identical requests that go write-then-write have a silent invalidator from the table above.

👉 Receipts, then machinery. What remains is scale: every one of these usage blocks is already sitting on your disk, one per turn, for every session you have ever run. The next chapter turns the terminal itself into the instrument and builds the full ledger.