Prompt caching: bending the cost curve

The concepts chapter planted the seed: the model is stateless, so every turn re-sends and re-reads the whole conversation, and prompt caching exists because of it. This chapter grows the seed into working knowledge: what the provider actually caches, the exact rules that make caching work or silently fail, the price arithmetic, and a lab that re-bills one realistic agent session four ways. For an agent platform, this is not an optimization chapter; caching routinely decides whether a fleet design is affordable at all.

What is actually being cached

"The provider keeps the prefix warm" was the concepts chapter's hand-wave; here is the decoded version. When a model reads your prompt, it builds internal reading state, token by token: for each token, a set of numeric summaries that later tokens consult when attending to earlier ones. (In transformer terms this is the key-value state; in plain terms, the model's working notes on what it has read so far.) That state is expensive to build, it is most of what you pay input price for, and, crucially, it depends only on the tokens read so far, in order.

That last property is the whole trick. If today's request starts with the exact same bytes as a recent request, the notes for that shared prefix are identical by construction, so the provider can reload them instead of recomputing, charge you a fraction, and start real work at the first new byte. It also explains the strictness: the notes for token 5,000 depend on tokens 1 through 4,999, so any earlier difference, one byte, one reordered field, invalidates everything after it. Caching is prefix-exact because attention is order-dependent, not because providers are fussy.

The rules

Five rules govern the mechanism, common in shape across the first-party API and Bedrock:

1. You mark cache points. A marker in the request (Bedrock's Converse calls it a cachePoint block; the Anthropic API calls it cache_control) says "cache everything up to here." Up to four markers per request; everything before the marker must be byte-stable to hit.
2. There is a minimum. Prefixes below a model-specific floor (about 4,096 tokens on recent Claude generations, lower on older ones) are not cached at all, silently. A marker on a short prefix simply does nothing, which is a debugging trap: no error, no cache.
3. Entries expire. The default lifetime (TTL, time to live) is 5 minutes, refreshed every time the entry is read. A 1-hour TTL is available at a higher write price. Let an entry lapse and the next request rebuilds it at full write cost.
4. The prices. Reads bill at 0.1x the input price. Writes bill at a premium over input price: 1.25x for the 5-minute TTL, 2x for the 1-hour TTL. So caching pays when reads outnumber writes, which for append-only agent loops is almost always.
5. Batch is excluded. The batch tier (next chapters) does not cache at all. Cheap and cached are two different discounts, and you pick one per request.

For agents, the rules compile down to two design habits. Freeze the prefix: the system prompt and tool definitions come first in every request, byte-identical every time; no timestamps, no per-request ids, no reordered tool lists, and note that switching models also starts the cache cold, because entries are model-scoped. Grow append-only: Part 1's loop already does this naturally, since each turn's request is the previous request plus new content at the end, which makes an agent loop very close to the ideal caching customer.

Don't be confused: caching vs memory. Prompt caching stores the processing of bytes you were going to send anyway; it changes what a request costs, never what the model sees. Memory (Part 7) stores content so it can enter future contexts at all. "The agent remembered because of the cache" is a category error: with a cold cache the agent answers identically, just at full price.

Lab 2.1: one session, four bills

The lab models a session shaped like real agent work: 40 turns, a 6,000-token frozen prefix (system prompt plus tools), a conversation growing by a few hundred tokens of tool results and text per turn, and, because real sessions are not metronomes, two 7-minute pauses mid-session while a slow tool runs. Then it computes the same session's bill under four schemes. Everything is deterministic; prices are the book's Opus-class snapshot ($5 in, $25 out per million).

python3 cache_ledger.py

session: 40 turns, prefix 6,000 tok, final request 39,470 tok
input tokens processed : 909,140
output tokens generated: 11,940

scheme                    input $  output $  total $  vs none
no caching                   4.55      0.30     4.84     1.0x
cache, 5-minute ttl          0.96      0.30     1.26     3.8x
cache, 1-hour ttl            0.83      0.30     1.13     4.3x
batch tier (offline)         2.27      0.15     2.42     2.0x

5m ttl lapsed after the pauses (turns [16, 31]): re-writing the prefix cost $0.28 extra
1h ttl never lapsed (lapses: []); its 2x first write was cheaper than re-writing twice
caching discounts the input column only; output is never discounted

Four readings, in rising order of subtlety:

• The headline: 3.8x to 4.3x. Nearly a million input tokens processed, the overwhelming majority of them re-reads of an append-only transcript, exactly the shape rule 4 rewards. This ratio, not the sticker price per million, is the number that decides fleet economics.
• The lapse tax is visible and attributable. The 5-minute scheme lost its cache twice, at the two 7-minute pauses, and re-paying the write premium on an by-then-large prefix cost $0.28, about 30% extra on its input bill. Any agent that waits (on slow tools, on humans, on queues) pays this tax at 5-minute TTL.
• The 1-hour TTL won because of the pauses. Its 2x first write is a handicap in a smooth session; two lapses flipped the comparison. The general rule falls out cleanly: pause-prone sessions want the long TTL, metronomic ones want the short one, and the crossover is computable from your own gap distribution, which your event ledger already records.
• Batch is a different shape, not a smaller bill. Half price on everything, including output, but hours of turnaround and no cache: useful when the work itself is offline (Capstone A's overnight audits route their judge passes there), useless for an interactive loop. Note it beat nothing-at-all and lost to either cache: for conversation-shaped work, caching is simply the stronger discount.

And the closing line generalizes beyond this lab: caching touches only the input column. This session was read-heavy (909k in, 12k out), so the win was 4x; the fleet in the lab catalog wrote proportionally more output and saw only 1.8x from the same mechanism. Sessions that mostly read cache brilliantly; sessions that mostly write need different levers, which is exactly where the batch tier and the model portfolio (Chapter 15) come in.

Lab 2.3, part two: verifying on real Bedrock

On the real service, trust the meter, not the vibes. Every response reports cache behavior in its usage block, and the follow-along verification is three calls (T1, cents; output illustrative):

from anthropic import AnthropicBedrockMantle

client = AnthropicBedrockMantle(aws_region="us-east-1")
big_prefix = open("system_prompt.txt").read()      # > 4,096 tokens of it

def ask(question: str):
    return client.messages.create(
        model="anthropic.claude-opus-4-8",
        max_tokens=300,
        system=[{"type": "text", "text": big_prefix,
                 "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": question}],
    ).usage

print(ask("What is our refund policy?"))       # first call: writes
print(ask("And the shipping cutoff?"))         # second call: reads

(illustrative)
Usage(input_tokens=21, cache_creation_input_tokens=5804, cache_read_input_tokens=0, ...)
Usage(input_tokens=19, cache_creation_input_tokens=0, cache_read_input_tokens=5804, ...)

First call: the prefix was written (billed at the premium). Second call: the same 5,804 tokens were read at 0.1x, and only the new question paid full price. If the second call shows cache_read of zero, something in the prefix changed between calls, and hunting the changed byte (a timestamp is the usual suspect) is the actual skill this lab teaches. In production this check runs continuously: the platform's token ledger (Part 9) records these fields per call, and a falling cache-read ratio pages someone, because it means the fleet silently started paying full price.

Full source

"""Lab 2.1: the cache ledger. One agent session, priced four ways.

Models a realistic 40-turn agent session (stable prefix, conversation
that grows every turn, two long mid-session pauses while a slow tool
runs) and computes what the same session costs under four pricing
schemes: no caching, prompt caching with the 5-minute TTL, prompt
caching with the 1-hour TTL, and the offline batch tier.

Prices are the book's worked-example snapshot for a Claude Opus 4.8
class model; check the live price pages before planning real spend.
Standard library only. Deterministic: same run, same numbers.
"""

from __future__ import annotations

PRICE_IN = 5.00        # $ per million input tokens
PRICE_OUT = 25.00      # $ per million output tokens
READ_MULT = 0.10       # cache reads bill at 0.1x the input price
WRITE_5M = 1.25        # cache-write premium, 5-minute TTL
WRITE_1H = 2.00        # cache-write premium, 1-hour TTL
BATCH_MULT = 0.50      # batch tier: half price, no caching, hours not ms

PREFIX = 6_000         # system prompt + tool definitions, byte-stable
TURNS = 40


def build_session() -> list[dict]:
    """40 turns; each request = prefix + everything said so far."""
    turns, history = [], 200            # the task itself is ~200 tokens
    for t in range(1, TURNS + 1):
        turns.append({
            "t": t,
            # seconds since the previous request; two 7-minute pauses
            # while a slow tool runs (turns 16 and 31 resume after them)
            "gap": 420 if t in (16, 31) else 45,
            "request": PREFIX + history,
            "output": 250 + (t * 17) % 100,
        })
        history += turns[-1]["output"] + 400 + (t * 37) % 300
    return turns


def cost_uncached(turns: list[dict]) -> tuple[float, float]:
    cin = sum(x["request"] for x in turns) / 1e6 * PRICE_IN
    cout = sum(x["output"] for x in turns) / 1e6 * PRICE_OUT
    return cin, cout


def cost_cached(turns: list[dict], ttl: int,
                write_mult: float) -> tuple[float, list[int]]:
    """Input cost with a cache breakpoint at the end of each request.

    The conversation is append-only, so each request extends the last:
    whatever is still cached is read at 0.1x, the new tail is written at
    the TTL's premium. A gap longer than the TTL empties the cache.
    """
    cached, cin, lapses = 0, 0.0, []
    for x in turns:
        if x["gap"] > ttl and cached:
            cached = 0
            lapses.append(x["t"])
        fresh = x["request"] - cached
        cin += (cached / 1e6 * PRICE_IN * READ_MULT
                + fresh / 1e6 * PRICE_IN * write_mult)
        cached = x["request"]
    return cin, lapses


if __name__ == "__main__":
    turns = build_session()
    tokens_in = sum(x["request"] for x in turns)
    tokens_out = sum(x["output"] for x in turns)
    print(f"session: {TURNS} turns, prefix {PREFIX:,} tok, "
          f"final request {turns[-1]['request']:,} tok")
    print(f"input tokens processed : {tokens_in:,}")
    print(f"output tokens generated: {tokens_out:,}\n")

    cin0, cout0 = cost_uncached(turns)
    cin5, lapses5 = cost_cached(turns, ttl=300, write_mult=WRITE_5M)
    cin1, lapses1 = cost_cached(turns, ttl=3600, write_mult=WRITE_1H)
    cinb, coutb = cin0 * BATCH_MULT, cout0 * BATCH_MULT

    rows = [
        ("no caching",             cin0, cout0),
        ("cache, 5-minute ttl",    cin5, cout0),
        ("cache, 1-hour ttl",      cin1, cout0),
        ("batch tier (offline)",   cinb, coutb),
    ]
    print(f"{'scheme':<24}{'input $':>9}{'output $':>10}"
          f"{'total $':>9}{'vs none':>9}")
    total0 = cin0 + cout0
    for name, cin, cout in rows:
        total = cin + cout
        print(f"{name:<24}{cin:>9.2f}{cout:>10.2f}"
              f"{total:>9.2f}{total0 / total:>8.1f}x")

    # what the two 7-minute pauses cost the 5-minute scheme
    cin5_ideal, _ = cost_cached(
        [{**x, "gap": 45} for x in turns], ttl=300, write_mult=WRITE_5M)
    print(f"\n5m ttl lapsed after the pauses (turns {lapses5}): "
          f"re-writing the prefix cost ${cin5 - cin5_ideal:.2f} extra")
    print(f"1h ttl never lapsed (lapses: {lapses1}); its 2x first write "
          f"was cheaper than re-writing twice")
    print("caching discounts the input column only; output is never "
          "discounted")

👉 Next: steering and structure, the controls that shape what models say and do, arranged by where they live and what each one can actually stop.