Budgeting a context window

What it is

A context budget is an explicit, written allocation of a model's input window across the competing things that want to live in it: system instructions, tool schemas, retrieved documents, conversation history, and the reserve you must leave for the model's own output. It is a capacity plan, and it is the same kind of artifact as a latency budget for a request path.

The thing people get wrong is treating the window as free space to fill. Two facts make that wrong. First, input and output share the window: the token cap on a response is subtracted from what you can put in, and on models where reasoning tokens count against that cap, thinking competes with the answer. Second, more context is not monotonically better: accuracy degrades for material buried in the middle of a long window, and past some point additional retrieved context adds distractors faster than it adds evidence.

This is commonly confused with "context length", which is a model capability, and with "prompt engineering", which is about how you phrase the instruction. Budgeting is about what occupies the window and in what order.

The problem it solves

Without an explicit budget, three failures are routine.

Silent truncation. History grows, retrieval returns more chunks than usual, and one day a request exceeds the window or the response gets cut off mid-answer. The symptom is intermittent and correlated with conversation length, which makes it hard to reproduce.

Cost drift. Every turn resends the whole conversation. A chat that starts at 2,000 input tokens per turn and grows to 80,000 has quietly become forty times more expensive per turn, and nobody notices until the bill arrives.

Quality decay. The agent that worked beautifully for five steps starts making mistakes at step twenty, because the instruction it needs is now in the middle of a 90,000-token window competing with fifteen stale tool outputs.

A budget makes all three visible before they happen.

Mechanics

The five allocations

Write these down as numbers, not as intentions:

Slot	What lives here	Volatility
System instructions	Role, rules, output format, safety constraints	Frozen
Tool schemas	Names, descriptions, JSON schemas for every tool	Frozen per version
Retrieved context	RAG chunks, file contents, fetched pages	Per turn
Conversation history	Prior user and assistant turns, tool calls and results	Grows
Output reserve	The maximum tokens the response may consume	Fixed by config

The volatility column is not decoration. It determines ordering, and ordering determines cost, because prompt caching is a prefix match: the cache key is derived from the exact bytes up to a marked breakpoint, and any change anywhere in the prefix invalidates everything after it.

The render order for a request is tools, then system, then messages. So the stable-to-volatile ordering you want is the order the API already uses, and your job is to avoid fighting it: keep tool definitions and the system prompt byte identical across requests, and put everything that changes per turn into the message history at the end.

The silent cache invalidators

These are the ones I check for in review, and they are the reason a team's cache hit rate is zero despite having set the flag:

A timestamp or "today's date" interpolated into the system prompt. It sits at the front of the prefix, so every request is unique.
A request ID or UUID anywhere early in the content.
Serialising a dict without sorting keys, or iterating a set. The bytes differ run to run even when the content does not.
Building the tool list per user, or reordering it. Tools render at position zero, so a varying tool set means nothing caches for anyone.
Conditional system-prompt sections. Every flag combination is a distinct prefix.

The verification is a single field: if the response's cache-read token count is zero across repeated requests with what should be an identical prefix, one of the above is happening. Diff the rendered prompt bytes between two requests to find it.

The economics, as ratios

Vendor prices change; the ratios are the durable part and they are what an interviewer is testing.

A cache read costs roughly 0.1x the normal input price.
A cache write costs 1.25x for a short (five minute) time to live, or 2x for a one hour TTL.

So the break-even is arithmetic: with the short TTL, two requests already pay for themselves ($1.25 + 0.1 = 1.35$ versus $2$ uncached). With the long TTL you need three ($2 + 0.2 = 2.2$ versus $3$). The long TTL is for bursty traffic with gaps longer than the short window; if requests arrive continuously, the short TTL is strictly cheaper.

Two further mechanics worth knowing because they produce confusing symptoms:

Minimum cacheable prefix. Below a model-dependent threshold (on the order of 512 to 4,096 tokens depending on the model), a marked prefix silently does not cache. No error, just a zero in the cache-creation counter. A 3,000-token system prompt caches on some models and not others.

Concurrency. A cache entry becomes readable only once the first response begins streaming. Fire N identical requests in parallel and all N pay full price, because none can read what the others are still writing. The fix for a fan-out is to send one request, wait for its first token, then fire the rest.

Ordering within the window

Two published effects drive the ordering rule.

Lost in the middle. Liu et al. measured a U-shaped accuracy curve against the position of relevant information in a long context: models recall material at the beginning and end better than material in the middle. So the highest-ranked evidence goes first, and the actual instruction goes last, immediately before the model generates.

Context rot. Beyond some point, adding context reduces accuracy. A retriever tuned to return 50 chunks because the window allows it is adding 45 distractors to help with 5 pieces of evidence.

A worked example: an agent on a large window

A code-assistant agent. The window is 200,000 tokens for this deployment (the number is what matters, not the model). Output reserve is 8,000 tokens because answers include diffs.

Window                                        200,000
  - output reserve (max_tokens)                 8,000
  ─────────────────────────────────────────────────────
  Usable input                                192,000

Fixed allocation (cacheable prefix, byte-stable):
  System instructions                           1,800
  Tool schemas (11 tools)                       4,400
  Project conventions file                      3,100
  ─────────────────────────────────────────────────────
  Cacheable prefix subtotal                     9,300   <- cache breakpoint here

Per-turn allocation:
  Retrieved file contents (cap 6 files x 2k)   12,000
  Conversation history                        ...grows
  Current user message                            300

Headroom policy: compact when history exceeds 120,000, leaving
  192,000 - 9,300 - 12,000 - 300 = 170,400 for history.
Compaction trigger at 120,000 gives ~50,000 tokens of slack for a
turn that retrieves unusually large files.

Now the cost per turn. Suppose input is priced at $P$ per token.

Without caching, turn 30 with 100,000 tokens of history costs $(9{,}300 + 12{,}000 + 100{,}000) \times P = 121{,}300P$.

With a breakpoint after the fixed prefix, the 9,300-token prefix reads at $0.1P$, so it contributes $930P$ instead of $9{,}300P$. That is a 7 percent saving, which sounds unimpressive until you put the breakpoint at the end of the history instead: each turn then reads the entire prior conversation from cache and pays full price only for the new turn. Turn 30 becomes roughly $(121{,}300 \times 0.1) + \text{new content} \approx 12{,}400P$, an order of magnitude less.

That is the single most valuable thing to say about caching in an interview: for multi-turn conversations, the breakpoint goes at the end of the most recently appended turn, not at the end of the system prompt. The system prompt is a rounding error next to the history.

One implementation detail that bites: a breakpoint walks backward a bounded number of content blocks (on the order of 20) looking for a prior cache entry. An agentic turn that emits 30 tool-call and tool-result blocks blows past that window, so the next turn's breakpoint finds nothing and silently misses. On tool-heavy turns, place an intermediate breakpoint every dozen or so blocks.

Compaction, when the budget is exceeded

Three strategies, in increasing order of fidelity loss and decreasing order of cost:

Rolling summarisation replaces the oldest N turns with a generated summary. Cheap, lossy, and it invalidates the cache prefix at the point of replacement, which is the hidden cost people miss: compacting is not free, it forces a full re-read of everything after the edit.

Structured state extraction keeps a JSON object of the facts that matter (files touched, decisions made, open questions) and drops the raw history entirely. Much more compact than a prose summary and far more robust, because the next turn reads a schema rather than parsing English. This is the one I would default to for agents.

Tool output truncation with a fetch-more affordance. A tool that returns 40,000 tokens of log gets truncated to the first 2,000 with a handle the model can use to request more. This is the highest-leverage single change in most agent deployments, because tool output is usually the largest and least valuable occupant of the window.

Sub-agent isolation is the fourth option and it is structural rather than compressive: give a sub-agent only the slice of context it needs, let it do the work, and return a short result to the parent. The parent's window never sees the sub-agent's tool outputs at all. It cuts both cost and pollution, at the price of the sub-agent occasionally lacking context the parent had.

Production evidence

Anthropic's prompt caching documentation specifies the mechanics used above: prefix matching with a bounded number of breakpoints, the tools-then-system-then-messages render order, model-dependent minimum cacheable prefix lengths, cache reads at roughly a tenth of input price and writes at 1.25x (five minute TTL) or 2x (one hour TTL), and the response fields that report cache creation and cache read tokens so you can verify hits rather than assume them.

Liu et al., "Lost in the Middle" (TACL 2024) is the measurement behind the ordering rule, across several models and both open and closed systems.

MemGPT / Letta (Packer et al., 2023) framed the whole problem as virtual memory: a small in-context working set plus an external store, with paging between them. Whether or not you adopt the architecture, the framing is the right one and it is a good thing to name.

Coding agents in general demonstrate the compaction pattern in production: long-running sessions that summarise or clear earlier turns as the window fills, rather than failing at the limit. Server-side compaction and context-editing features now exist in provider APIs precisely because every agent builder was implementing the same thing.

The debate

The alternative to budgeting is buying a bigger window. Frontier models now offer a million tokens, which makes "just put everything in" genuinely viable for a lot of applications, and it is simpler than any retrieval or compaction system.

The case for it: no retriever to tune, no chunking strategy, no recall metric, no staleness. For a corpus that fits, this is straightforwardly better engineering.

The case against, in order of how often it bites: cost, because you pay for every token on every turn and only caching mitigates it; the position effect, which does not disappear at a million tokens; and access control, which a long context handles by not handling it, since you cannot put one user's documents in a shared prefix.

My position: budget explicitly regardless of window size, because the budget is what turns "we ran out of context" from an incident into a policy. Use a large window to remove the retriever where the corpus is small, stable and not access-controlled. Use retrieval plus a budget everywhere else. And put the cache breakpoint at the end of the history, not the system prompt, because that is where the tokens are.

Budgeting is the wrong focus when the real problem is a single oversized tool output or an unbounded retriever. Fix those first: capping tool output and retrieving five chunks instead of fifty often makes the budget question disappear.

Follow-up Q&A

"How do you budget a 128k context window for an agent?" Subtract the output reserve first, since input and output share the window. Then allocate a fixed, byte-stable prefix for system instructions and tool schemas, a per-turn cap for retrieved context, and give history the remainder with a compaction trigger well below the limit so an unusually large turn does not blow through it. Order it stable-to-volatile so the prefix caches, put the highest-value evidence at the start and the instruction at the end, and cap tool output with a fetch-more affordance. Write the numbers down; a budget nobody can quote is not a budget.

"Your prompt is 3,000 tokens of business rules and each new edge case adds a paragraph. What do you do?" That is knowledge encoded in the wrong place. The prompt should carry the task and the format; the rules belong in a retrievable store keyed by the situation, so the model sees the three rules relevant to this request rather than all ninety. It also fixes the maintenance problem: rules in a prompt cannot be tested individually, versioned by owner, or audited. The migration is incremental: move the largest, most conditional block out first, retrieve it by case, and measure on a golden set before moving the next.

"Your cache hit rate is zero. Diagnose." Check the cache-read token count in the response first to confirm it really is zero rather than assumed. Then diff the rendered prompt bytes between two consecutive requests and look for the usual invalidators: a timestamp or UUID in the system prompt, unsorted JSON serialisation, a per-user tool list, or a conditional system section. If the bytes are identical, check that the prefix exceeds the model's minimum cacheable length, and check whether the requests are concurrent, since parallel requests cannot read a cache still being written.

"When would you compact rather than retrieve?" Compaction is for history, retrieval is for knowledge. History is a linear record you own and can summarise; knowledge is a corpus you query. If the thing filling your window is prior turns, compact. If it is documents, fix the retriever. Teams that reach for summarisation to solve a retrieval problem end up with a lossy summary of the wrong documents.

"What breaks when you summarise the conversation?" Three things. The cache prefix is invalidated at the point of the edit, so everything after it re-reads at full price. Details the summary dropped are gone permanently, and you cannot know in advance which ones the next turn needs. And the summary itself is model output, so it can be wrong in ways that are hard to detect, which is the argument for structured state extraction over prose summarisation: a JSON object with typed fields fails loudly, a paragraph fails quietly.

Common misconceptions

The most common is that a large context window makes budgeting unnecessary. The window sets the ceiling; the budget is about cost, ordering and quality, all of which still apply at a million tokens.

The second is that caching is a flag you turn on. It is a prefix property: one interpolated timestamp in the system prompt disables it entirely, silently, and the only way to know is to check the cache-read counter.

The third is that the system prompt is where caching pays off. In a multi-turn conversation the history dwarfs the system prompt within a few turns, and the breakpoint belongs at the end of the most recent turn.

Interview delivery note

Say this: "I write the budget down as five numbers: system, tools, retrieved context, history, and the output reserve, which comes off the top because input and output share the window. Then I order it stable to volatile so the prefix caches, because caching is a prefix match and one timestamp in the system prompt disables it. For a multi-turn conversation the cache breakpoint goes at the end of the latest turn, not the system prompt, since that is where the tokens are. And I cap tool output with a fetch-more handle, because tool output is usually the largest and least useful thing in the window."

The depth signal is the break-even arithmetic on cache writes versus reads, and naming the position effect as a reason to order rather than just to trim. Anyone can say "manage your context". Saying "a write costs 1.25x and a read costs 0.1x, so two requests break even" shows you have costed it.

The Staff & Team Lead Interview Reference