Context evals: does the model actually use what you send?

TL;DR. Every chapter so far optimized what goes into the window; this one asks the question underneath: once a fact is in the window, does the model reliably use it? The answer is "mostly, and less than the spec sheet implies": retrieval quality inside a context degrades with length, position (the lost-in-the-middle curve), and distractor density, a family of effects the field calls context rot. This chapter covers the standard evals (needle-in-a-haystack, RULER, multi-needle and QA variants), what they consistently find, and, more usefully, how to run a small honest version against your own setup with Claude Code as the harness: generate a haystack with planted facts at controlled depths, ask through claude -p, and score. The engineering consequences all point the same way as the cost chapters, which is the good news: shorter, denser, better-selected context is a quality optimization too.

Contents

• Context rot: the effect
• The standard evals
• What the evals consistently find
• Rolling your own with Claude Code
• Engineering consequences
• Further reading
• Takeaways

Chapter 14 explained how million-token windows became affordable. Affordable is not the same as uniform: a context window is not RAM, where address 900,000 reads as reliably as address 10. Attention is a learned, soft mechanism, and how well a fact is used depends on where it sits, what surrounds it, and how much else is competing for the same attention mass. Ignoring that gap is how teams ship systems that pass small tests and quietly degrade at scale, and measuring it is cheap enough that there is no excuse not to.

Context rot: the effect

Three degradations, separable and compounding:

1. Length. Task accuracy on the same question with the same supporting fact declines as the total context grows, even when everything added is irrelevant padding. The model does not fall off a cliff at the window limit; it slides gradually long before it.
2. Position. Accuracy depends on where the fact sits: strong at the beginning, strong at the end, weakest in the middle, the U-shaped curve documented by Liu et al.'s "Lost in the Middle" and replicated broadly since. Chapter 28 showed Claude Code exploiting the good ends of this curve deliberately (rules at the front, reminders at the tail); this chapter is the curve itself.
3. Distraction. Accuracy falls faster when the padding is plausible: near-duplicate passages, entities of the same type, content from the same domain. Ten distractors that look like the answer hurt more than a hundred pages of unrelated prose, which is exactly the failure mode over-retrieval (Chapter 31) manufactures: the chunks past the knee are, by construction, the most plausible-looking irrelevant text available.

Newer long-context models push all three curves outward, and none flattens them. Treat any specific accuracy-at-depth number as perishable; treat the shape as durable.

The standard evals

The single most useful concept out of this literature is effective context length: the input size at which your task's accuracy drops below your bar, on your model. It is always measured, never quoted, and it is the number that should size compaction thresholds (Chapter 11) and retrieval budgets (Chapter 31).

What the evals consistently find

• Verbatim single-needle recall is the easiest case and modern frontier models do it well over huge spans; headlines built on it flatter every real workload.
• Anything harder (paraphrased needles, multi-fact aggregation, reasoning over the retrieved span) degrades earlier and faster, so benchmark the task shape you actually run.
• The U-curve holds within-context: middle placement costs accuracy at every length, and the penalty grows with length.
• Distractor similarity is a stronger predictor of failure than raw length, which converts directly into engineering advice: curating out plausible-irrelevant content buys more quality than trimming bulk.
• Structure helps: clearly delimited, labeled sections (headers, XML-style tags around documents) reliably outperform undifferentiated walls of text at equal length, because they give attention anchors to index on.

Rolling your own with Claude Code

A NIAH grid for your own setup is an afternoon, and Claude Code is the harness (Chapter 29's pattern, pointed at quality instead of cost). The protocol:

1. Generate haystacks. Concatenate representative filler from your own domain (your docs, your code, your tickets) to target sizes, say 10k, 50k, 150k, 400k tokens (count with count_tokens, Chapter 30 layer 0). Domain filler matters: it supplies realistic distractors, which random essays do not.

2. Plant needles at controlled depths. A fact that cannot be guessed ("the deploy freeze code for Q3 is MAGENTA-41"), inserted at 10, 25, 50, 75, and 90 percent depth. One variant per (size, depth) cell.

3. Ask through print mode. For each cell:

   claude -p "Read haystack_50k_d50.md in full, then answer: what is the Q3 deploy
   freeze code? Reply with the code only." \
     --output-format json --allowed-tools "Read"
   

4. Score and plot. The result JSON gives the answer text (grade it mechanically: the code is either present or not), total_cost_usd, and duration_ms, so one loop yields the accuracy heatmap and the cost of quality at each length. session_audit.py on the transcripts confirms each run actually read the whole file rather than a truncated slice, the classic silent invalidator of homemade NIAH results (a Read that returned 2,000 lines of a 10,000-line haystack tests nothing).

5. Escalate to your real shape. Once the single-needle grid exists, swap in paraphrased needles, then multi-needle questions, then your actual task with planted facts. Each step costs the same loop and answers a harder question. For a maintained harness instead of a shell loop, promptfoo (Chapter 26) runs the same grid as a config file with graders, and keeps it green in CI.

The output that matters is one number per task shape: the length where accuracy leaves your acceptable band. That is your effective context length; write it down next to your compaction threshold, because it is your compaction threshold.

Engineering consequences

Everything this book charged you money for turns out to also be a quality lever, and this chapter is the receipts:

• Selection budgets protect accuracy, not just cost. Past the retrieval knee, added chunks are high-similarity distractors, the most damaging kind. The knee from Chapter 31 is a quality optimum too.
• Compaction before rot, not before overflow. Bound the window by your measured effective length, not the advertised maximum; Chapter 11's levers exist for quality as much as for cost. The /usage warning about sessions over 150k context (Chapter 21) is this effect wearing a billing hat.
• Place deliberately. Instructions and identity at the front, live state re-asserted at the tail, bulk reference in between, delimited and labeled. That is Chapter 28's architecture, justified twice over.
• Structure the bulk. Tag documents, keep headers, prefer many labeled sections to one undifferentiated paste; the packers' XML framing (Chapter 27) is not decoration.
• Distrust demos at 10k. A pipeline validated on short contexts has been validated on the easiest region of the curve. Re-run the eval grid at the lengths production will see.

Remember. The window's size is a capacity spec; its usable size is a quality measurement that depends on your task, and it is always smaller. You already own every tool needed to measure it: a filler corpus, a planted fact, claude -p, and an afternoon.

Further reading

• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts": the positional U-curve.
• Hsieh et al., "RULER: What's the Real Context Size of Your Long-Context Language Models?": effective context length and the generated suite.
• Chroma Research, "Context Rot" (research.trychroma.com): the distractor and similarity axes, with protocols worth copying.
• Greg Kamradt's needle-in-a-haystack repository, the original heatmap harness; and promptfoo's docs for running position/length grids in CI.

Takeaways

• Context use degrades with length, middle placement, and distractor similarity (context rot); newer models shift the curves without flattening them. The shape is durable; specific numbers are perishable, so measure your own.
• Effective context length, the size where your task leaves your quality band, is the number to size compaction and retrieval budgets by, and it is always below the advertised window.
• The eval ladder: single-needle (easy, flattering), paraphrased, multi-needle, then your real task with planted facts. Claude Code's -p JSON output makes the whole grid a shell loop, with session_audit.py guarding against truncated reads.
• Distractor similarity beats raw length as a failure predictor: curating out plausible-irrelevant content is the highest-yield quality move, and it is the same move the cost chapters already told you to make.
• Structure (delimited, labeled sections) measurably helps at equal length; placement follows the U-curve the harness already exploits.

👉 The model uses what you send, imperfectly and measurably. The last question about the window is darker: what happens when some of what you send is trying to use you. Continue to Hostile context.