Lost in the middle, and context rot

What it is

Two distinct degradations that both make long contexts perform worse than their size suggests, and that get conflated because they produce similar complaints.

Lost in the middle is positional. Within a single request, information placed in the middle of the context is used less reliably than the same information at the start or the end. It is a property of one inference call and it is measurable with a controlled experiment: put the same fact at different depths and measure retrieval accuracy.

Accuracy by position of the relevant fact in a 32k window:

 90% ┤ ●                                                    ●
     │  ●                                                  ●
 80% ┤   ●                                                ●
     │     ●                                            ●
 70% ┤       ●                                        ●
     │          ●                                  ●
 60% ┤              ●     ●     ●     ●     ●   ●
     └──┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┬──
        0    10%   20%   30%   40%   50%   60%   70%   90%
                     position in context

Context rot is temporal. Across a long conversation, quality degrades as the accumulated context grows and its signal-to-noise ratio falls. It is a property of a session rather than a request, and it comes from a different mechanism: the context fills with resolved questions, superseded answers, failed tool calls and stale retrievals, so the useful fraction shrinks even as the token count grows.

What they are confused with: neither is the model "forgetting." In lost-in-the-middle the information is present and under-attended. In context rot the information may be present, contradicted by something else in the window, or crowded out by noise. Calling either "forgetting" leads to the wrong fix, which is usually "use a bigger window," and that makes both worse.

The problem it solves

Understanding these is what stops two expensive mistakes.

Mistake one: buying context length instead of retrieval quality. A team whose RAG system misses answers concludes the window is too small, moves from 8k to 128k, retrieves 50 documents instead of 5, and gets worse results at 16 times the cost. The relevant document is now in the middle of a large context alongside 49 distractors.

Mistake two: letting a conversation accumulate. An agent that appends every turn, every tool call and every tool result to its context has a session whose quality declines monotonically. The 40th turn is answered from a context that is mostly the debris of the first 39.

The measurable shapes:

Lost in the middle, 10 documents, one relevant:
  relevant doc at position 1:   accuracy 87%
  relevant doc at position 5:   accuracy 61%
  relevant doc at position 10:  accuracy 82%

Same task, 20 documents, one relevant:
  relevant doc at position 1:   accuracy 85%
  relevant doc at position 10:  accuracy 54%
  relevant doc at position 20:  accuracy 79%

Adding distractors deepens the trough. More retrieved documents is not neutral; it actively harms the case where the answer was already retrieved.

Mechanics

Why the U-shape exists

Three contributing mechanisms, and the honest answer is that no single one fully explains it:

Attention dilution. Softmax over S positions means the average attention weight is 1/S. As S grows, a token must earn a much higher score to stand out, and the model's learned attention patterns were shaped by the length distribution it trained on.

Positional encoding effects. With RoPE (see RoPE and ALiBi), the rotation-based decay means distant positions have systematically lower attention scores unless learned otherwise. Positions near the query benefit from recency; positions at the very start benefit from having been attended to by every subsequent token during training.

Training distribution. Instructions appear at the start of training examples and the answer follows the end of the input, so those regions are where the model learned to look. The middle of a long document is the region least likely to have been load-bearing during training.

The practical upshot is that the effect is robust across models, architectures and context lengths, which means it should be designed around rather than expected to disappear with the next model.

Context rot: the mechanism

Rot is about signal-to-noise, and the sources of noise are enumerable:

Turn 1:  user question + retrieved docs + answer            useful
Turn 5:  ... + 4 more Q/A pairs, 2 of them resolved         some noise
Turn 12: ... + a failed tool call and its error             noise
         ... + a retrieval that returned nothing relevant   noise
         ... + a correction ("no, I meant the OTHER order") CONTRADICTION
Turn 25: ... + 24 turns of accumulated everything           mostly noise

Contradictions are the worst category, because they are not merely uninformative. When the context contains "the order is #4471" from turn 3 and "sorry, it is #4472" from turn 8, the model must resolve which is current, and it resolves by attention rather than by recency logic. It will sometimes pick the stale one.

Measured quality by turn, no compaction, on a support agent:

turn range   task success   mean context tokens   useful fraction (audited)
  1-5           94%              2,100                 91%
  6-15          89%              7,400                 62%
 16-30          76%             16,800                 38%
 31-50          58%             31,200                 21%

Success falls from 94 to 58 percent while the context grows 15x. The useful fraction column is the mechanism: by turn 40 roughly four fifths of the context is content that no longer bears on the current question.

The mitigations, ranked

1. Retrieve less, rank better. The highest-value change, and it is counterintuitive.

20 documents retrieved, 1 relevant, at position 10:  54% accuracy
 5 documents retrieved, 1 relevant, at position 3:   79% accuracy

Fewer, better-ranked documents beat more documents, because every additional distractor deepens the trough. This is the argument for investing in a reranker rather than in context length: see cross-encoder and LLM reranking.

2. Order by relevance, with the best nearest the query.

# The ranked list is [best, ..., worst]. Reverse it so the best is
# adjacent to the query, exploiting the recency end of the U.
context = system + "\n\n" + "\n\n".join(reversed(ranked_docs)) + "\n\n" + query

Free, and it moves the top-ranked document from the weakest region to the strongest.

3. Compact, do not accumulate. For conversations, replace old turns with structured state rather than appending indefinitely. See compaction.

4. Explicitly mark stale content, if you must keep it:

[SUPERSEDED, turn 3] Order #4471
[CURRENT, turn 8]    Order #4472

Cheap, and it converts an ambiguous contradiction into an instruction. It is a partial mitigation and better than leaving both unmarked.

5. Reset when the topic changes. A conversation that has moved to a new subject should not carry the previous subject's retrievals and tool results. Detecting a topic change is imperfect and even a crude heuristic beats never resetting.

Measuring both, in your own system

The needle-in-a-haystack test measures lost-in-the-middle. Run it on your task, not on a synthetic one, because the published curves are for synthetic fact retrieval and your task's curve will differ:

def positional_sweep(model, task_examples, filler_corpus, depths=(0.0,0.1,...,1.0)):
    results = {}
    for depth in depths:
        correct = 0
        for ex in task_examples:
            ctx = insert_at_depth(filler_corpus, ex.relevant_passage, depth)
            answer = model(ctx + ex.question)
            correct += grade(answer, ex.expected)
        results[depth] = correct / len(task_examples)
    return results          # plot it; the trough depth is what you design around

Context rot needs a different measurement: success rate by turn index, which requires either real sessions or a simulated multi-turn evaluation. It is more work and it is the one that catches the degradation people actually complain about.

A worked example: 128k made it worse

A legal-document assistant. Answered questions about contract sets, using RAG over about 40,000 clauses.

Starting point:

model context:          8k
documents retrieved:    5
accuracy (eval set):    81.2%
p50 latency:            1.9s
cost per query:         $0.008

Complaints centred on questions whose answer was in a clause the retriever ranked sixth or lower. The diagnosis was "we are not retrieving enough," which was correct, and the fix chosen was wrong.

Change 1: move to a 128k model, retrieve 60 documents.

accuracy:               81.2% -> 73.4%      <- WORSE
p50 latency:            1.9s -> 7.2s
cost per query:         $0.008 -> $0.094

Eight points worse at 12 times the cost. Retrieval recall genuinely improved (the relevant clause was now in the context 96 percent of the time, against 84 percent before), and the model's ability to use it fell further than recall rose.

A positional sweep on their own task explained it:

relevant clause at position (of 60):   accuracy
   1-5                                   88%
   6-15                                  71%
  16-45                                  49%     <- most of the range
  46-55                                  64%
  56-60                                  78%

Their retriever placed the relevant clause in positions 16 to 45 about 55 percent of the time, which is exactly the trough. Recall went up and usable recall went down.

Change 2: retrieve 60, rerank, keep 8.

candidates = retriever.search(query, k=60)          # keep the recall gain
ranked = cross_encoder.rank(query, candidates)      # a real reranker
top = ranked[:8]                                    # keep only what fits well
context = assemble(system, reversed(top), query)    # best nearest the query

accuracy:               73.4% -> 89.7%
relevant clause in top 8: 91%   (vs 96% in top 60: 5 points of recall traded)
p50 latency:            7.2s -> 2.4s
cost per query:         $0.094 -> $0.014

Trading 5 points of recall for 16 points of accuracy, plus a third of the latency and a seventh of the cost. The recall metric got worse and the system got much better, which is the same shape as the leakage story on the classical ML page: the metric being optimised was not the goal.

Change 3: the multi-turn problem, found separately. Users asked follow-up questions, and the agent appended each turn's retrieved clauses to the context.

success rate by turn:
  turn 1:    89.7%
  turn 5:    84.1%
  turn 10:   71.3%
  turn 20:   52.8%

By turn 20 the context held roughly 160 clauses from 20 retrievals, most irrelevant to the current question, and several contradicting each other because different contract versions had been retrieved at different turns.

# The fix: retrieved documents are per-TURN, not accumulated.
# Conversation state carries the thread; documents are re-retrieved each turn.
@dataclass
class SessionState:
    contract_ids_in_scope: list[str]      # narrowed as the conversation proceeds
    facts_established: list[str]          # "termination clause is section 14.2"
    open_questions: list[str]

def build_turn_context(state, question):
    docs = rerank(retrieve(question, scope=state.contract_ids_in_scope), k=8)
    return assemble(SYSTEM, state.as_text(), reversed(docs), question)
    #                        ^ ~400 tokens   ^ fresh, 8 docs, never accumulated

success rate by turn, after:
  turn 1:    89.7%
  turn 5:    89.1%
  turn 10:   88.4%
  turn 20:   86.9%

Nearly flat. The remaining decline is genuine question difficulty (later questions are more specialised), not context degradation.

Final:

                        original(8k)  128k+60docs   reranked   + no accumulation
accuracy (turn 1)          81.2%        73.4%        89.7%        89.7%
accuracy (turn 20)         n/a          52.8%        52.8%        86.9%
p50 latency                1.9s         7.2s         2.4s         2.5s
cost per query             $0.008       $0.094       $0.014       $0.015
context tokens (turn 20)   n/a          118,000      118,000      6,200

The 128k window was used for 6,200 tokens in the final design. They kept the model, because occasional single-shot whole-contract analysis needs the room, and the ordinary path uses a twentieth of it.

The transferable lesson: retrieval recall and answer accuracy are different metrics and they can move in opposite directions. Adding documents raises the first and can lower the second, because every distractor deepens the trough. The team optimised recall for a quarter before measuring the thing they actually cared about.

Production evidence

Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023) established the U-shaped curve across multiple models, context lengths and tasks, using both multi-document QA and key-value retrieval. It is the most practically consequential long-context result and the one to cite.

Anthropic's and Greg Kamradt's needle-in-a-haystack evaluations popularised the positional sweep as a standard test. Anthropic's own analysis notes that models perform better when the question is placed after the documents than before, which is the ordering finding stated from the other direction.

Chroma's "Context Rot" technical report (2024) measured degradation as input length grows across 18 models, including on tasks where the relevant information was unambiguous, finding that performance declines with length even when the task is trivially easy. That is the strongest evidence that the effect is about length itself rather than about task difficulty.

RULER (Hsieh et al., NVIDIA, 2024) measures effective context length as opposed to claimed length, using synthetic tasks of varying difficulty. Its consistent finding is that effective length is substantially below advertised length for most models, which is the benchmark to point at when someone quotes a context number.

Every serious RAG implementation reranks and truncates rather than passing everything retrieved, and the design of Cohere Rerank, cross-encoder rerankers and LlamaIndex's node postprocessors all assume that the retrieved set should be reduced before assembly.

The debate

Is this getting better with newer models? Somewhat, and not enough to design around. Newer long-context models show flatter curves on synthetic retrieval and still degrade on tasks requiring reasoning over information at several positions. RULER's effective-length measurements remain well below advertised lengths. My position: treat the U-shape as a persistent property, design for it, and re-measure on each model rather than assuming it is fixed.

Should you use a long context at all? Yes, for genuinely joint reasoning over a document that cannot be decomposed, where the alternative is retrieval that might miss the connection. No, as a substitute for retrieval quality. The distinguishing question is whether the task needs all of the content simultaneously or a subset of it: the first justifies the window, the second is a retrieval problem wearing a context-length costume.

How many documents should you retrieve? Retrieve generously, rank aggressively, and include few. The pattern that works is k=50 to 100 from the retriever, a cross-encoder over those, and 5 to 10 in the final context. Retrieval recall and inclusion count are different decisions and conflating them is the mistake in the worked example: you want high recall into the candidate pool and low count into the context.

Is "context rot" a real phenomenon or bad engineering? Mostly the latter, and the distinction is useful. A conversation that accumulates every turn's retrievals and tool results degrades because the system was built to accumulate, not because of a model property. The genuine model-side effect is the length-related degradation Chroma measured. Calling it rot is fair as a description of the symptom and misleading as a diagnosis: the fix is almost always compaction and per-turn retrieval rather than anything about the model.

Does ordering really matter enough to bother? It is free and it moves the top-ranked document from the weakest region of the curve to the strongest. The effect is a few points rather than tens of points, and there is no cost, so the question is why you would not. The caveat is prompt caching: if reordering changes the cacheable prefix per request, you lose the cache discount, so stable content should stay at the top and only the ranked block should reorder.

Follow-up Q&A

"What is lost in the middle?"

Within one request, information placed in the middle of the context is used less reliably than the same information at the start or the end, producing a U-shaped accuracy curve against position. It is robust across models and context lengths. The practical consequence is that adding retrieved documents is not neutral: every distractor deepens the trough, so a relevant document that moves from position 3 of 5 to position 10 of 20 can lose 25 points of accuracy while retrieval recall improved.

"How is context rot different?"

Lost in the middle is positional and within a request. Context rot is temporal and across a session: as a conversation accumulates turns, resolved questions, failed tool calls and superseded facts, the useful fraction of the context falls even as the token count rises. In one measured system the useful fraction went from 91 percent at turn 5 to 21 percent at turn 40, and task success fell from 94 to 58 percent. Contradictions are the worst component, because the model must resolve which of two stated facts is current and it does so by attention rather than by recency.

"Your RAG accuracy went down when you retrieved more documents. Explain."

Retrieval recall and answer accuracy are different metrics and they moved in opposite directions. Going from 5 to 60 documents put the relevant clause in the context 96 percent of the time instead of 84, and it also placed it in the middle of the context most of the time, where accuracy is lowest. The fix is to keep the recall gain and reduce the inclusion: retrieve 60, rerank, include 8. That traded 5 points of recall for 16 points of accuracy, plus a third of the latency and a seventh of the cost.

"How would you measure this in your own system?"

A positional sweep on your own task: insert the known-relevant passage at controlled depths in a filler context and measure accuracy at each depth. The published curves are for synthetic fact retrieval and your curve will differ, and the trough depth is what you design around. For rot, measure success rate by turn index across real or simulated multi-turn sessions, which is more work and catches the degradation users actually report.

"What mitigations, in order of value?"

Retrieve less and rank better, which is the largest effect and the counterintuitive one. Order by relevance with the best document nearest the query, which is free. Compact conversation history into structured state rather than accumulating turns. Mark superseded content explicitly if you must keep it, which converts an ambiguous contradiction into an instruction. And re-retrieve per turn rather than accumulating retrievals, which was the change that took turn-20 accuracy from 53 percent to 87 percent in one system.

"Do bigger windows fix it?"

No, and they can make it worse if you fill them. A larger window lets you include more distractors, and every distractor deepens the trough. RULER's measurements of effective versus advertised context length consistently show effective length well below the number in the model card. Large windows are useful for genuinely joint reasoning over content that cannot be decomposed; they are not a substitute for retrieval quality.

Common misconceptions

"The model forgets things in long contexts." The information is present and under-attended. Framing it as forgetting suggests a memory fix (a bigger window) when the correct fix is fewer, better-placed documents.

"More retrieved context is more information." Every additional document is also a distractor that deepens the positional trough. Recall into the candidate pool and count in the final context are different decisions.

"Lost in the middle and context rot are the same thing." One is positional within a request, the other temporal across a session, and they need different fixes: reordering and reranking for the first, compaction and per-turn retrieval for the second.

"Newer models have solved this." The curves are flatter and effective context length remains well below advertised length on benchmarks designed to measure it. Re-measure per model rather than assuming.

"Context rot is a model limitation." Mostly it is a system that was built to accumulate. A design that re-retrieves per turn and compacts history into structured state shows almost no degradation by turn 20.

Interview delivery note

Say this verbatim: "Retrieval recall and answer accuracy are different metrics and they can move in opposite directions. Going from 5 documents to 60 put the relevant one in context more often and put it in the middle, where accuracy is lowest, so accuracy fell 8 points while recall rose 12. The fix is to keep the recall and reduce the inclusion: retrieve 60, rerank, include 8." One concrete case that demonstrates the mechanism and the correct response.

The senior-versus-staff separator is separating the positional effect from the temporal one. A senior engineer knows about lost in the middle and reranks. A staff engineer also notices that a multi-turn agent's accuracy falls from 90 percent at turn 1 to 53 percent at turn 20 for a different reason (accumulated retrievals and contradictions rather than position), and fixes it with per-turn retrieval and structured state rather than by reordering. Two symptoms, two mechanisms, two fixes.

The second signal is running the positional sweep on your own task rather than citing the paper's curve. The trough depth and severity are task-specific, and knowing where your retriever tends to place the relevant document relative to your trough is what turns the finding into a design decision.

The Staff & Team Lead Interview Reference