Compaction: rolling, hierarchical and structured state extraction

"Your agent is at step 25 and the context is full. What do you do?"

What it is

Reducing an accumulated conversation or agent trajectory so it fits a context budget while preserving what the next step actually needs.

THE PROBLEM SHAPE

  step 1:   system prompt + task                    2k tokens
  step 5:   + 4 tool calls and results             14k tokens
  step 15:  + 14 tool calls and results            58k tokens
  step 25:  + 24 tool calls and results           112k tokens

  Cost per step rises with the context, so step 25 costs
  roughly 50x what step 1 did. A step cap does NOT bound
  cost; only a token budget does.

Three strategies, and they preserve different things:

ROLLING WINDOW        Keep the last N turns, drop the rest.
                      Preserves: recency.
                      Loses: everything early, including the
                      task's own constraints.

HIERARCHICAL          Summarise older turns into progressively
SUMMARISATION         coarser summaries. Recent turns verbatim,
                      middle turns summarised, old turns
                      summarised into one paragraph.
                      Preserves: a lossy trace of everything.
                      Loses: specifics, irreversibly.

STRUCTURED STATE      Extract the agent's state into an explicit
EXTRACTION            schema (facts learned, decisions made,
                      open questions, artifacts produced) and
                      carry that instead of the transcript.
                      Preserves: what the schema names.
                      Loses: everything the schema does not.

Commonly confused with truncation. Truncation drops tokens; compaction decides what to keep, and the difference is whether anything reads the content before discarding it.

Also commonly confused with a context-window problem. A 1M-token window does not remove the need, because cost is linear in context, attention degrades over long contexts, and an agent carrying 800k tokens of irrelevant history is both expensive and worse at the task.

The problem it solves

Three costs that all scale with context length, and only one is the window limit.

1. MONEY. Cost is linear in input tokens. An agent whose
   context grows to 100k by step 25 pays 50x step 1's input
   cost on every subsequent step. This is why a step cap
   does not bound spend.

2. LATENCY. Prefill is compute-bound and linear in prompt
   length, so a 100k-token prompt is roughly 100x the prefill
   of a 1k one. Time to first token grows with the history.

3. QUALITY. Attention over a long context is not uniform.
   The "lost in the middle" effect means information in the
   middle of a long context is used less reliably than
   information at either end, so a long transcript can be
   worse than a short summary containing the same facts.

Point 3 is the one that surprises people: more context is not monotonically better, so compaction is sometimes a quality improvement rather than a cost concession.

Mechanics

Rolling window: cheap, and it loses the task

def rolling(messages, keep_recent=10):
    # ALWAYS keep the system prompt and the original task.
    # Dropping the task is the classic failure: at step 25
    # the agent has forgotten what it was asked to do and is
    # optimising the last thing it saw.
    return messages[:2] + messages[-keep_recent:]

What it is good for: single-task conversations where recency genuinely dominates, like a chat assistant answering follow-ups about the last thing discussed.

What it is bad for: anything where an early constraint matters. "Only use the staging database" said at step 1 is gone by step 15, and the agent will happily use production.

The mitigation that makes it usable: pin the constraints. Keep the system prompt, the original task, and an explicitly maintained constraints list outside the window.

Hierarchical summarisation: lossy, and it compounds

Levels, each summarising the one below:

  L0  turns 21-25          verbatim          8k tokens
  L1  turns 11-20          summarised        2k tokens
  L2  turns 1-10           summarised        400 tokens
  L3  the task + constraints                 300 tokens

Total: ~11k instead of 112k.

def hierarchical(messages, budget):
    recent = messages[-5:]                       # verbatim
    middle = summarise(messages[-15:-5], target=2000)
    old    = summarise(messages[:-15], target=400)
    return [system, task] + [old, middle] + recent

The failure mode that matters: summaries of summaries lose specifics irreversibly. By L2, "we tried three approaches and the second worked" has replaced the actual configuration that worked, and if the agent needs it later it cannot recover it.

The mitigation: summarise from the ORIGINAL, not from the previous summary.

WRONG:  L2 = summarise(L1)
        Errors and omissions compound at each level.

RIGHT:  L2 = summarise(original_turns_1_to_10)
        More expensive (you re-read the originals) and each
        level is one lossy step from the truth rather than
        three.

And keep the originals addressable. Store the full transcript externally and give the agent a tool to retrieve a specific turn by id, so a summary can say "the working configuration is in turn 14" and the agent can fetch it. That converts compaction from lossy to lazy, which is a categorically better property.

Structured state extraction: the strongest, and the most work

Instead of compressing the transcript, extract what the transcript was for.

@dataclass
class AgentState:
    task: str                        # never changes
    constraints: list[str]           # accumulates, never drops
    facts_learned: dict[str, str]    # key -> value, with the
                                     # turn id it came from
    decisions: list[Decision]        # what was chosen and why
    open_questions: list[str]
    artifacts: dict[str, str]        # name -> reference
    failed_approaches: list[str]     # so it does not retry them

# After every N steps, or when the budget is approached:
new_state = model.extract(
    schema=AgentState,
    current_state=state,
    recent_turns=messages[-10:],
    instruction="Update the state from these turns. Preserve "
                "every existing constraint and fact unless a "
                "turn explicitly contradicts it.",
)

Why this is better than summarisation: a summary is prose, so what it preserves is whatever the summariser found salient. A schema is a contract: constraints is never dropped because the field exists and the instruction says preserve it, and failed_approaches prevents the loop where an agent retries something it already tried and forgot about.

The costs, honestly:

- The schema has to be designed for the task, so it is not
  generic. A coding agent's state and a research agent's
  state have different fields.
- Extraction is an extra model call every N steps.
- Anything the schema does not name is lost completely,
  which is worse than a summary's partial preservation for
  the things you did not anticipate.
- Extraction can hallucinate a fact. Carry the turn id each
  fact came from so it is checkable.

failed_approaches is the field worth arguing for, because agent loops are the most common production failure and an agent that has forgotten it already tried something will try it again, indefinitely.

The hybrid, which is what to actually build

ALWAYS PINNED (never compacted)
  system prompt
  original task
  constraints list
  the last 3 to 5 turns verbatim

STRUCTURED STATE (extracted, updated every ~10 steps)
  facts, decisions, artifacts, failed approaches

EXTERNAL AND ADDRESSABLE (not in context)
  full transcript, retrievable by turn id
  large tool results, stored and referenced

SUMMARISED (from originals, not from summaries)
  the gap between the state extraction and the recent window

The rule that governs it: put things in context only if the next step might need them, and make everything else retrievable. A 200 KB tool result should be stored with a reference and a one-line description, not pasted into the transcript, and that single discipline prevents most context exhaustion.

When to compact

TRIGGERS, in order of preference

  token budget threshold      compact at 70% of the window,
                              not at 100%, because the
                              compaction call itself needs
                              room and the next step needs
                              headroom for a large tool result

  step count                  every N steps, predictable and
                              easy to reason about

  semantic boundary           when a sub-task completes.
                              The BEST trigger, because a
                              completed sub-task compresses
                              cleanly into its outcome, and
                              compacting mid-sub-task loses
                              the working state.

Compacting at a semantic boundary is the improvement most implementations miss, and it is usually available: an agent that has just finished "find the failing test" can compress that entire phase to "the failing test is X, at line Y" with no loss.

A worked example: an agent that failed at step 22

SYMPTOM
  A code-fixing agent capped at 30 steps consistently failed
  around step 22, and its final actions were incoherent:
  editing files it had already fixed, re-running tests it
  had already passed.

DIAGNOSIS
  Rolling window of 15 turns. By step 22 the context held
  turns 8 to 22, and:
    - the original task ("fix the failing tests in the
      payments module, do not change the API") was gone
    - the constraint about the API was gone, and it had
      changed a signature at step 19
    - the record of which tests it had already fixed was
      gone, so it re-fixed one and broke another

  The cost side: at step 22 the context was 84k tokens, so
  each step cost roughly 40x step 1. The 30-step cap allowed
  far more spend than anyone had estimated.

THE FIX
  1. Pin the system prompt, the task and an explicit
     constraints list. Never compacted. (~600 tokens.)
  2. Structured state extracted every 8 steps:
       tests_fixed: list
       tests_still_failing: list
       files_modified: list
       failed_approaches: list
       constraints: list
  3. Rolling window of 5 verbatim turns.
  4. Tool results over 4 KB stored externally with a
     reference and a one-line summary. Test output in
     particular was 60% of the context.
  5. Compaction triggered at a semantic boundary: after
     each test transitions to passing.

RESULT
  Context at step 22: 84k -> 11k tokens.
  Cost per step roughly flat rather than growing.
  The incoherence disappeared, because the two causes were
  the lost task and the lost record of what was already
  done, and the schema names both.

The finding worth extracting: the failure looked like a model capability problem and was a context management problem. The agent was not confused about the code; it had forgotten the task and its own history, and no better model fixes that.

Production evidence

Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023) documents that information in the middle of a long context is used less reliably than information at either end, which is the basis for the claim that compaction can improve quality rather than only reduce cost.

Anthropic's guidance on long-context prompting recommends placing the most important material at the start and end of a long context and using structured formats over raw transcripts, which is the same finding applied as practice.

Anthropic's published agent guidance describes compaction and external memory as the standard approaches to long-running agent context, including storing large results externally and referencing them.

The MemGPT / Letta line of work (Packer et al., 2023) treats the context window explicitly as a memory hierarchy with paging between an in-context working set and external storage, which is the strongest form of the "addressable, not lost" principle.

Prompt caching (as offered by several providers) changes the economics here in a way worth knowing: a stable prefix is charged at a fraction of fresh input tokens, so a pinned, unchanging system prompt and constraints block is much cheaper than one that is rewritten each step. That is an argument for keeping the pinned block byte-identical across steps rather than regenerating it.

The debate

The case for rolling windows: simple, predictable, no extra model calls, no schema design, and for conversational assistants where recency dominates it is sufficient. Complexity that is not needed is a cost.

The case for hierarchical summarisation: it preserves something from every part of the history rather than a hard cutoff, and it is generic, so one implementation works across tasks.

The case for structured state: it is a contract rather than a hope. Constraints are not dropped because the field exists, failed approaches are recorded because the field exists, and the most common agent failure (looping on something already tried) is addressed by construction.

The case for a very large context window instead: just fit everything. Simple, and it is expensive linearly, slow in prefill, and worse in quality because of the lost-in-the-middle effect, so it does not remove the problem.

My position: pin the task and constraints, extract structured state, keep a short verbatim window, and store everything else externally and addressably.

The pinning is non-negotiable and it is the cheapest fix available. The failure I have seen most often is an agent that has forgotten its own task, and in the worked example that was the entire cause of what looked like a model capability problem. A few hundred tokens of never- compacted task and constraints prevents a whole failure class.

Structured state over summarisation, because a summary preserves what the summariser found salient and a schema preserves what you named. The specific field I would argue hardest for is failed_approaches, because agent loops are the most common production failure and an agent that has forgotten it already tried something will try it again indefinitely, which no step cap prevents cheaply.

The property I would design for is addressable rather than lost: store the full transcript and large tool results externally, give the agent a retrieval tool, and let summaries reference turn ids. That converts compaction from lossy to lazy, which is categorically better, and it means a summary saying "the working configuration is in turn 14" is actionable rather than frustrating.

Two implementation details worth insisting on. Summarise from the originals, not from the previous summary, because errors compound at each level and a three-level summary of a summary of a summary is unrecoverable. And compact at a semantic boundary where one is available, because a completed sub-task compresses cleanly to its outcome while compacting mid-sub-task destroys the working state.

Where I would push back on the framing: a bigger context window does not solve this. Cost is linear, prefill is linear, and the lost-in-the-middle effect means an agent carrying 800k tokens of history can be worse at the task than one carrying an 11k summary of the same information. Compaction is sometimes a quality improvement, not a cost concession.

Follow-up Q&A

"Your agent is at step 25 and the context is full. What do you do?" First, pin what must never be compacted: the system prompt, the original task and an explicit constraints list. The most common agent failure I have seen is that it has forgotten what it was asked to do and is optimising the last thing it saw. Then extract structured state rather than summarising prose, keep the last three to five turns verbatim, and store everything else externally with a retrieval tool so it is addressable rather than lost.

"Why structured state rather than summarisation?" Because a summary preserves whatever the summariser found salient, and a schema preserves what you named. constraints is not dropped because the field exists and the instruction says preserve it. failed_approaches prevents the loop where an agent retries something it already tried and forgot, which is the most common production failure and which no step cap prevents cheaply. The cost is that the schema must be designed per task and anything it does not name is lost completely.

"Doesn't a million-token context window remove the problem?" No, for three reasons. Cost is linear in input tokens, so an agent at 800k tokens pays that on every subsequent step, and a step cap therefore does not bound spend. Prefill is compute-bound and linear, so time to first token grows with the history. And the lost-in-the-middle effect means information in the middle of a long context is used less reliably, so a long transcript can be worse than a short summary containing the same facts. Compaction is sometimes a quality improvement.

"What's the failure mode of hierarchical summarisation?" Compounding loss. If each level summarises the previous summary, errors and omissions accumulate, and by the third level "we tried three approaches and the second worked" has replaced the actual configuration that worked. The fix is to summarise from the originals each time, which is more expensive because you re-read them, and it means each level is one lossy step from the truth rather than three.

"How do you avoid losing something you later need?" Make it addressable rather than dropped. Store the full transcript externally, keyed by turn id, and give the agent a tool to retrieve a turn. Then a summary can say "the working configuration is in turn 14" and the agent can fetch it. That converts compaction from lossy to lazy, which is a categorically different property, and the same applies to large tool results: store them with a reference and a one-line description rather than pasting 200 kilobytes into the transcript.

"When should compaction trigger?" Preferably at a semantic boundary, when a sub-task completes, because a completed sub-task compresses cleanly to its outcome and compacting mid-sub-task destroys the working state. Failing that, at a token threshold around 70 percent of the window rather than at 100, because the compaction call itself needs room and the next step needs headroom for a large tool result. Step count is the simplest trigger and the least informed.

"What does compaction cost?" An extra model call every N steps for the extraction or summarisation, which is small relative to what it saves. The subtler cost is that prompt caching economics change: a pinned block that is byte-identical across steps is charged at a fraction of fresh input tokens, so regenerating the pinned section each step throws away that saving. That is an argument for keeping the pinned block stable rather than rebuilding it.

"Walk me through a real failure." A code-fixing agent capped at 30 steps consistently failed around step 22 with incoherent actions: editing files it had already fixed, re-running passing tests. The cause was a 15-turn rolling window, so by step 22 the original task and the constraint "do not change the API" were both gone, along with the record of which tests it had fixed. It had changed a signature at step 19. Pinning the task and constraints, extracting a schema with tests_fixed and failed_approaches, and storing test output externally took the context from 84k to 11k tokens and the incoherence disappeared. It looked like a model capability problem and it was a context management problem.

Common misconceptions

"Compaction is truncation." Truncation drops tokens; compaction reads the content and decides what to keep. Rolling windows are the truncation-shaped version and lose the task.

"A bigger window removes the need." Cost and prefill are linear in context, and lost-in-the-middle means quality is not monotonic in context length.

"Summarise the summary to save calls." Errors compound at every level. Summarise from the originals.

"A step cap bounds agent cost." Context grows per step, so later steps cost far more. Only a token or cost budget bounds spend.

"Anything dropped is gone." Only if you did not store it. Externally addressable transcripts make compaction lazy rather than lossy.

Interview delivery note

Lead with the pinning, because it is the cheapest fix and it addresses the most common failure: "The first thing I'd do is pin what must never be compacted: the system prompt, the original task, and an explicit constraints list. The failure I've seen most often is an agent that has forgotten what it was asked to do and is optimising the last thing it saw. In one case an agent had lost 'do not change the API' by step 15 and changed a signature at step 19."

Then the strategy choice with the reason: "Then structured state extraction rather than summarisation, because a summary preserves what the summariser found salient and a schema preserves what you named. The field I'd argue hardest for is failed_approaches, because agent loops are the most common production failure and an agent that has forgotten it already tried something will try it again."

Give the property that changes the character of the problem: "And I'd make everything addressable rather than dropped. Full transcript stored externally by turn id, large tool results stored with a reference, and a retrieval tool. Then a summary can say 'the working config is in turn 14' and the agent can fetch it. That turns compaction from lossy into lazy."

Correct the big-window assumption, because it is the common objection: "And a million-token window doesn't remove this. Cost is linear, prefill is linear, and the lost-in-the-middle finding means information in the middle of a long context is used less reliably. So an agent carrying eight hundred thousand tokens of history can be worse at the task than one carrying an eleven-thousand-token summary of the same facts. Compaction is sometimes a quality improvement, not a cost concession."

The detail that shows implementation experience: "and I'd trigger compaction at a semantic boundary where one exists, when a sub-task completes, because that compresses cleanly to its outcome. Compacting mid-sub-task destroys the working state, which is what a fixed step trigger does."

The Staff & Team Lead Interview Reference