Context engineering vs prompt engineering, and the six-stage pipeline

What it is

Prompt engineering is authoring the static text you send: instructions, format specifications, few-shot examples, tone. It is a writing task, done once, versioned like code, and it is largely finished when the prompt works.

Context engineering is the design of the system that assembles what goes into the window on each request: what gets retrieved, what gets included from history, what gets summarised away, what gets dropped, and in what order. It is a runtime data pipeline, and it is never finished, because its inputs change with every conversation and every document in the corpus.

The distinction matters because they fail differently and are owned by different people:

	Prompt engineering	Context engineering
Artifact	A string	A pipeline
Changes	On deploy	On every request
Failure	Model misunderstands the task	Model has the wrong information
Debugging	Read the prompt	Reconstruct what was assembled
Owner	Whoever wrote the feature	Whoever owns the retrieval and memory systems
Testing	Golden prompts, eval set	Requires replaying real assembled contexts

What this is confused with: "just use a bigger context window." A 128k window does not remove the need to choose what goes in it, for three independent reasons developed throughout this page: cost is linear in tokens, latency is linear in prefill, and quality is not uniform across the window. The last one is the surprise, and it is why context engineering became a discipline exactly as windows got large enough that people assumed it would not need to be.

The problem it solves

Once an LLM feature is more than a single-turn prompt, the model's input is assembled from several sources with competing claims on a fixed budget:

System instructions          ~800 tokens     (fixed, required)
Tool definitions            ~2,400 tokens     (grows with every tool added)
Conversation history         variable         (grows without bound)
Retrieved documents          variable         (as many as you choose)
User's current message       variable
Space reserved for output    ~1,000 tokens    (must be reserved, not borrowed)

Nothing in that list shrinks on its own. History grows every turn, the tool registry grows every quarter, and retrieval will return as much as you ask for. Without a deliberate policy, one of three things happens: you exceed the window and the request fails; you truncate arbitrarily and lose something important; or you fill the window with low-value content and pay for it in cost, latency and quality.

The failures this produces are recognisable and are usually misdiagnosed as model problems:

The agent forgets a constraint stated 20 turns ago, because history was truncated from the front.
The model ignores a retrieved document that clearly answers the question, because it landed in the middle of a large context.
Cost per conversation grows superlinearly, because every turn re-sends the whole history: turn n costs O(n) tokens, so a conversation of N turns costs O(N^2).
Answers get worse as the conversation gets longer, which is context rot, covered on its own page.

That quadratic cost is worth stating explicitly because it surprises people:

20-turn conversation, ~500 tokens added per turn, no compaction:
  turn 1:    500 tokens
  turn 10:  5,000 tokens
  turn 20: 10,000 tokens
  total input across the conversation: ~105,000 tokens for ~10,000 of content

Ten times the content, billed. Compaction and caching are the two levers, and both are context engineering rather than prompt engineering.

Mechanics

The six stages

        ┌──────────────┐
user ──▶│ 1. SELECT    │  what sources are eligible for this request?
        └──────┬───────┘
               ▼
        ┌──────────────┐
        │ 2. RETRIEVE  │  fetch candidates from each source
        └──────┬───────┘
               ▼
        ┌──────────────┐
        │ 3. RANK      │  score and order by relevance
        └──────┬───────┘
               ▼
        ┌──────────────┐
        │ 4. COMPRESS  │  summarise, extract, truncate to fit
        └──────┬───────┘
               ▼
        ┌──────────────┐
        │ 5. ASSEMBLE  │  ORDER matters; position is a quality decision
        └──────┬───────┘
               ▼
        ┌──────────────┐
        │ 6. OBSERVE   │  log what was actually assembled, for debugging
        └──────────────┘

1. Select. Decide which sources are eligible before retrieving from any of them. A question about a person's own account should not search the public documentation corpus, and a question about API syntax should not search the CRM. This stage is cheap, it is frequently skipped, and skipping it means every downstream stage does more work on candidates that were never relevant.

It is also where access control belongs. Filtering retrieved results after retrieval is both slower and more dangerous than filtering the candidate set before: a bug in post-filtering leaks data, where a bug in pre-filtering returns nothing.

def select_sources(request) -> list[Source]:
    sources = []
    if request.mentions_account_data:
        sources.append(AccountSource(tenant_id=request.tenant_id))   # scoped
    if request.is_technical:
        sources.append(DocsSource(product=request.product))
    if request.has_history:
        sources.append(ConversationSource(session=request.session_id))
    return sources

2. Retrieve. Fetch candidates. Over-fetch deliberately, because ranking is cheaper than retrieval and you cannot rank what you did not retrieve. See hybrid retrieval and RRF.

3. Rank. Score and order. A cross-encoder reranker over 50 candidates is the standard shape (see cross-encoder and LLM reranking). This stage is what lets stage 2 be generous.

4. Compress. Fit the budget. The options in order of information preserved per token:

Extraction:      pull only the relevant spans from each document
Summarisation:   an LLM call to compress (costs a call, and can lose specifics)
Truncation:      cut to a token limit (cheapest, loses the most)
Dropping:        remove the lowest-ranked entirely (better than truncating all)

Dropping low-ranked documents beats truncating every document, and it is the choice teams get wrong. Truncating all ten documents to 200 tokens gives you ten fragments, each possibly cut mid-fact. Keeping the top four intact gives you four usable documents. The same budget, very different utility.

5. Assemble. The stage that is invisible and matters most. Ordering is a quality decision, because of "lost in the middle":

def assemble(system, tools, history, docs, query, budget):
    # Most important at the EDGES; least important in the middle.
    return "\n\n".join([
        system,                          # start: highest attention
        tools,
        format_docs(reversed(docs)),     # best doc LAST in this block,
                                         # so it is nearest the query
        compress(history, budget),       # middle: lowest attention
        query,                           # end: highest attention
    ])

Placing the highest-ranked document immediately before the user's query, rather than first in the document block, exploits the recency end of the U-shaped attention curve. It costs nothing.

6. Observe. Log the assembled context, or you cannot debug the system.

@dataclass
class ContextTrace:
    request_id: str
    sources_selected: list[str]
    candidates_retrieved: dict[str, int]      # per source
    docs_after_rank: list[tuple[str, float]]  # id and score
    compression_applied: dict[str, str]       # what was summarised or dropped
    final_token_counts: dict[str, int]        # per section
    assembled_hash: str                       # to correlate with the output

Without this you cannot answer "why did the model say that." The prompt is in git; the context is not, and the context is what the model actually saw. This is the single most commonly missing piece in production LLM systems and it turns every quality investigation into guesswork.

Budget allocation, as a policy

CONTEXT_LIMIT = 128_000
RESERVED_OUTPUT = 4_000

BUDGET = {
    "system":    ("fixed",     1_000),   # never compressed
    "tools":     ("fixed",     3_000),   # never compressed
    "history":   ("elastic",  20_000),   # compacted when over
    "documents": ("elastic",  40_000),   # documents dropped when over
    "query":     ("fixed",     2_000),
}
# Total allocated: 66,000 of 124,000 usable. Deliberate headroom.

Allocating well under the limit is the right default, for the three reasons that recur: cost is linear, prefill latency is linear, and quality degrades before the limit does. A team that fills 124k because the window allows it is paying three times over for content the model attends to poorly.

The elastic sections need an eviction order, and it should be explicit:

def fit_to_budget(sections, limit):
    total = sum(s.tokens for s in sections)
    while total > limit:
        # Evict in a DEFINED order, never arbitrarily.
        victim = min((s for s in sections if s.elastic), key=lambda s: s.priority)
        freed = victim.compress_one_step()      # summarise, or drop lowest-ranked
        total -= freed
        if freed == 0:
            raise ContextOverflow(f"cannot compress {victim.name} further")
    return sections

The raise is important. Silently truncating when compression is exhausted produces a model call with missing information and a plausible wrong answer. Failing loudly lets the caller decide (split the request, drop a source, escalate).

A worked example: an agent that forgot its instructions

A customer-service agent with tool access. Conversations averaged 14 turns; some ran past 40.

Reported symptoms, over three months:

"The agent forgets it is not allowed to issue refunds over $500."
"It stops using the order-lookup tool after a while and guesses."
"Long conversations give worse answers than short ones."
"Cost per conversation is way above our model."

Four reports, treated as four separate issues, filed against "model quality."

What the context assembly actually did:

# The original: naive concatenation, truncate from the front on overflow.
def build_context(session):
    parts = [SYSTEM_PROMPT, TOOL_DEFINITIONS]
    parts += [format_turn(t) for t in session.turns]        # ALL of them
    parts.append(session.current_message)
    text = "\n\n".join(parts)
    if count_tokens(text) > LIMIT:
        text = truncate_from_front(text, LIMIT)              # <- the bug
    return text

truncate_from_front removed the system prompt and the tool definitions first, because they were at the front. All four symptoms follow from that one line:

Refund limit forgotten: it was in the system prompt, which was truncated away.
Stopped using tools: the tool definitions were truncated away, so the model had no tools to call and guessed instead.
Long conversations worse: they were the ones that overflowed.
Cost: full history re-sent every turn, so a 40-turn conversation cost O(N^2).

They had no context trace, so nobody could see this. The prompt in git looked correct. The model was being sent something entirely different.

The rebuild, stage by stage:

def build_context(session, request):
    trace = ContextTrace(request_id=request.id)

    # 1. SELECT
    sources = select_sources(request)                        # tenant-scoped
    trace.sources_selected = [s.name for s in sources]

    # 2. RETRIEVE (over-fetch; ranking is cheaper than retrieval)
    candidates = [d for s in sources for d in s.retrieve(request.query, k=30)]

    # 3. RANK
    ranked = reranker.rank(request.query, candidates)[:8]

    # 4. COMPRESS: history compacted, documents DROPPED not truncated
    history = compact(session.turns, budget=BUDGET["history"])
    docs, dropped = fit_documents(ranked, budget=BUDGET["documents"])
    trace.compression_applied = {"history": history.method, "docs_dropped": dropped}

    # 5. ASSEMBLE: fixed sections are NEVER evictable
    return assemble(
        system=SYSTEM_PROMPT,          # fixed
        tools=TOOL_DEFINITIONS,        # fixed
        docs=docs,
        history=history.text,
        query=request.current_message, # last, adjacent to the highest-ranked doc
        trace=trace)

Plus the structural change that fixed the cost:

# History compaction: keep the last 6 turns verbatim, and maintain a
# STRUCTURED STATE object for everything older, rather than a prose summary.
@dataclass
class ConversationState:
    customer_id: str
    order_ids_discussed: list[str]
    constraints_established: list[str]   # "customer wants refund not exchange"
    actions_taken: list[str]             # "looked up order #4471, status shipped"
    unresolved: list[str]

Structured state beat prose summarisation, which they measured. A prose summary of 20 turns lost specifics (order numbers, exact amounts) that the model later needed; a structured object kept exactly the fields that turned out to matter and cost a fifth of the tokens. See compaction.

Measured over the following six weeks:

                                    before      after
system prompt present in context     78%        100%
tool definitions present             71%        100%
constraint-violation reports         31/wk        0/wk
tool-call rate (turns 20+)           0.31        0.94    (vs 0.96 for turns 1-5)
answer quality (human eval, 40+ turns) 3.1/5     4.4/5
mean input tokens per turn           14,200     4,900    (-65%)
cost per conversation                $0.41      $0.11    (-73%)
p50 latency                          3.9s       1.4s

Zero constraint violations, from 31 a week, and a 73 percent cost reduction. No model change, no prompt change: the system prompt text was identical throughout. The entire difference was in what got assembled and what got evicted.

The number that reframed the team's thinking was "system prompt present in context: 78 percent." Nobody had considered that a measurable property. The prompt was in the repository, reviewed, tested, and absent from more than one call in five. Once they started asserting on the assembled context rather than on the prompt template, three more latent bugs surfaced within a month.

# The test that would have caught it, and now runs on every assembled context.
def test_fixed_sections_always_present(assembled: str):
    assert SYSTEM_PROMPT_MARKER in assembled
    assert TOOL_DEFINITIONS_MARKER in assembled
    assert count_tokens(assembled) < CONTEXT_LIMIT - RESERVED_OUTPUT

Production evidence

Anthropic's engineering writing on context engineering frames it explicitly as the successor discipline to prompt engineering, with the argument that as models improved, the binding constraint moved from instructing the model to supplying it with the right information. The framing of context as a scarce resource to be allocated rather than a buffer to be filled is theirs.

"Lost in the Middle" (Liu et al., 2023) is the empirical basis for stage 5 mattering. Accuracy is high when relevant information sits at the start or end of the context and drops substantially in the middle, consistently across models. Ordering is therefore a quality decision rather than a formatting one.

LangChain, LlamaIndex and Semantic Kernel all converged on comparable pipeline abstractions (retrievers, rerankers, context compressors, prompt assemblers), which is reasonable evidence that the six stages are the natural decomposition rather than one team's taxonomy.

Prompt caching, offered by Anthropic, OpenAI and Google, changes the cost calculus directly: a stable prefix (system prompt, tool definitions, retrieved documents that do not change within a conversation) is cached and charged at a large discount. This makes assembly order a cost decision as well as a quality one, because anything before the first varying token is cacheable and anything after it is not. Putting a per-request timestamp near the top of a prompt destroys the cache for everything after it.

LangSmith, Langfuse, W&B Weave and Braintrust all record the fully-assembled prompt per call. That the entire observability product category treats this as the primary artifact confirms the stage-6 argument: the assembled context, not the template, is what you debug.

The debate

Is context engineering a real discipline or a rebranding? It is real, and the test is that the artifacts differ: prompt engineering produces a string in a repository, context engineering produces a pipeline with retrieval, ranking, compression, budget policy and tracing. They are owned by different people and they fail differently. The rebranding objection is fair about the term and wrong about the substance, and the substance is that assembling the context is now where most of the engineering effort in an LLM feature goes.

Does a large context window make this unnecessary? No, on three independent grounds, and it is worth being able to state all three because people usually offer only the first. Cost is linear in input tokens, so filling 128k costs 32 times what filling 4k does. Latency is linear in prefill. And quality is not uniform: information in the middle of a long context is measurably less likely to be used. Large windows make context engineering more forgiving (you can be less precise) and not optional.

Should you compress with an LLM call or with rules? Rules first. A structured state object extracted with explicit fields is cheaper, faster, deterministic, and testable, and in the worked example it preserved the specifics (order numbers, amounts) that prose summarisation lost. Use an LLM summariser for genuinely unstructured content where you cannot enumerate the fields that matter, and accept that it costs a call, adds latency, and can silently drop a detail. If you can name the fields, extract them.

How much of the window should you use? Well under the limit, and I would treat 50 to 70 percent of the usable window as a working target. The remaining space is not waste, it is headroom for a long user message, a large tool result, or an unusually long history, and running close to the limit means an occasional request that cannot be assembled at all. Systems that routinely fill the window have no margin for their own worst case.

What is the highest-value thing to build first? Stage 6, the trace. Every other stage is guesswork without it, and the worked example is the argument: four separate bug reports over three months, all one line of code, invisible because nobody could see what the model was sent. Build the trace before you build the pipeline, because the pipeline will be wrong and you will need to see how.

Follow-up Q&A

"What is the difference between prompt engineering and context engineering?"

Prompt engineering is authoring the static text: instructions, format, examples. It changes on deploy, lives in git, and is debugged by reading it. Context engineering is the runtime pipeline that assembles what actually goes into the window: selection, retrieval, ranking, compression, ordering and tracing. It changes on every request and is debugged by reconstructing what was assembled. The practical consequence is that a correct prompt in your repository tells you very little, because the prompt is one input to a system that decides what the model actually sees.

"Why not just use the whole context window?"

Three independent reasons. Cost is linear in input tokens, so 128k costs 32 times what 4k does per request. Prefill latency is linear, so a full window is noticeably slower. And quality is not uniform across the window: information in the middle is measurably less likely to be used than the same information at the edges. A large window makes context engineering more forgiving, not unnecessary.

"How do you decide what to evict when you are over budget?"

By a defined priority order, never arbitrarily, and with fixed sections marked non-evictable. The failure I have seen is truncating from the front, which removes the system prompt and tool definitions first, because they are at the front. And when compressing documents, drop the lowest-ranked entirely rather than truncating all of them: four intact documents beat ten fragments cut mid-fact at the same budget. If compression is exhausted, fail loudly rather than sending a call with missing information.

"What is the most common context engineering bug?"

Something important silently absent, and the reason it persists is that nobody is measuring presence. In one system the system prompt was in the assembled context 78 percent of the time and the tool definitions 71 percent, which produced four separate bug reports over three months, all attributed to model quality. The fix is asserting on the assembled context rather than the template, and the first such assertion usually finds more than the one you wrote it for.

"Where does ordering matter and why?"

At assembly. Because of the U-shaped attention curve, information at the start and end of the context is used more reliably than information in the middle. So fixed instructions go at the start, the user's query at the end, and the highest-ranked retrieved document immediately before the query rather than first in its block. It costs nothing and it exploits the recency end of the curve. Prompt caching adds a second constraint pulling the same way: anything before the first varying token is cacheable, so stable content belongs at the top.

"What would you build first?"

The trace. Logging the fully-assembled context per request, with per-section token counts and what was compressed or dropped, is what makes every other stage debuggable. Without it, a quality investigation is guesswork, because the prompt in the repository is not what the model received. Every observability product in this space treats the assembled prompt as the primary artifact, which is a fair signal about where the information is.

Common misconceptions

"A bigger context window removes the need for this." It changes the constraint from "does it fit" to "is it worth including," and the second question is harder. Cost, latency and the non-uniform quality across the window all remain.

"The prompt in the repository is what the model sees." It is one input to an assembly process that also decides what to retrieve, what to keep from history, and what to evict. In one measured case the system prompt reached the model 78 percent of the time.

"Truncating is a reasonable fallback." Truncating from the front removes your instructions; truncating every document produces fragments cut mid-fact. Both are worse than dropping the lowest-priority content entirely, and both hide the failure.

"Summarisation is the way to compress history." Structured state extraction is cheaper, deterministic and testable, and it preserves the specifics that prose summaries lose. Reach for LLM summarisation when you genuinely cannot enumerate the fields that matter.

"Context engineering is just RAG." Retrieval is one of six stages. Selection, budget policy, compression, ordering and tracing are the rest, and the failures in the worked example were all outside retrieval.

Interview delivery note

Say this verbatim: "Prompt engineering produces a string; context engineering produces a pipeline. The prompt is in git and reviewed, and in one system I would point to, the system prompt actually reached the model 78 percent of the time, because overflow truncated from the front. That produced four separate bug reports attributed to model quality." A concrete, memorable demonstration that the artifact you review is not the artifact the model sees.

The senior-versus-staff separator is treating presence as a measurable property. A senior engineer designs the retrieval and compression stages well. A staff engineer adds that the assembled context must be traced and asserted on, because the most common failure is something important silently missing, and no amount of prompt review catches it. "System prompt present in 78 percent of calls" is a metric nobody thinks to collect and it resolved four issues at once.

The second signal is knowing that dropping beats truncating. Given a budget, keeping four documents intact beats truncating ten to fit, because a fragment cut mid-fact is worse than absent, and the model cannot tell you which it received.

The Staff & Team Lead Interview Reference