Drills 7 to 9: context and graph engineering
Three drills, ninety seconds each, out loud. These are the newest questions in the bank and the ones where a candidate is most likely to give a fashionable answer rather than a costed one.
The pattern across all three: the good answer includes an arithmetic reason for the design, and the weak answer names a technique. "We'd use RAG" or "we'd add a knowledge graph" or "we'd summarise the context" are techniques. The costs are what make them decisions.
Drill 7. Your prompt is 3,000 tokens of business rules. What do you do?
First I'd ask what fraction of a request actually needs them, because the answer is usually a small one. Three thousand tokens of rules covering every product line and every jurisdiction, when a given request touches one product and one jurisdiction, means most of that context is paid for on every call and used on almost none.
So: retrieve the applicable rules rather than including all of them. Classify the request first, which is cheap, then pull the two or three rule sets that apply. That typically takes three thousand tokens to three or four hundred.
Second, order the prompt for the provider's cache. Stable content first, variable content last, so the system prompt and the few-shot examples form a cacheable prefix. For a workload with a large stable prefix and a small variable suffix, cache reads cost a fraction of fresh input tokens, and that is usually the largest single cost saving available. Which also means the rules that genuinely are universal should stay in the prefix rather than being retrieved, because they are nearly free there.
Third, and this is the one people skip: check whether the rules should be in the prompt at all. A deterministic rule like "orders over five thousand need approval" is better enforced in code, where it is testable and cannot be talked out of by a model. What belongs in the prompt is the judgement, not the arithmetic.
The thing I'd measure afterwards is not just token count but whether accuracy moved, because trimming context can improve it. Lost-in-the-middle means information in the middle of a long prompt is used less reliably, so a shorter, more relevant prompt is sometimes better rather than merely cheaper.
Depth signal: the cache-ordering point, and the observation that some of these rules belong in code rather than in the prompt.
Full treatment: Budgeting a context window and Compaction.
Drill 8. When does a knowledge graph beat a vector index?
Four cases, and outside them a vector index is better and cheaper.
Multi-hop questions, where the answer needs facts from different documents and no single chunk contains the chain. "Which of our suppliers are affected by the port closure" needs closure to region, region to supplier, supplier to us. Vector search returns chunks about closures and chunks about suppliers and the join happens inside the model, unverifiably.
Global questions about the corpus rather than a passage. "What are the recurring themes across these four hundred incidents" cannot be answered by top-k retrieval at any k, because the question is about all of it. That is what GraphRAG's community summarisation does and it has no vector-index equivalent.
Relationship questions where the structure is the answer, like dependency impact analysis.
And anywhere the reasoning path has to be auditable, because a traversal is explainable and "cosine similarity was 0.83" is not.
Then the part that decides whether it is affordable. The expensive bit is extraction, one LLM call per chunk, so a hundred thousand chunk corpus is a hundred thousand calls before answering anything, and you pay it again when the schema changes. So the first thing I'd check is how much of the graph already exists in structured systems: service catalogue, tracing data, org directory, ticket system, product taxonomy. In one case that was the difference between extracting over three thousand postmortems and over forty thousand documents, about a thirteenth of the cost, and the structured edges were more accurate than extracted ones.
And I wouldn't build one until I'd categorised the retrieval failures, because a good share of what looks multi-hop is ordinary retrieval failure that a reranker and better chunking fix for a fraction of the cost.
Depth signal: the extraction cost as the deciding factor, and checking for an existing structured graph before building one.
Full treatment: Knowledge graph vs vector index.
Drill 9. How do you budget a 128k context window for an agent?
I'd allocate it by category rather than filling it, and I'd reserve rather than let things grow into each other.
Pinned and never compacted: system prompt, the original task, and an explicit constraints list. Maybe a thousand tokens. The most common agent failure I've seen is that it forgot what it was asked to do and is optimising the last thing it saw, and this prevents a whole class of that.
Working set, the last three to five turns verbatim, so the immediate reasoning has full fidelity.
Structured state, extracted every eight to ten steps into a schema: facts learned, decisions, artifacts, and crucially failed approaches, because agent loops are the most common production failure and an agent that has forgotten it already tried something will try it again indefinitely.
Retrieved content for the current step, which is the largest variable block.
And headroom, deliberately, maybe twenty percent, because the next tool call might return something large and there has to be room for it plus the response.
Two rules that govern the allocation. Anything over about four kilobytes from a tool gets stored externally and returned as a reference with a one-line summary, because a single large tool result is the most common way a run dies. And compaction triggers at seventy percent of the window rather than a hundred, because the compaction call itself needs room.
And I'd say that a bigger window doesn't remove the need for this. Cost is linear in input tokens, prefill is linear, and lost-in-the-middle means an agent carrying eight hundred thousand tokens of history can be worse at the task than one carrying an eleven-thousand-token summary of the same facts. So a step cap doesn't bound cost either: you need a token budget.
Depth signal: the headroom reserve, the four-kilobyte tool-result rule, and the point that a step cap does not bound cost.
Full treatment: Budgeting a context window and Compaction.
How to practise these
These three are where a fashionable answer is most tempting, and the practice is to attach a number to every design choice.
"retrieve rather than include" -> 3,000 tokens to ~400
"order for the cache" -> cache reads are a
fraction of fresh input
"build a knowledge graph" -> one LLM call per chunk,
so 100k chunks is 100k
calls before answering
anything
"budget the window" -> 20% headroom, 4 KB tool
result threshold, compact
at 70%
Three tests for your own answer:
- Did you cost the technique? Extraction cost is what decides whether a knowledge graph happens. Cache ordering is what decides whether a large prompt is affordable. A technique without its cost is a suggestion.
- Did you check whether something cheaper works first? Categorise the retrieval failures before building a graph. Check for an existing structured graph before extracting one. Fix chunking before adding a stage.
- Did you name what belongs outside the model? Deterministic rules belong in code, large tool results belong in external storage, and the reasoning about what belongs where is the actual engineering.
And the observation that runs through all three, worth saying explicitly when it fits: less context is sometimes better, not merely cheaper. Lost-in-the-middle means relevance beats volume, which inverts the instinct that a bigger window solves the problem.