Drills 1 to 6: AI, LLM and RAG

Six questions, ninety seconds each, out loud. Spoken form: what you would actually say, not what you would write.

The pattern in every answer: direct answer first, two or three supporting points, one thing that shows depth, then stop. The stopping matters as much as the content.

Drill 1. Why is prefill compute-bound and decode memory-bandwidth-bound?

Because of arithmetic intensity, meaning FLOPs per byte of weight read. In prefill you process the whole prompt at once, so it's a matrix-matrix multiply and you do roughly sequence-length operations for every byte of weight you pull out of memory. In decode you generate one token at a time, so it becomes matrix-vector: you read every single weight to produce one token per sequence, which is roughly batch-size operations per byte.

An H100 does around a thousand teraflops with about three and a third terabytes a second of memory bandwidth, so the ridge point is roughly 300 FLOPs per byte. Prefill with a two-thousand-token prompt sits well to the right of that and is compute-bound. Decode at batch size 8 sits at intensity 8, two orders of magnitude to the left, and is bandwidth-bound.

That single distinction explains most of a serving stack. It's why batching transforms decode throughput and does almost nothing for prefill, because in decode the weight read is amortised across the whole batch. It's why time-to-first-token and time-per-output-token are tracked separately. And it's why people now run prefill and decode on separate machine pools, since one wants compute and the other wants bandwidth.

Depth signal: naming the ridge point and doing the intensity comparison, rather than just asserting the two labels.

Full treatment: Prefill vs decode.

Drill 2. Your RAG answers are confidently wrong. Walk the diagnosis.

First I get a full trace for one specific failing query: the retrieved chunks with their scores, the assembled prompt, the model version, and the raw completion. Aggregate complaints aren't debuggable; one reproducible trace is.

Then the test that splits the problem in five minutes. I paste the known-correct passage into the context by hand and re-run. If the answer becomes right, it's a retrieval bug. If it stays wrong, it's a grounding bug. Those two have completely different fixes and the expensive mistake is treating a retrieval failure as a hallucination problem.

If it's retrieval, I'd check in order: vocabulary mismatch, which hybrid search fixes; a chunk boundary splitting the answer; a filter eating it, especially an access-control filter post-applied to an ANN result; and a stale index. If it's grounding: is the evidence buried in the middle of a long context, is there no abstention path so the model has to guess, and are the citations actually verified or just generated text.

Then I'd turn the anecdote into a measurement, with retrieval and generation metrics kept separate, because recall@k is the ceiling and no prompt work gets you above it.

Depth signal: the context injection test, and separating recall@k from faithfulness as different numbers owned by different parts of the system.

Full treatment: Diagnosing confidently wrong RAG.

Drill 3. When would you fine-tune instead of improving retrieval?

Fine-tuning is for form, retrieval is for facts. That's the one-line version and it decides most cases.

I'd fine-tune when I need the model to adopt a style, an output format, or a domain-specific way of reasoning that's hard to specify in a prompt: a consistent tone of voice, a rigid schema, a classification task where I have thousands of labelled examples and want a small cheap model to match a large one. And when latency matters enough that a retrieval hop doesn't fit.

I'd stay with retrieval when the knowledge changes, because a fine-tuned model is a snapshot and retraining is not a deploy. When I need citations, because a fine-tuned model can't tell you where it got something. And when access control matters, because a fine-tuned model can't forget one user's documents.

In practice they compose rather than compete: retrieval for recall, a fine-tune or good few-shot prompting for output form. The failure I'd watch for is a team fine-tuning to fix a retrieval problem, which is expensive, slow to iterate, and doesn't work, because the model still doesn't know the fact you failed to retrieve.

Depth signal: "form not facts", and naming access control as a reason retrieval wins, which almost nobody mentions.

Drill 4. How do you evaluate an agent that takes 20 steps?

On task-level success, not per-step accuracy, because they compound. Ninety-five percent per step over twenty steps is 0.95 to the twentieth, about thirty-six percent task success, so a per-step number that sounds excellent describes a system that fails two times in three.

So I'd define success as a checkable end state and assert it against the environment: the order is refunded, a refund record exists for the right amount, the email went out, and nothing else changed. That last assertion catches the agent that succeeds by doing something destructive alongside.

Alongside that: trajectory efficiency in steps and tokens, cost per successful task so a cheaper-model tradeoff is honest, and pass^k rather than pass@1. pass@1 asks whether it can do the task; pass^k asks whether it succeeds on all k independent attempts, and agents degrade sharply as k rises. For an autonomous workflow, consistency is the thing that decides whether you can ship.

Per-step analysis I'd use as a diagnostic rather than a metric, to categorise failures. My prior is that most are tool design rather than model capability: wrong tool selected usually means the description doesn't say when to call it.

Depth signal: pass^k, and the claim that most agent failures are tool-design failures.

Full treatment: Evaluating an agent.

Drill 5. How do you defend against indirect prompt injection?

At the architecture layer, not the prompt layer. Instructions and data arrive in the same channel, so a defensive instruction is just more text competing with the attacker's text, and the attacker gets unlimited attempts. Every published prompt-level defence has been broken.

I'd use the lethal trifecta as the checklist: private data, exposure to untrusted content, and a way to communicate externally. Remove any one leg and the attack can't complete. Usually the cheapest leg is egress, so an allowlist of hosts the agent may call, and not auto-rendering remote images from model output, because markdown image exfiltration needs no user click at all.

Then least privilege on the tool registry, because the blast radius of a successful injection is exactly the set of tools I granted, and human confirmation for anything irreversible. Then treat model output as untrusted input everywhere: never eval it, never pass it to a shell, escape it before rendering.

If I need stronger separation, the dual-LLM pattern: a quarantined model reads the untrusted content and returns only structured output, and a privileged model acts on that structure without ever seeing the raw text. The cost is that the privileged model has less context.

Depth signal: the lethal trifecta as a design checklist, and knowing that image exfiltration requires no click.

Full treatment: Prompt injection and the lethal trifecta.

Drill 6. Cut LLM spend 60 percent without hurting quality. What's the order?

Measure first, per feature, with an outcome field on every trace so I'm optimising cost per successful task rather than cost per call. The Pareto always surprises, and in every deployment I've seen one feature is doing something nobody intended.

Then, in order of return: prompt caching, which is usually the biggest single lever and is usually broken for a silly reason like a timestamp in the system prompt, since caching is a prefix match. Then context trimming, especially retrieval k, which typically improves quality at the same time because you removed distractors. Then model routing, cheap model first, escalate on a validator failure. Then batch APIs for anything not latency-sensitive, which is commonly around half price.

Semantic caching and distillation last. Semantic caching carries a correctness risk, because two similar queries can need different answers and in a multi-tenant product a near-miss returns another customer's context. Distillation is a real project with a payback calculation, not a tactic.

One thing about routing: the escalation rate matters more than the price ratio, because escalated requests pay for both calls. At thirty percent escalation with a fifth-price model you save about half; at sixty percent you save almost nothing and you've added latency to most requests.

And I'd push back gently on "without hurting quality", because the first two moves usually improve it.

Depth signal: the routing arithmetic, and challenging the premise that this is a quality tradeoff.

Full treatment: LLM cost engineering.

How to practise these

Read only the question. Ninety seconds, out loud, standing. Then read the answer and note the one thing you missed. Move on without re-answering, because immediately retrying just recites what you read.

Two failure modes specific to this set. Over-length: these are dense topics and the temptation is to keep going; the discipline is that a seventy-second answer that ends cleanly beats a three-minute one with the same content. And jargon without grounding: saying "PagedAttention" is worth nothing unless you can say what fragmentation it eliminates. If you cannot unpack a term one level, do not use it.