Prefill vs decode

What it is

Serving one request to an autoregressive language model has two phases with completely different hardware behaviour.

Prefill processes the entire prompt in one forward pass. Every token attends to every earlier token, and the work is expressed as large matrix-matrix multiplications over a sequence of length $S$. Arithmetic intensity is high, so the GPU's floating point units are the constraint. Prefill is compute bound.

Decode generates one token at a time. Each step is a forward pass over a single new token, which turns those matrix-matrix products into matrix-vector products. The model still has to read every weight and every cached key and value out of HBM to do a tiny amount of arithmetic. Decode is memory bandwidth bound.

This is not a subtlety. It is the single distinction that explains most of the architecture of a modern inference stack: why batching helps enormously in one phase and barely at all in the other, why time-to-first-token and time-per-output-token are tracked separately, why KV cache size limits your throughput more than model size does, and why serious deployments now run the two phases on different machines.

The problem it solves

Treating inference as one homogeneous workload leads to a deployment that is wrong for both halves. You size for FLOPs and discover you are bandwidth starved; you set a single latency SLO and discover that a long prompt blows the first-token budget while a long generation blows the total; you batch naively and discover that one 30,000-token prefill stalls every decode step behind it for hundreds of milliseconds.

Separating the phases gives you two independent levers with different physics, and lets you say precise things like "we are bandwidth bound in decode at batch size 8, so raising batch size is nearly free until KV cache exhausts memory".

Mechanics

Arithmetic intensity, the underlying reason

Take a linear layer with weight matrix $W \in \mathbb{R}^{d \times d}$ in fp16.

Prefill with $S$ tokens: $2 S d^2$ FLOPs, reading $2d^2$ bytes of weights. Intensity is roughly $S$ FLOPs per byte.
Decode, one token, batch $B$: $2 B d^2$ FLOPs, reading the same $2d^2$ bytes. Intensity is roughly $B$ FLOPs per byte.

An H100 SXM delivers on the order of 1,000 TFLOP/s of dense fp16 with about 3.35 TB/s of HBM bandwidth, so its ridge point on the roofline is roughly 300 FLOPs per byte. Prefill with a 2,000-token prompt sits far to the right of that ridge and is compute bound. Decode at batch size 8 sits at intensity 8, two orders of magnitude to the left, and is bandwidth bound. Getting decode back to compute bound would need a batch in the hundreds, which is exactly why every serving stack is built around making the batch as large as memory allows.

The KV cache, and why it is the real constraint

To avoid recomputing attention over the whole prefix at every decode step, the keys and values for every previous token are cached. Its size is:

$$\text{bytes} = 2 \times L \times H_{kv} \times d_{head} \times S \times B \times b$$

where $L$ is layers, $H_{kv}$ is key/value heads (fewer than query heads under grouped-query attention), $d_{head}$ is head dimension, $S$ is sequence length, $B$ is batch size, $b$ is bytes per element, and the leading 2 is for K and V.

Worked, for Llama 3 70B in fp16: $L = 80$, $H_{kv} = 8$ (GQA), $d_{head} = 128$, $b = 2$.

per token = 2 x 80 x 8 x 128 x 2 bytes = 327,680 bytes = 320 KiB

  8,000-token context, 1 sequence  ->  2.5 GiB
  8,000-token context, batch 32    ->  80 GiB

Weights in fp16 = 140 GB, which already needs 2x H100 80GB.
On 4x H100 (320 GB total): 320 - 140 = 180 GB left for KV cache,
activations and fragmentation. At 2.5 GiB per 8k sequence, that is
roughly 60 concurrent sequences if you had perfect packing, and
noticeably fewer in practice.

Two consequences follow immediately. First, KV cache, not weights, sets your maximum concurrency, so grouped-query attention (which cut $H_{kv}$ from 64 to 8 here, an 8x reduction in cache) is a serving optimisation as much as a quality one. Second, naive allocation wastes most of that memory: reserving a contiguous block for the maximum possible sequence length per request leaves it mostly empty, which is the problem PagedAttention solves by allocating the cache in fixed-size blocks with a page table, the same way an operating system handles virtual memory.

Why batching helps decode and barely helps prefill

In decode, the weight read is amortised across the batch: reading 140 GB of weights once serves 1 sequence or 64 sequences at almost identical cost. Going from batch 1 to batch 32 is close to a 32x throughput improvement, right up to the point where KV cache reads (which scale with the batch) start to rival weight reads.

In prefill, the GPU is already saturated with arithmetic from a single long prompt. Batching two 2,000-token prefills does not make them faster; it just queues them. This asymmetry is why continuous batching (admitting new requests at every decode step rather than waiting for a batch to drain) is the central throughput technique, and why a naive static batcher wastes most of the GPU.

The interference problem, and chunked prefill

Prefill and decode fight each other. A 30,000-token prefill occupies the GPU for hundreds of milliseconds; every decode step for every other in-flight request waits, so their inter-token latency spikes. Two fixes are in production use:

Chunked prefill splits a long prompt into pieces (say 512 tokens) and interleaves them with decode steps, so a long prompt raises everyone's latency slightly instead of stalling it badly. This trades a little TTFT for much better tail TPOT.

Disaggregation runs prefill and decode on separate GPU pools and ships the KV cache between them over a fast interconnect. Each pool can then be sized, batched and even quantised independently: prefill wants compute, decode wants bandwidth and capacity.

A worked example: reading the metrics

A chat endpoint reports p50 TTFT of 180 ms, p99 TTFT of 2.4 s, and a steady TPOT of 22 ms. Average prompt is 900 tokens, average output 300 tokens.

Total p50 latency is $180 + 300 \times 22 = 6.8$ seconds, of which 97 percent is decode. So optimising prefill would be nearly pointless for total latency, and extremely valuable for perceived latency, because TTFT is what the user feels before the first token appears.

The p99 TTFT of 2.4 seconds against a p50 of 180 ms is a 13x spread, which is not prompt-length variance alone. That signature is queueing: requests waiting for a scheduler slot behind long prefills. The diagnosis order is queue depth first, then prompt length distribution, then whether chunked prefill is enabled.

TPOT of 22 ms means about 45 tokens per second per sequence. If the deployment is bandwidth bound, the theoretical floor is (bytes read per step) / (HBM bandwidth). For a 70B fp16 model on 4 GPUs with tensor parallelism, each GPU reads roughly 35 GB per step, giving about 10 ms at 3.35 TB/s, so 22 ms means roughly 45 percent of peak bandwidth, which is a normal, healthy number once communication and kernel launch overhead are included. Being able to compute that floor and compare it to the observed number is the difference between tuning and guessing.

Production evidence

vLLM built its entire scheduler on this distinction. The PagedAttention paper (Kwon et al., SOSP 2023) documents that naive KV cache allocation wasted 60 to 80 percent of cache memory to internal and external fragmentation, and that paging it recovered nearly all of it, raising throughput by 2 to 4x at the same latency against the then-current serving systems. That memory recovery translates directly into larger decode batches.

Orca (Yu et al., OSDI 2022) introduced iteration-level scheduling, now universally called continuous batching, which is the direct consequence of decode being the batchable phase.

Microsoft's Splitwise (ISCA 2024) and DistServe (OSDI 2024) both disaggregate prefill and decode onto separate machine pools and report substantially better throughput under latency constraints than colocated serving, because the phases stop interfering and each pool can use hardware suited to its bottleneck. NVIDIA's TensorRT-LLM and vLLM have both shipped disaggregated serving support since.

NVIDIA's inference guidance and the metrics exposed by vLLM, TGI and TensorRT-LLM all separate TTFT from TPOT (sometimes called inter-token latency), which is the operational fingerprint of this split: you cannot have one latency SLO for an LLM endpoint.

The debate

The alternative to caring about this at all is to buy managed inference and treat latency as a vendor problem. That is a legitimate choice, and for most product teams the right one: the engineering to run vLLM well is not free, and a managed endpoint removes GPU capacity planning, which is the hardest part.

Where the choice tips: once you are spending more than roughly the fully loaded cost of an engineer per year on tokens, or once you need a model the vendors do not host, or once you need prefix caching over a large shared system prompt that the vendor does not expose, self-hosting starts to pay. Prefix caching is the underrated one: if every request shares a 4,000-token system prompt, caching that prefill turns the dominant cost of short-query workloads into a lookup.

Between chunked prefill and disaggregation: chunked prefill first, always. It is a configuration flag, it costs nothing in hardware, and it fixes the common case where a few long prompts ruin everyone's inter-token latency. Disaggregation is a real architecture change with a KV-cache transfer on the critical path, and it earns its complexity only at a scale where you are running distinct pools anyway, and where you have measured that prefill and decode want genuinely different hardware.

Prefill-versus-decode reasoning is the wrong lens when the model is small enough to be compute bound in decode too (small models at high batch), or when the bottleneck is not the GPU at all. A surprising share of "our LLM endpoint is slow" turns out to be tokeniser overhead, JSON serialisation, a synchronous retrieval call, or a guardrail model in the path.

Follow-up Q&A

"Why is prefill compute bound and decode memory bandwidth bound?" Because prefill does $O(S)$ FLOPs per byte of weight read while decode does $O(B)$. Both phases read the same weights; prefill has a long sequence to multiply them against and decode has one token per sequence. On an H100, the ridge point is around 300 FLOPs per byte; prefill with a long prompt is well past it, and decode at any realistic batch size is far short of it.

"How do you improve TTFT without touching the model?" Prefix caching, so a shared system prompt is prefilled once and reused; chunked prefill so long prompts do not queue behind each other; shorter prompts, which is a context engineering problem rather than a serving one; and streaming the response so the user sees the first token as soon as it exists. If TTFT is dominated by queueing rather than compute, add capacity or admission control, because no per-request optimisation fixes a queue.

"You doubled the GPU count and throughput went up 30 percent. Why?" Most likely you added tensor parallelism, which splits the weights across GPUs and so reduces per-GPU bytes read per decode step, but adds an all-reduce per layer. The communication cost eats part of the bandwidth win. Tensor parallelism helps latency; it does not give linear throughput. For throughput, replicating the model and load balancing across replicas is usually better, provided the model fits.

"What limits your batch size?" KV cache memory, not weights and not compute. Compute the per-token cache size from the formula, multiply by your context length and target concurrency, and compare against (GPU memory minus weights minus activation workspace). If the answer is uncomfortable, the levers are grouped-query attention or multi-head latent attention in the model, KV cache quantisation to fp8 or int8, shorter contexts, or paging with vLLM so you stop wasting cache on unused reservation.

"Your p99 TPOT degrades whenever a user pastes a long document. Diagnose." Prefill interference. A long prefill monopolises the GPU and every other request's decode step waits behind it. Enable chunked prefill, cap the maximum prompt length, or route long-context requests to a separate pool. This is the question that most cleanly separates people who have run a serving stack from people who have read about one.

Common misconceptions

The most common is that model size determines how many concurrent users you can serve. Weights are a fixed cost paid once; KV cache is the per-user cost and it scales with context length. A 7B model with 128k contexts can run out of memory faster than a 70B model with 2k contexts.

The second is that batching always helps. It transforms decode throughput and does very little for prefill, and a batching strategy that waits to fill a batch adds latency to every request in it. Continuous batching exists precisely so you never wait.

The third is treating TTFT and total latency as the same SLO. They have different causes, different fixes and different user impact, and a single "p99 latency" number for an LLM endpoint hides both.

Interview delivery note

Say this: "Prefill is compute bound because it does a matrix-matrix multiply over the whole prompt; decode is memory bandwidth bound because it reads every weight and the whole KV cache to produce one token. That is why batching transforms decode throughput and does almost nothing for prefill, why I track TTFT and TPOT separately, and why KV cache size rather than model size sets my maximum concurrency."

Then do the KV cache arithmetic out loud for the specific model under discussion. The depth signal here is not knowing the terms, it is producing the per-token cache size from layer count, KV head count and head dimension, and converting it into a concurrency limit. That calculation is what a staff-level answer looks like, and it takes about forty seconds.

The Staff & Team Lead Interview Reference