Prefill and decode: why output costs five times more
TL;DR. Every price list in this book charges output tokens about five times input, and every latency you have felt splits into a wait-for-first-token and a drip-of-tokens-after. Both facts have the same cause: a transformer processes your prompt in one parallel, compute-limited pass (prefill), then generates output one token at a time, each step re-reading every weight from memory (decode). The lab prices both phases on one honest skeleton (a 7B model on an A100): prefill moves ~11,000 tokens/s while batch-1 decode ceilings at ~146 tokens/s, a 77x gap in machine time per token, and the decode side stays memory-bound until ~77 requests share the weight read. The same arithmetic explains why prompt caching is a latency feature, why cutting output helps twice, and why the serving world's favorite tricks (continuous batching, speculative decoding) all attack the decode side. A seeded Monte Carlo of speculative decoding closes the chapter, matching the paper's closed form to three decimals.
Contents
- Two phases, two machines
- The lab: one model, one GPU, both phases priced
- Reading the numbers
- What the serving world does about decode
- What you can do about it from the client
- Further reading
- Takeaways
Chapter 2 took the 5x output premium as a given and Chapter 43 toured the systems built around it. This chapter derives it, because the derivation changes how you optimize: once you see that input and output tokens are physically different workloads, half this book's advice stops being rules and becomes arithmetic.
Two phases, two machines
When your request arrives, the model does two different jobs:
- Prefill. Every prompt token is embedded and pushed through the network at once: one enormous batched matrix multiply per layer. The GPU's tensor cores are saturated; the limit is FLOPs. This phase produces the KV cache (Chapter 6) and the first output token; its duration is your time to first token (TTFT).
- Decode. Every subsequent token depends on the one before it, so generation is sequential. Each step is a matrix-vector multiply that must stream all model weights (plus the growing KV cache) from HBM into the compute units to produce a single token. The limit is memory bandwidth; the tensor cores mostly idle. Its pace is your time per output token (TPOT), the drip you watch during streaming.
The rule-of-thumb arithmetic is old and sturdy: a dense transformer's forward pass costs about 2 FLOPs per parameter per token (from the scaling-laws literature), and a fp16 model occupies 2 bytes per parameter, so decode at batch 1 does ~2 FLOPs per byte moved while an A100 can do ~153 FLOPs per byte moved. That two-orders-of-magnitude mismatch, arithmetic intensity far below the machine's balance point, is the entire story of LLM serving economics.
Don't be confused. "Output is slower because the model thinks harder about what to say" is folklore; per token, prefill and decode do the same FLOPs. The difference is parallelism: prefill amortizes one weight-read over thousands of tokens, decode spends one weight-read per token. Output tokens are not smarter, they are lonelier, and loneliness is what you pay for.
The lab: one model, one GPU, both phases priced
"""Prefill vs decode: the arithmetic under the 5x output price.
An API bill splits tokens into input and output and prices output about five
times higher. This lab derives why from first principles, with one model and
one accelerator:
PREFILL every prompt token is processed in PARALLEL in one pass; the GPU
multiplies big matrices and is limited by COMPUTE (FLOPs)
DECODE output tokens are generated ONE AT A TIME; every step must re-read
the entire weight matrix from memory, so at small batch the GPU is
limited by MEMORY BANDWIDTH and its arithmetic units mostly idle
The rule of thumb for a dense transformer forward pass is ~2 FLOPs per
parameter per token (Kaplan et al., 2020). The accelerator here is an A100
80GB SXM: 312 TFLOPS dense BF16, 2,039 GB/s of HBM bandwidth (NVIDIA
datasheet). The model is a 7B in fp16 (14 GB of weights). Real serving stacks
add KV-cache traffic, attention FLOPs, and imperfect utilization; this is the
skeleton those corrections hang on.
Also included: a seeded Monte Carlo of speculative decoding checked against
the closed form from Leviathan et al. (2023). NumPy only.
Run: python3 prefill_decode.py
"""
import numpy as np
P = 7e9 # parameters
BYTES = 2 * P # fp16 weights resident in HBM
FLOPS_TOK = 2 * P # forward-pass FLOPs per token (the 2N rule)
PEAK_FLOPS = 312e12 # A100 BF16 dense
PEAK_BW = 2039e9 # A100 HBM bandwidth, bytes/s
MFU = 0.5 # assumed utilization of peak compute in prefill
print("=== The model and the machine ===")
print(f"model: 7B fp16 -> {BYTES / 1e9:.0f} GB of weights, {FLOPS_TOK / 1e9:.0f} GFLOPs/token")
print(f"A100: {PEAK_FLOPS / 1e12:.0f} TFLOPS BF16, {PEAK_BW / 1e9:.0f} GB/s HBM")
print(f"machine balance point: {PEAK_FLOPS / PEAK_BW:.0f} FLOPs per byte moved")
# ---------------------------------------------------------------------------
# Prefill: compute-bound, parallel.
# ---------------------------------------------------------------------------
prefill_rate = PEAK_FLOPS * MFU / FLOPS_TOK
print(f"\n=== Prefill: parallel, compute-bound ===")
print(f"throughput at {MFU:.0%} utilization: {prefill_rate:,.0f} tokens/s")
print(f"{'prompt':>10} {'time to first token':>21}")
for prompt in (2_000, 30_000, 200_000, 960_000):
print(f"{prompt:>10,} {prompt / prefill_rate:>20.1f}s")
print("(a cached prefix skips its share of this: prompt caching is a LATENCY")
print(" feature first, which is why TTFT collapses on warm calls)")
# ---------------------------------------------------------------------------
# Decode: bandwidth-bound at small batch.
# ---------------------------------------------------------------------------
print("\n=== Decode: sequential, bandwidth-bound ===")
print("every step re-reads all weights; a batch shares that read.")
print(f"{'batch':>6} {'bytes/step':>11} {'limited by':>11} {'tok/s total':>12} {'tok/s/user':>11}")
for B in (1, 8, 32, 128, 512):
t_mem = BYTES / PEAK_BW # weight read, shared by the batch
t_cmp = B * FLOPS_TOK / (PEAK_FLOPS * MFU)
t = max(t_mem, t_cmp)
lim = "memory" if t_mem >= t_cmp else "compute"
print(f"{B:>6} {BYTES / 1e9:>9.0f}GB {lim:>11} {B / t:>12,.0f} {1 / t:>11,.0f}")
au = FLOPS_TOK / BYTES
print(f"decode arithmetic intensity at batch 1: {au:.0f} FLOPs/byte, vs the")
print(f"machine's {PEAK_FLOPS / PEAK_BW:.0f}: the tensor cores idle until batch ~{PEAK_FLOPS * MFU / PEAK_BW * BYTES / FLOPS_TOK:.0f}")
# ---------------------------------------------------------------------------
# The price ratio, from machine time.
# ---------------------------------------------------------------------------
t_in = 1 / prefill_rate # machine-seconds per input token
t_out = BYTES / PEAK_BW # per output token at batch 1
print("\n=== Why output costs more ===")
print(f"machine time, one input token (prefill): {t_in * 1e6:>7.1f} us")
print(f"machine time, one output token (batch 1): {t_out * 1e6:>7.1f} us ({t_out / t_in:,.0f}x)")
print("batching compresses that gap but never closes it; the uniform 5x")
print("output premium across Claude's price list is the commercial echo.")
# ---------------------------------------------------------------------------
# Speculative decoding: simulate, then check the closed form.
# ---------------------------------------------------------------------------
print("\n=== Speculative decoding: draft gamma tokens, verify in one pass ===")
rng = np.random.default_rng(0)
GAMMA, N = 4, 200_000
print(f"{'accept rate':>12} {'E[tok/pass] sim':>16} {'closed form':>12} {'ceiling':>9}")
for alpha in (0.60, 0.75, 0.90):
draws = rng.random((N, GAMMA)) < alpha
accepted = np.where(draws.all(axis=1), GAMMA,
np.argmin(draws, axis=1)) # tokens kept per pass
sim = (accepted + 1).mean() # +1 from the verify pass
closed = (1 - alpha ** (GAMMA + 1)) / (1 - alpha)
print(f"{alpha:>12.2f} {sim:>16.3f} {closed:>12.3f} {sim:>8.2f}x")
print("""
Lessons: input tokens are a matrix multiply you do once; output tokens are a
14 GB memory scan you repeat per token, which is why the two are different
products with different prices, why cutting output helps latency twice as
hard as cutting input, and why the serving world's tricks are all about the
decode side: batch it (continuous batching), skip it (speculative decoding,
several tokens per weight-scan), or shrink what each step reads (quantized
and compressed KV, chapter 43's lossy branch).""")
Verified output:
=== The model and the machine ===
model: 7B fp16 -> 14 GB of weights, 14 GFLOPs/token
A100: 312 TFLOPS BF16, 2039 GB/s HBM
machine balance point: 153 FLOPs per byte moved
=== Prefill: parallel, compute-bound ===
throughput at 50% utilization: 11,143 tokens/s
prompt time to first token
2,000 0.2s
30,000 2.7s
200,000 17.9s
960,000 86.2s
(a cached prefix skips its share of this: prompt caching is a LATENCY
feature first, which is why TTFT collapses on warm calls)
=== Decode: sequential, bandwidth-bound ===
every step re-reads all weights; a batch shares that read.
batch bytes/step limited by tok/s total tok/s/user
1 14GB memory 146 146
8 14GB memory 1,165 146
32 14GB memory 4,661 146
128 14GB compute 11,143 87
512 14GB compute 11,143 22
decode arithmetic intensity at batch 1: 1 FLOPs/byte, vs the
machine's 153: the tensor cores idle until batch ~77
=== Why output costs more ===
machine time, one input token (prefill): 89.7 us
machine time, one output token (batch 1): 6866.1 us (77x)
batching compresses that gap but never closes it; the uniform 5x
output premium across Claude's price list is the commercial echo.
=== Speculative decoding: draft gamma tokens, verify in one pass ===
accept rate E[tok/pass] sim closed form ceiling
0.60 2.303 2.306 2.30x
0.75 3.051 3.051 3.05x
0.90 4.096 4.095 4.10x
Lessons: input tokens are a matrix multiply you do once; output tokens are a
14 GB memory scan you repeat per token, which is why the two are different
products with different prices, why cutting output helps latency twice as
hard as cutting input, and why the serving world's tricks are all about the
decode side: batch it (continuous batching), skip it (speculative decoding,
several tokens per weight-scan), or shrink what each step reads (quantized
and compressed KV, chapter 43's lossy branch).
The model is deliberately a skeleton: real serving adds attention FLOPs, KV-cache reads that grow with context, imperfect utilization, and multi-GPU sharding. None of those corrections changes the shape; they mostly make decode worse relative to prefill, because the KV read grows with every token of context while the weight read stays constant.
Reading the numbers
- TTFT is prefill, and prefill is linear in the prompt. 0.2 seconds at 2k tokens, 86 at 960k on this skeleton. This is the second, less-advertised reason prompt caching matters: a cache hit skips the cached prefix's share of prefill compute as well as its bill. The autopsy's 960k-token session (Chapter 44) was interactively usable only because 97.5% of its prompt tokens never re-ran prefill.
- Batch-1 decode wastes 99% of the machine. 1 FLOP per byte against a 153 FLOP/byte machine: the tensor cores are idle 99% of every step. Nobody serves at batch 1; providers pool your request with dozens of others so the 14 GB weight-read is shared. That pooling is invisible to you except as the difference between your per-user 146 tokens/s ceiling and the price you actually pay.
- The 77x is the physics; the 5x is the price. A batch-1 output token occupies the machine 77 times longer than a prefill token; production batching compresses the realized gap toward the batch crossover (~77 concurrent decodes on this skeleton). What survives commercially is a uniform 5x output premium across Claude's entire price list, Haiku to Fable. The exact multiple is a business choice; that output must carry a large premium is arithmetic.
- The speculative table is the honest ceiling. With a 90% acceptance rate and four draft tokens, one full-model pass yields 4.1 tokens on average, and the Monte Carlo agrees with the paper's closed form to three decimals. That is a 4x attack on the weight-read-per-token problem, at zero quality cost by construction (the verify step accepts exactly what the big model would have sampled).
What the serving world does about decode
Three families, all decode-side, all from Chapter 43's cast:
- Batch it: continuous batching. Orca (OSDI 2022) introduced iteration-level scheduling: instead of batching whole requests (and idling while the longest one finishes), admit and retire requests every decode step. vLLM and SGLang schedule this way; it is why provider throughput survives wildly mixed request lengths.
- Skip it: speculative decoding. Leviathan, Kalman, and Matias (ICML 2023) and Chen et al. at DeepMind (2023) published the same trick concurrently: a small draft model proposes several tokens, the big model verifies them in one parallel pass (a mini-prefill), and rejection sampling keeps the output distribution exactly the target model's. Medusa (extra decoding heads instead of a draft model) and EAGLE (feature-level drafting; the current state of the art in the family) refine it. From the client you never see any of this except as speed: several tokens per weight-scan.
- Shrink it: the lossy KV branch. Every decode step also reads the KV cache, which at long context can rival the weights; StreamingLLM, H2O, SnapKV, and KIVI (Chapter 43) exist to cut that read. This is also the arithmetic behind GQA and MLA's KV reductions: smaller cache, faster decode, longer affordable context.
What you can do about it from the client
The physics assigns your levers to phases, which is the practical payoff of this chapter:
| Symptom | Phase | Lever |
|---|---|---|
| Slow to start responding | prefill | shorter prompt (Ch 3, Ch 5); warm cache (Ch 6): a hit skips that share of prefill |
| Slow while responding | decode | fewer output tokens (Ch 4): terser instructions, schemas, effort dial; there is no cache for output |
| Expensive overall | both | output cuts pay 5x per token; input cuts pay 1x but compound per turn (Ch 2) |
Two non-obvious corollaries worth keeping. First, streaming does not make decode faster, it only shows you TPOT honestly; if the drip is too slow, the fix is fewer tokens or a smaller model, not a different API shape. Second, output brevity is a latency optimization even when cost is irrelevant: a 400-token answer arrives ~3 seconds sooner than an 800-token one at typical decode speeds, which for an agent in a loop (Chapter 22) multiplies by every turn.
Remember. Input tokens ride together; output tokens travel alone. Everything about LLM serving economics (the 5x, the TTFT/TPOT split, caching's latency dividend, the existence of speculative decoding) falls out of that one sentence, and your levers sort cleanly by which phase they touch.
Further reading
- Kaplan et al., "Scaling Laws for Neural Language Models" (2020): the source of the 2-FLOPs-per-parameter-per-token accounting used here.
- Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models" (OSDI 2022): continuous batching.
- Leviathan, Kalman, Matias, "Fast Inference from Transformers via Speculative Decoding" (ICML 2023) and Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling" (2023): the two independent speculative-decoding papers; the expected-tokens formula the lab verifies is theirs.
- Chapter 8 and Chapter 43: the serving systems that implement all of this.
Takeaways
- Prefill processes the whole prompt in parallel and is compute-bound; decode emits one token per full weight-read and is memory-bandwidth-bound until large batch. Same FLOPs per token, 77x apart in machine time on the lab's skeleton.
- TTFT is prefill (linear in prompt length; caching skips the cached share), TPOT is decode (fixed by bandwidth and batch; nothing caches it).
- The uniform 5x output premium on Claude's price list is the commercial echo of decode being the scarce phase; treat output tokens as the expensive, slow resource in every design decision.
- The serving stack attacks decode three ways: continuous batching shares the weight-read, speculative decoding gets ~2 to 4x tokens per pass (the lab's Monte Carlo matches the closed form), and the lossy KV branch shrinks what each step reads.
- Client-side: prompt work fixes the start, output work fixes the drip, and output cuts pay five times per token plus a latency dividend on every agent turn.
👉 Prefill priced the prompt's length in seconds; the next question is how models came to accept prompts that long at all. The answer is a geometry trick: position as rotation, and three generations of increasingly careful ways to stretch it. Continue to How the window got long.