Design inference serving on one GPU under a latency SLA

"You have one A100. Serve an 8B model to as many concurrent users as possible, with a p95 time-to-first-token under 500 ms and streaming output at 30 tokens per second per user."

Step 1: clarify, and split the SLA in two (4 minutes)

LLM serving has two latency metrics and they are governed by different bottlenecks. Saying this first is the whole framing:

TTFT   Time to first token. Dominated by PREFILL: processing the
       whole prompt. Compute-bound: a matrix-matrix multiply over
       all prompt tokens at once.

TPOT   Time per output token. Dominated by DECODE: one token at a
       time, each requiring a full pass over the model weights.
       MEMORY-BANDWIDTH-bound: a matrix-VECTOR multiply, so the
       arithmetic intensity is terrible and the GPU sits idle
       waiting on HBM.

A design that optimises throughput without separating these will hit the token rate and miss TTFT, or vice versa. See prefill vs decode for the underlying roofline argument.

The clarifications:

Model         8B parameters, fp16 -> 16 GB of weights
Hardware      One A100 80GB: 312 TFLOPS fp16 dense,
              2,039 GB/s HBM bandwidth
Prompts       p50 800 tokens, p95 4,000, max 8,000
Outputs       p50 200 tokens, p95 600
SLA           TTFT p95 < 500 ms, TPOT < 33 ms (30 tok/s)
Workload      Interactive chat, so requests arrive continuously
              and outputs stream

The question that changes the answer: "Is this interactive or batch? If some of the traffic is offline summarisation with no latency requirement, I would separate it, because mixing latency-sensitive and throughput-oriented work on one GPU means the batch job's long prefills block the interactive stream."

Step 2: capacity math, from the roofline (5 minutes)

This is where the design is decided, and the arithmetic is the answer.

DECODE, one request at a time
  Every generated token reads ALL model weights from HBM.
  16 GB of weights / 2,039 GB/s = 7.8 ms per token, minimum.
  -> 128 tokens/sec absolute ceiling for a single sequence.
  Compute used: 2 x 8e9 FLOPs = 16 GFLOPs per token,
  in 7.8 ms = 2 TFLOPS out of 312 available.
  *** The GPU is at 0.7% compute utilisation. ***

That number is the entire justification for batching.

DECODE, batch of B
  The SAME weight read serves all B sequences: the weights are
  loaded once and multiplied against B vectors instead of one.
  Time per step stays roughly 7.8 ms until compute becomes the
  limit.
  -> Throughput scales nearly LINEARLY with batch size, for free,
     until either compute saturates or memory runs out.

  At B=64:  64 tokens per 7.8 ms = 8,200 tokens/sec
            Compute: 64 x 16 GFLOPs / 7.8 ms = 131 TFLOPS. Still
            under the 312 roof, so still bandwidth-bound.
  At B=160: ~20,500 tokens/sec, ~328 TFLOPS -> now compute-bound.

PREFILL
  A 800-token prompt: 2 x 8e9 x 800 = 12.8 TFLOPs
  At ~50% MFU on 312 TFLOPS -> ~82 ms. Compute-bound, as expected.
  A 4,000-token prompt: ~410 ms. This ALONE nearly exhausts the
  500 ms TTFT budget, which is the constraint that forces chunked
  prefill in step 4.

MEMORY, and what actually limits batch size
  Weights                                    16 GB
  KV cache per token (8B model, GQA 8 kv-heads, 32 layers,
     head_dim 128, fp16):
     2 (K and V) x 32 layers x 8 heads x 128 x 2 bytes = 128 KB/token
  A sequence at 800 prompt + 200 output = 1,000 tokens
     -> 128 MB of KV cache PER SEQUENCE
  Available for KV: 80 - 16 - ~4 (activations, fragmentation) = 60 GB
  -> 60 GB / 128 MB = ~468 concurrent sequences by memory

  WITHOUT grouped-query attention (32 kv-heads instead of 8):
     512 KB/token -> 512 MB/sequence -> only ~117 sequences.
  *** GQA quadruples the achievable batch size. ***

Three conclusions fall straight out of the arithmetic, and stating them in this order is the answer to the question:

A single sequence uses 0.7 percent of the GPU's compute. Batching is not an optimisation, it is the difference between a usable system and a toy.
Memory, not compute, limits concurrency, and the KV cache is the memory. Which is why GQA and paged allocation are the two highest-leverage architectural facts.
A 4,000-token prefill takes about 410 ms on its own, which nearly exhausts the TTFT budget and means prefill scheduling is a first-class concern.

Step 3: continuous batching

Static batching is the naive design and its failure is quantifiable.

STATIC: collect B requests, run them together, return when ALL finish.
  Batch of 8, output lengths [20, 45, 60, 800, 30, 25, 90, 40].
  Every slot is occupied until the 800-token sequence finishes.
  Total useful tokens: 1,110. Slots x steps: 8 x 800 = 6,400.
  *** GPU utilisation: 17%. ***

CONTINUOUS (iteration-level scheduling): after EVERY decode step,
finished sequences leave and waiting ones join.
  Slot 3 finishes at step 20 -> a new request starts at step 21.
  Utilisation approaches 100% as long as the queue is non-empty.

def serve_loop(scheduler, model):
    running = []                       # sequences currently decoding
    while True:
        # 1. Admit new work if memory allows. This runs EVERY step,
        #    which is what "continuous" means.
        while scheduler.has_waiting() and kv_cache.can_admit(
                scheduler.peek(), running):
            running.append(scheduler.pop())

        if not running:
            continue

        # 2. One decode step for the whole batch. The weights are
        #    read once for all sequences: that is where the win is.
        logits = model.decode_step(running)

        # 3. Sample, append, and evict anything finished.
        for seq, tok in zip(running, sample(logits)):
            seq.append(tok)
            if seq.is_done():
                kv_cache.free(seq)
                seq.respond_complete()
        running = [s for s in running if not s.is_done()]

Continuous batching typically delivers a several-fold throughput improvement over static batching at the same latency, and the reason is entirely in the utilisation arithmetic above rather than in any kernel optimisation.

Step 4: the prefill/decode interference problem

This is the deep dive and the thing that separates a real answer.

A 4,000-token prefill takes about 410 ms of pure compute. While it runs, every decoding sequence stalls, because they share the GPU. Sixty-four users streaming at 30 tokens per second each see a 410 ms gap, which is 12 missed tokens and a visibly stuttering stream.

Timeline without chunked prefill:
  step  decode  decode  decode  [ PREFILL 410 ms ]  decode  decode
  user             smooth                stall              smooth
                                     12 tokens missed

Three mitigations, and the design uses the first two:

1. Chunked prefill. Split a long prefill into fixed-size chunks and interleave them with decode steps.

CHUNK = 512     # tokens of prefill per scheduling step

def schedule_step(waiting, running, budget_tokens=1024):
    # Fill the step's token budget with decode first (they are cheap
    # per token and latency-critical), then top up with prefill chunks.
    batch = [(s, 1) for s in running]            # 1 token each
    used = len(running)
    for seq in waiting:
        if used >= budget_tokens:
            break
        take = min(CHUNK, budget_tokens - used, seq.remaining_prefill)
        batch.append((seq, take))
        used += take
    return batch

Now the 4,000-token prefill is 8 chunks of 512, each roughly 50 ms, interleaved with decode steps. TTFT for that request rises slightly; TPOT for everyone else stays smooth. That is the correct trade for interactive serving, and it is what Sarathi-Serve demonstrated and what vLLM and TensorRT-LLM now implement.

2. Prefill/decode disaggregation, when you have more than one GPU: dedicate GPUs to prefill and others to decode, shipping the KV cache between them. Eliminates the interference entirely at the cost of KV transfer bandwidth. Not available here, since the premise is one GPU, and worth naming as what you would do with two.

3. Prioritise by remaining work, so a request that has been waiting does not starve behind a stream of new prefills.

Step 5: PagedAttention, and why fragmentation was the hidden cost

The naive KV cache allocates contiguously for the maximum possible sequence length.

Request declares max_tokens=2048. Allocate 2048 x 128 KB = 256 MB.
It generates 60 tokens and finishes.
Used: 7.5 MB. Reserved: 256 MB.
*** 97% of the allocation was waste. ***

Measured across a real workload, naive allocation wastes 60-80% of
KV memory to internal fragmentation and over-reservation.

PagedAttention (Kwon et al., 2023) applies operating-system virtual memory to the KV cache: fixed-size blocks (typically 16 tokens), a per-sequence block table, and physical blocks allocated on demand and non-contiguously.

Sequence A: logical blocks [0,1,2]  -> physical [7, 3, 19]
Sequence B: logical blocks [0,1]    -> physical [4, 11]

Allocation is per-block on demand. Waste is bounded by the last
partial block: at most 15 tokens per sequence, so under 4% instead
of 60-80%.

The second benefit is sharing. Sequences with a common prefix (the same system prompt, the same few-shot examples, parallel samples from one prompt) share physical blocks with copy-on-write.

100 concurrent requests, all with the same 2,000-token system prompt:
  Without sharing: 100 x 2,000 x 128 KB = 25.6 GB of KV cache
  With prefix sharing: 1 x 2,000 x 128 KB = 256 MB
  *** 25 GB freed, which is 200 more concurrent sequences. ***

For a production system with a large stable system prompt, prefix sharing is a larger win than any batching change, and it is worth volunteering because most candidates stop at continuous batching.

Step 6: putting the numbers together

Configuration
  8B model, fp16, GQA with 8 kv-heads
  PagedAttention, 16-token blocks, prefix sharing enabled
  Continuous batching with chunked prefill (512-token chunks,
    1,024-token per-step budget)
  KV memory available: 60 GB

Concurrency
  Average sequence 1,000 tokens x 128 KB = 128 MB
  Shared 2,000-token system prompt across all -> ~256 MB once
  -> ~460 concurrent sequences by memory

Throughput at batch 64 (a typical steady-state running set)
  Decode step: ~7.8 ms bandwidth-bound
  64 tokens / 7.8 ms = 8,200 tokens/sec aggregate

Per-user token rate
  8,200 / 64 = 128 tokens/sec per user. SLA is 30. Comfortable.
  -> We can push the batch higher. At B=192:
     decode step lengthens as compute saturates, ~12 ms
     192 / 12 ms = 16,000 tokens/sec, 83 tokens/sec/user.
     Still above the 30 tok/s SLA.

Where the SLA actually binds
  At B=460 (memory limit), step time is compute-bound at ~25 ms
  -> 40 tokens/sec/user. Approaching the 30 tok/s floor.
  *** So the binding constraint is TPOT, not memory. ***
  Set the admission limit to ~350 concurrent, leaving headroom.

TTFT check
  p50 prompt 800 tokens = 2 chunks of 512 -> ~2 scheduling steps
  of prefill, ~100 ms plus queueing. Within 500 ms.
  p95 prompt 4,000 tokens = 8 chunks -> ~410 ms of prefill spread
  over 8 steps, plus decode interleaving -> ~500-600 ms.
  *** p95 TTFT is at or slightly over budget. ***
  Fix: give prefill a larger share of the token budget when the
  TTFT queue is deep, accepting a small TPOT dip. This is an
  adaptive scheduler, not a static one.

Ending with a constraint that is marginally violated and then fixing it is stronger than ending with everything comfortably within budget, because it demonstrates the arithmetic was real rather than reverse-engineered.

Step 7: failure modes and degradation

Memory exhaustion mid-generation
  -> A running sequence needs a new KV block and none is free.
     Options: PREEMPT the newest sequence (recompute its prefill
     later, cheap for short prompts) or SWAP its KV cache to host
     memory (avoids recompute, costs PCIe transfer).
     vLLM implements both. Preempt-and-recompute is usually better
     for short prompts, swap for long ones.
     Never OOM the process: preemption must be a designed path.

Prompt longer than the context window
  -> Reject at admission, before allocating anything.

One user requests 8,000 output tokens
  -> They occupy a slot for ~4 minutes. Cap max_tokens per request
     and enforce a fair-share policy, or one user degrades the
     batch for everyone.

Queue grows unboundedly
  -> Admission control with a queue depth limit and a 429. Queueing
     forever converts a throughput problem into a timeout problem
     and the client has already given up.

Model load / restart
  -> 16 GB of weights from disk is 30-120 s. Keep a warm standby
     or accept the outage window explicitly; do not discover it
     during an incident.

Traffic mix shifts to long prompts
  -> Prefill share rises, decode throughput falls, TPOT degrades
     for everyone. Monitor the prefill:decode token ratio as a
     leading indicator, not just aggregate throughput.

The degradation ladder: reduce the max batch size (protects TPOT for admitted users), then cap max_tokens per request, then shed at admission with a 429 and a Retry-After. Degrading TPOT for everyone is worse than rejecting some requests, because a stream at 10 tokens per second reads as broken.

Step 8: what changes at ten times the scale

At ten A100s and 10x the traffic:

Prefill/decode disaggregation becomes available and worth it. Dedicate two GPUs to prefill and eight to decode, transferring KV cache over NVLink. This removes the interference entirely rather than mitigating it, and it lets each pool be tuned for its own bottleneck: prefill wants compute, decode wants bandwidth.

Routing becomes prefix-aware. Route requests sharing a system prompt to the same replica so the prefix cache hits. That is a meaningful throughput multiplier and it makes the load balancer part of the inference design rather than a commodity.

Quantisation enters the trade space. fp8 or int8 weights halve the bandwidth per decode step, so the 7.8 ms floor becomes roughly 4 ms and per-sequence throughput doubles, at a measurable quality cost that has to be evaluated rather than assumed.

Speculative decoding becomes worthwhile. A small draft model proposes several tokens and the large model verifies them in one forward pass. It converts memory-bound decode into compute-bound verification, which is exactly the direction that helps, and it is most effective at low batch sizes where the GPU is idle anyway.

Production evidence

Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP 2023) is the vLLM paper. It documents the 60 to 80 percent KV memory waste under naive allocation and the throughput gains from paged allocation plus prefix sharing.

Yu et al., "Orca: A Distributed Serving System for Transformer-Based Generative Models" (OSDI 2022) introduced iteration-level (continuous) batching and is the primary source for scheduling at the step rather than the request boundary.

Agrawal et al., "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve" (OSDI 2024) is the chunked-prefill reference, documenting the prefill/decode interference and quantifying the TPOT improvement from interleaving.

Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (2023) is the grouped-query attention paper, and the KV-cache size reduction is why modern models ship with it.

Dao et al., FlashAttention and FlashAttention-2, for the IO-aware attention kernels that make the prefill compute estimate achievable rather than theoretical.

NVIDIA's A100 specifications (312 TFLOPS fp16 dense, 2,039 GB/s HBM2e on the 80GB model) are the basis for every number in step 2, and quoting the bandwidth figure is what makes the roofline argument concrete.

The debate

The case for maximising batch size: throughput scales nearly linearly with batch while bandwidth-bound, so a larger batch is close to free revenue. Under-batching leaves a GPU running at single-digit utilisation, which at A100 prices is the most expensive idle hardware in the building.

The case for capping batch size: past the bandwidth-bound region, step time grows and TPOT degrades for every user in the batch. A stream at 10 tokens per second reads as broken regardless of how good the aggregate throughput number looks on a dashboard.

The case for separate pools: run interactive traffic and batch traffic on different GPUs, so long prefills never interfere with streaming. Clean, and it means each pool is under-utilised some of the time.

My position: continuous batching with chunked prefill, an admission limit derived from the TPOT SLA rather than from memory, and prefix sharing enabled.

The admission limit is the decision worth defending. The obvious limit is memory (about 460 sequences here), and the correct limit is the batch size at which the decode step still delivers 30 tokens per second per user, which in this configuration is around 350. Sizing to memory rather than to the SLA is the common mistake, and it produces a system that reports excellent throughput while every user experiences a stuttering stream.

Chunked prefill I would treat as mandatory for interactive serving, because the alternative is a 410 ms stall for every streaming user each time a long prompt arrives. It costs a little TTFT and it protects the metric users actually perceive, which is smoothness rather than aggregate tokens per second.

And prefix sharing is the largest single win available in most real deployments and it is the one candidates omit. A 2,000-token shared system prompt across 100 concurrent requests is 25 GB of KV cache without sharing and 256 MB with it. That is 200 additional concurrent sequences from a memory-management change, larger than anything the batching scheduler contributes.

Where I would push back: if part of the traffic is offline and latency-insensitive, do not mix it. Running batch summarisation on the same GPU as interactive chat means the batch job's long prefills degrade the interactive stream, and separating them is cheaper than any scheduler sophistication.

Follow-up Q&A

"Why does batching help so much?" Because decode is memory-bandwidth-bound, not compute-bound. Generating one token requires reading all 16 gigabytes of weights from HBM, which at 2 terabytes a second is 7.8 milliseconds, and the arithmetic in that step is 16 GFLOPs against 312 TFLOPS available. So a single sequence uses under one percent of the GPU's compute. Batching amortises the same weight read across many sequences, so throughput scales nearly linearly with batch size for free until compute saturates.

"What actually limits the batch size?" Memory, specifically the KV cache, until the latency SLA binds first. At 128 kilobytes per token for this model with grouped-query attention, a thousand-token sequence is 128 megabytes, so 60 gigabytes of available KV memory holds about 460 sequences. But at that batch size the decode step is compute-bound at around 25 milliseconds, which is 40 tokens per second per user against a 30 token floor. So I would set admission at around 350, and the point is that the limit comes from the SLA rather than from memory. Sizing to memory is the common mistake and it produces great throughput numbers with a stuttering user experience.

"What is continuous batching and why is static so bad?" Static batching collects a batch, runs it, and returns when all sequences finish, so a batch with outputs of 20, 45 and 800 tokens keeps every slot occupied for 800 steps. That is around 17 percent utilisation. Continuous batching schedules at the iteration level: after every decode step, finished sequences leave and waiting ones join, so utilisation approaches 100 percent while the queue is non-empty. Typically several times the throughput at the same latency.

"A 4,000-token prompt arrives. What happens to everyone else?" Without chunked prefill, they stall for about 410 milliseconds, which at 30 tokens a second is 12 missed tokens and a visibly stuttering stream. Chunked prefill splits it into eight 512-token chunks and interleaves them with decode steps, so the prefilling request's TTFT rises slightly and everyone else's token rate stays smooth. For interactive serving that is clearly the right trade, because smoothness is what users perceive.

"What is PagedAttention actually solving?" Fragmentation and over-reservation. Naive allocation reserves contiguous KV memory for the declared maximum length, so a request declaring 2,048 tokens that generates 60 wastes 97 percent of its allocation. Measured across a real workload that is 60 to 80 percent of KV memory wasted. Paged allocation uses fixed 16-token blocks with a per-sequence block table, so waste is bounded by the last partial block, under 4 percent. And it enables prefix sharing with copy-on-write.

"How big is prefix sharing?" Often the largest single win, and it is the one people omit. A hundred concurrent requests sharing a 2,000-token system prompt is 25 gigabytes of KV cache without sharing and 256 megabytes with it. That 25 gigabytes is roughly 200 additional concurrent sequences, which is more than any scheduling change delivers. For a production RAG or agent system with a large stable prompt, it is the first thing I would verify is enabled.

"You run out of KV memory mid-generation. What do you do?" Preempt, as a designed path, never OOM. Two options: evict the newest sequence and recompute its prefill later, which is cheap for short prompts, or swap its KV cache to host memory, which avoids recomputation and costs PCIe transfer time. vLLM implements both. Preempt-and-recompute is generally better for short prompts and swapping for long ones, and having neither means the process dies under exactly the load it was built for.

"What would you do with a second GPU?" Prefill/decode disaggregation. Dedicate one to prefill and one to decode and ship the KV cache between them, which eliminates the interference entirely rather than mitigating it with chunking. It also lets each pool be tuned for its own bottleneck, since prefill wants compute and decode wants bandwidth. With one GPU that is not available, which is why chunked prefill is the answer here.

"How would you monitor this?" TTFT and TPOT separately at the p95, never a combined latency number, because they have different bottlenecks and a single metric hides which one moved. Then batch size and KV cache utilisation, preemption rate, and the prefill-to-decode token ratio, which is the leading indicator: if the traffic mix shifts toward long prompts, prefill's share of the token budget rises and TPOT degrades for everyone before aggregate throughput shows anything.

Common misconceptions

"Bigger batches are always better." Throughput scales while bandwidth-bound. Past that, step time grows and every user's token rate falls.

"The bottleneck is compute." For decode it is HBM bandwidth, and a single sequence uses under one percent of the GPU's FLOPS.

"KV cache size is a detail." It is what limits concurrency, which is why GQA quadruples achievable batch size and why paged allocation exists.

"TTFT and TPOT are the same problem." They are governed by prefill and decode respectively, which are compute-bound and bandwidth-bound. Optimising one can worsen the other, which is exactly the chunked-prefill trade.

"Continuous batching is a kernel optimisation." It is a scheduling change at the iteration boundary, and its benefit comes entirely from utilisation arithmetic.

Interview delivery note

Split the SLA first, because it establishes the roofline framing: "There are two latency metrics here and they have different bottlenecks. Time to first token is prefill, which is compute-bound, a matrix-matrix multiply over the whole prompt. Time per output token is decode, which is memory-bandwidth-bound, because every token requires reading all the weights from HBM. Optimising one can worsen the other."

Then do the arithmetic that justifies everything: "Sixteen gigabytes of weights at two terabytes a second is 7.8 milliseconds per token, so 128 tokens a second is the ceiling for a single sequence. And the compute used in that step is 2 TFLOPS out of 312. The GPU is at 0.7 percent utilisation, which is why batching isn't an optimisation, it's the difference between a system and a toy."

Volunteer the interference problem, since it is what separates a real answer: "The thing I would design for specifically is prefill blocking decode. A 4,000-token prompt is about 410 milliseconds of compute, and while it runs every streaming user stalls, which at 30 tokens a second is 12 missed tokens and a visibly stuttering stream. So chunked prefill: 512-token chunks interleaved with decode steps. The prefilling request's TTFT rises slightly and everyone else stays smooth."

The line most candidates miss: "and I'd check prefix sharing before anything else. A hundred requests sharing a 2,000-token system prompt is 25 gigabytes of KV cache without it and 256 megabytes with it. That's 200 more concurrent sequences from a memory management change, which is bigger than anything the scheduler contributes."

Close on the admission limit, because it is the decision: "And I'd set the admission limit from the TPOT SLA rather than from memory. Memory allows about 460 sequences; at that batch the step is compute-bound and users get 40 tokens a second against a 30 token floor, so I'd admit around 350. Sizing to memory gives you a great throughput number and a stuttering user experience."

The Staff & Team Lead Interview Reference