Sampling strategies, with demonstrated outputs

What it is

A language model does not produce text. It produces a probability distribution over the vocabulary at each step, and a sampling strategy turns that distribution into a chosen token. The model is the same in every case; the sampler decides what you get from it.

Prompt: "The capital of France is"

Model output (logits -> softmax over 128,256 tokens):
  " Paris"     0.9124
  " the"       0.0312
  " located"   0.0189
  " a"         0.0104
  " known"     0.0071
  ... 128,251 more, summing to 0.0200

The samplers, and what each does to that distribution:

Strategy	Rule	Determinism
Greedy	Always take the argmax	Deterministic
Temperature	Divide logits by `T` before softmax	Stochastic
Top-k	Keep the `k` highest, renormalise, sample	Stochastic
Top-p (nucleus)	Keep the smallest set summing to `p`, renormalise	Stochastic
Min-p	Keep tokens with `prob >= min_p x max_prob`	Stochastic
Beam search	Track `n` sequences by cumulative log-probability	Deterministic

What this is confused with: temperature is not a "creativity" knob in any meaningful sense. It is a sharpening or flattening of the distribution. High temperature does not make the model more imaginative; it makes it more likely to pick tokens it thinks are wrong. That distinction matters because it tells you when high temperature is useful (when there are many equally good continuations) and when it is destructive (when there is one right answer).

The problem it solves

Greedy decoding is not obviously wrong and produces bad text. Taking the most likely token at every step gets stuck in loops and produces flat, repetitive output:

Prompt: "Write a short description of a coffee shop."

Greedy (T=0):
"The coffee shop is a place where people can enjoy a cup of coffee. The coffee
shop is a place where people can relax and enjoy a cup of coffee. The coffee
shop is a place where people can enjoy a cup of coffee and relax."

The mechanism is that the highest-probability continuation of a common phrase is often the same common phrase, so the model enters a cycle. The Holtzman et al. finding is that human text is not high-probability text: real writing regularly contains surprising tokens, and always choosing the most likely one produces something recognisably unlike what a person would write.

The opposite failure is equally real:

Same prompt, T=1.8, no truncation:
"The coffee shop nestles amid pigeons whose ceramic laughter, brewing quarterly
against the vinegar hypothesis, tessellates whenever Tuesday admits its
copper..."

At high temperature the long tail of the distribution (tens of thousands of tokens with tiny probabilities that collectively sum to something noticeable) becomes reachable, and sampling from it produces incoherence. Truncation samplers exist to remove the tail before temperature is applied to what remains.

Mechanics

Temperature

$$P(x_i) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$

def apply_temperature(logits, T):
    if T == 0:
        return one_hot(argmax(logits))     # greedy, by convention
    return softmax(logits / T)

Logits: [3.2, 2.8, 1.1, 0.4, -0.2]

T = 0.2:  [0.879, 0.119, 0.000, 0.000, 0.000]   sharpened, nearly greedy
T = 0.7:  [0.487, 0.274, 0.026, 0.010, 0.005]   moderate
T = 1.0:  [0.395, 0.265, 0.048, 0.024, 0.013]   the model's own distribution
T = 1.5:  [0.320, 0.245, 0.081, 0.052, 0.036]   flattened
T = 2.0:  [0.279, 0.228, 0.098, 0.069, 0.052]   very flat

T = 1.0 is the model's actual calibrated distribution. Below 1 you are sharpening (more confident than the model is), above 1 you are flattening (less confident than the model is). Framing it as "creativity" obscures this: at T = 2.0 the fifth-ranked token has a 5.2 percent chance instead of 1.3 percent, and the model ranked it fifth for a reason.

Top-k

def top_k(probs, k=50):
    kth = torch.topk(probs, k).values[-1]
    probs = torch.where(probs < kth, 0.0, probs)
    return probs / probs.sum()

Simple, and its weakness is that k is fixed while the distribution's shape is not:

Confident context: "The capital of France is"
  P(" Paris") = 0.91, everything else negligible.
  top-k=50 keeps 50 tokens, 49 of which are wrong.
  Combined with temperature, one of those 49 can be sampled. Bad.

Uncertain context: "The best programming language is"
  20 languages all plausible, each ~0.03-0.08.
  top-k=50 is roughly right here.

Fixed k is too permissive when the model is confident and can be too restrictive when it is not. That is exactly the problem top-p solves.

Top-p (nucleus sampling)

Keep the smallest set of tokens whose cumulative probability reaches p:

def top_p(probs, p=0.9):
    sorted_probs, idx = torch.sort(probs, descending=True)
    cumulative = torch.cumsum(sorted_probs, dim=-1)
    # Keep everything up to and including the token that crosses p.
    cutoff = (cumulative > p).float().argmax()
    mask = torch.zeros_like(probs, dtype=torch.bool)
    mask[idx[:cutoff + 1]] = True
    probs = torch.where(mask, probs, 0.0)
    return probs / probs.sum()

Confident:   " Paris" 0.91 alone exceeds p=0.9  ->  nucleus size 1. Effectively greedy.
Uncertain:   needs 14 tokens to reach 0.9       ->  nucleus size 14. Diverse.

The nucleus adapts to the model's confidence, which is the property that made top-p the default. It is permissive exactly when the model is uncertain and restrictive exactly when it is not.

Min-p: the newer, better-behaved one

Top-p has a flaw that appears at high temperature: after flattening, the cumulative sum reaches p more slowly, so the nucleus grows and includes tokens the model considered poor.

Min-p keeps tokens whose probability is at least a fraction of the maximum:

def min_p(probs, min_p=0.05):
    threshold = min_p * probs.max()
    probs = torch.where(probs < threshold, 0.0, probs)
    return probs / probs.sum()

Confident:  max = 0.91, threshold = 0.0455  ->  keeps 2-3 tokens
Uncertain:  max = 0.08, threshold = 0.004   ->  keeps ~30 tokens

It scales with the model's confidence directly rather than through a cumulative sum, so it stays sensible at higher temperatures. This is why min-p plus a higher temperature has become a popular combination for creative work: you get genuine diversity without the tail leaking in.

Repetition controls

# Repetition penalty: divide the logit of any token already generated.
for token in set(generated):
    logits[token] /= penalty if logits[token] > 0 else 1/penalty   # 1.05-1.15

# Frequency penalty: proportional to how often it appeared.
logits[token] -= frequency_penalty * count[token]

# Presence penalty: flat, once it has appeared at all.
logits[token] -= presence_penalty * (count[token] > 0)

Repetition penalty above about 1.2 damages fluency badly, because legitimate repeated words (articles, common nouns, a variable name in code) get suppressed. On code it is actively harmful: for and i and return must repeat. Set repetition penalty to 1.0 for code generation and rely on the model.

Beam search, and why it is largely gone

Beam search maintains n candidate sequences ranked by cumulative log-probability. It is standard in machine translation and largely abandoned for open-ended generation, because it optimises for high total probability, and high-probability text is exactly the flat, generic text that greedy produces. It also costs n times the compute. Use it for translation and constrained tasks with a single correct answer; do not use it for open-ended text.

Recommended settings by task

Factual QA / extraction / classification:
    temperature = 0            deterministic, reproducible, no tail risk

Code generation:
    temperature = 0.2, top_p = 0.95, repetition_penalty = 1.0
    (low temperature: syntax has one right answer; no rep penalty: code repeats)

Structured output (JSON):
    temperature = 0 + constrained decoding (see structured output)

Summarisation:
    temperature = 0.3, top_p = 0.9

Conversational:
    temperature = 0.7, top_p = 0.9

Creative writing:
    temperature = 1.0, min_p = 0.05
    (or temperature = 1.2 with min_p, which min-p handles better than top-p)

Generating diverse candidates for reranking:
    temperature = 0.8-1.0, n = 5-20, then rank with a separate model

A note on T = 0 and reproducibility. Greedy decoding is deterministic in principle and not reproducible in practice on a GPU: floating-point reductions are non-associative and their order depends on batch composition, so the same prompt in a different batch can produce a different argmax when two logits are nearly tied. Vendors offering a seed parameter document it as best-effort for the same reason. If you need byte-identical outputs, cache them.

A worked example: three bugs, all sampling parameters

A product with three LLM features, all using one shared client with default settings (temperature=0.7, top_p=0.9) because nobody had revisited them.

Feature 1: extracting structured data from invoices.

Reported: "the extracted totals are sometimes wrong"
Measured: 3.4% of extractions had at least one incorrect numeric field.

Sampling from a distribution where the correct token had 0.94 probability meant a 6 percent chance per token of taking something else, and an invoice has many numeric tokens.

temperature 0.7 -> 0:      error rate 3.4% -> 0.2%

A 17x reduction from one parameter. For extraction there is one correct answer and any stochasticity is pure loss. The residual 0.2 percent was genuine model error on poor scans.

Feature 2: generating SQL from natural language.

Reported: "queries sometimes have syntax errors"
Measured: 8.1% of generated queries failed to parse.

temperature 0.7 -> 0.1:    parse failures 8.1% -> 1.2%
repetition_penalty 1.1 -> 1.0:  parse failures 1.2% -> 0.4%

The repetition penalty was the interesting one. SQL repeats tokens constantly (SELECT, FROM, table aliases, AND), and penalising them was pushing the model toward alternatives that did not parse. A repetition penalty on code is actively harmful and it had been copied from the prose configuration.

Feature 3: generating marketing copy variants.

Reported: "the variants are all basically the same"
Measured: pairwise similarity across 5 generated variants: 0.87 cosine

This was the opposite problem, and the naive fix made it worse:

temperature 0.7 -> 1.4, top_p 0.9:
    similarity 0.87 -> 0.61      (better diversity)
    but: 11% of outputs contained incoherent phrases or invented words

At T = 1.4, top-p's cumulative sum reaches 0.9 more slowly, so the nucleus grew and admitted tokens the model rated poorly. Switching the truncation sampler fixed it:

temperature 1.2, min_p 0.05 (top_p disabled):
    similarity 0.87 -> 0.64      (diversity retained)
    incoherent outputs 11% -> 0.7%

Min-p thresholds relative to the maximum probability, so it stays tight even after flattening. Same temperature range, dramatically fewer failures.

Summary across the three:

feature          before                          after                       key metric
extraction       T=0.7, top_p=0.9                T=0                         3.4% -> 0.2% errors
SQL              T=0.7, top_p=0.9, rep=1.1       T=0.1, top_p=0.95, rep=1.0  8.1% -> 0.4% parse fails
marketing copy   T=0.7, top_p=0.9                T=1.2, min_p=0.05           0.87 -> 0.64 similarity,
                                                                             0.7% incoherent

Three features, three different correct configurations, and they had been sharing one. The shared-client default was the root cause of all three bugs, and the fix in every case was a parameter change with no model change, no prompt change and no cost.

The organisational fix mattered as much as the parameters:

# Sampling config is part of the TASK definition, not a client default.
@dataclass(frozen=True)
class TaskProfile:
    temperature: float
    top_p: float | None = None
    min_p: float | None = None
    repetition_penalty: float = 1.0

EXTRACTION   = TaskProfile(temperature=0.0)
SQL_GEN      = TaskProfile(temperature=0.1, top_p=0.95)
CREATIVE     = TaskProfile(temperature=1.2, min_p=0.05)

Making the sampling profile a named, reviewed property of each task is what stopped a default from silently governing three unrelated features.

Production evidence

Holtzman et al., "The Curious Case of Neural Text Degeneration" (2019) introduced nucleus sampling and demonstrated the core finding: human text is not maximum-probability text, so likelihood-maximising decoding (greedy, beam search) produces degenerate output. The perplexity of human text under the model is substantially higher than that of greedy output, which is the measurement that made the argument.

Min-p (Nguyen et al., 2024) was proposed as a confidence-scaled alternative to top-p and was adopted in llama.cpp, text-generation-webui and vLLM before formal publication, which is a recurring pattern in this area: practitioners find and adopt these first.

OpenAI's API exposes temperature, top_p, frequency_penalty and presence_penalty, and their documentation recommends changing one of temperature or top_p rather than both, since the interaction is hard to reason about. That is good advice and widely ignored.

vLLM's SamplingParams supports the full set including min-p, and its documentation of the interaction order (temperature, then top-k, then top-p, then min-p) is worth knowing because the order determines the result.

Every provider documents temperature=0 as best-effort determinism, not a guarantee. The reason is GPU floating-point non-associativity: reduction order depends on batch composition, so nearly-tied logits can resolve differently across runs. OpenAI's seed parameter and system_fingerprint exist to make the non-determinism observable rather than to remove it.

The debate

Temperature or top-p: which do you tune? Change one. They interact multiplicatively in ways that are hard to reason about, and the standard advice from providers is to tune temperature and leave top_p at 0.9 to 1.0, or tune top_p at temperature 1.0. My position: tune temperature, keep a truncation sampler fixed as a safety net. Temperature maps to an intuition (how much do I trust the model's ranking) and the truncation sampler is doing a different job, removing the tail.

Is top-p or min-p better? Min-p, for anything above T ≈ 1.0, and the mechanism is clear: top-p's cumulative threshold grows the nucleus as the distribution flattens, which is exactly backwards. Below T = 1.0 they behave similarly and top-p has broader support, so it remains the safe default. For creative work at higher temperature, min-p is meaningfully better, and the 11 percent to 0.7 percent incoherence change in the worked example is the kind of difference it makes.

Should anything use temperature > 1.0? Yes, deliberately: generating diverse candidates for reranking. Sample 10 at T = 1.0 with min-p, score them with a separate model or a heuristic, return the best. That is best-of-N and it beats a single low- temperature sample on many tasks, at N times the cost. Raising temperature and returning the first sample is not the same thing and is usually worse.

Is temperature = 0 safe for production? For extraction, classification and anything with a single correct answer, yes, and it is what I would default to. Two caveats. It is not truly reproducible on GPUs, so do not build a system that depends on byte-identical outputs without caching them. And it makes failures consistent: if the model gets a case wrong at T = 0, it gets it wrong every time, where a small temperature would occasionally get it right. That consistency is usually a feature (reproducible bugs) and occasionally hides a fragile prompt behind a lucky argmax.

Do sampling parameters matter compared to prompt and model choice? Less on average and much more in the specific failure cases. The three bugs in the worked example were all sampling-parameter bugs presenting as model quality issues, and all three were fixed with no model or prompt change. The cheapest thing to check when output quality is inconsistent is the sampler, because it is one line and it is frequently inherited from a default that suited a different task.

Follow-up Q&A

"What does temperature actually do?"

Divides the logits before softmax, sharpening the distribution below 1 and flattening it above 1. T = 1.0 is the model's own calibrated distribution. It is not a creativity knob: at T = 2.0 you are making tokens the model ranked poorly substantially more likely, which produces diversity when there are many good continuations and incoherence when there is one right answer. Framing it as sharpening versus flattening tells you immediately which case you are in.

"Top-k or top-p, and why?"

Top-p, because k is fixed while the distribution's shape is not. In a confident context (" Paris" at 0.91), top-k=50 keeps 49 tokens that are all wrong, and combined with temperature one of them can be sampled. Top-p keeps the smallest set summing to p, so the nucleus is 1 token when the model is confident and 14 when it is not. The nucleus adapts to confidence, which is the property that made it the default.

"When would you use min-p over top-p?"

Above roughly T = 1.0. Top-p's cumulative threshold means that as temperature flattens the distribution, the nucleus grows to include tokens the model rated poorly, which is backwards. Min-p thresholds at a fraction of the maximum probability, so it scales with confidence directly and stays tight after flattening. In one measurement, moving from T=1.4, top_p=0.9 to T=1.2, min_p=0.05 kept the diversity and took incoherent outputs from 11 percent to 0.7 percent.

"Why is a repetition penalty bad for code?"

Code repeats tokens by necessity: for, return, i, table aliases, closing brackets. Penalising a token because it appeared already pushes the model toward alternatives that do not parse. In one measurement, removing a 1.1 repetition penalty took SQL parse failures from 1.2 percent to 0.4 percent. Set it to 1.0 for code and rely on the model, which is not prone to the degenerate loops that motivated the penalty for prose.

"Is temperature=0 reproducible?"

Deterministic in principle, not reproducible in practice on a GPU. Floating-point reductions are non-associative and their order depends on batch composition, so two nearly-tied logits can resolve differently depending on what else was in the batch. Providers offer seed as best-effort for exactly this reason. If you need byte-identical outputs, cache them rather than relying on the sampler.

"How would you get diverse outputs without incoherence?"

Best-of-N rather than high temperature on a single sample. Generate 10 at T = 1.0 with min-p, score them with a reranker or a heuristic, return the best. That gives real diversity in the candidate pool and a quality floor from the ranking, at N times the generation cost. Turning temperature up and returning the first sample gives you diversity and no floor, which is why it produces the incoherent-output complaints.

Common misconceptions

"Temperature controls creativity." It sharpens or flattens the distribution. High temperature makes tokens the model ranked poorly more likely, which reads as creativity when many continuations are good and as incoherence when one is right.

"Greedy decoding gives the best output." It gives the highest-probability output, which for open-ended text is flat and repetitive, because human text is not maximum-probability text. That is the whole finding of the nucleus sampling paper.

"Set temperature and top_p both." They interact multiplicatively and providers recommend changing one. Tuning both makes results hard to attribute.

"A repetition penalty prevents repetition, so it is always good." Above about 1.2 it damages fluency, and on code it is actively harmful because code must repeat tokens.

"Sampling parameters are a minor detail." Three production bugs in the worked example were all sampling parameters, all presented as model quality problems, and all were fixed without touching the model or the prompt. It is the cheapest thing to check.

Interview delivery note

Say this verbatim: "Temperature sharpens or flattens the distribution, it is not a creativity knob. T = 0 for anything with one correct answer, because sampling from a distribution where the right token has 0.94 probability means a 6 percent error rate per token. And for creative work I would use min-p rather than top-p above T = 1, because top-p's nucleus grows as the distribution flattens, which is exactly backwards." Mechanism, the default, and the non-obvious refinement.

The senior-versus-staff separator is treating sampling config as a per-task property rather than a client default. A senior engineer knows the parameters and reasonable values. A staff engineer notices that one shared client default was causing three unrelated bugs across three features (extraction errors, SQL parse failures, and insufficiently diverse copy), that all three presented as model quality issues, and that the fix is to make the sampling profile a named, reviewed part of each task definition.

The second signal is the repetition penalty on code. It is a specific, checkable thing that is routinely copied from a prose configuration and measurably harms code generation, and knowing it comes from having debugged it.

The Staff & Team Lead Interview Reference