The decoder's dials: logits and how to control them

TL;DR. A language model does not pick a word. On every step it produces one real number per vocabulary token, a logit, and every decoding control you have ever heard of (temperature, top-k, top-p, min-p, logit bias, banned strings, structured outputs) is a transformation applied to that logit vector before a single token is drawn. This chapter builds the whole pipeline from scratch in NumPy on an 8-word vocabulary, proving with a seeded 4000-draw experiment that temperature is the dial that trades determinism for diversity and that truncation makes a token unsamplable. Then it maps the from-scratch knobs onto what Anthropic's API actually exposes, and this is the surprising part: the frontier models this book uses (claude-opus-4-8) reject temperature, top_p, and top_k with a 400 error, have never offered logit_bias at all, and steer generation instead through prompting, stop sequences, and constrained decoding (structured outputs and strict tools), which is the same logit-masking the lab performs, computed from a grammar. That constrained-decoding idea is the bridge to the next two chapters.

Contents

• The model does not choose a word
• Temperature: the determinism dial
• Truncation: top-k, top-p, min-p
• logit_bias: editing the scores by hand
• The demo
• What Claude actually exposes, and what it took away
• Constrained decoding: a grammar that masks the logits
• Using the real tool: commands and before/after proof
• Further reading
• Takeaways

Every chapter so far has been about the input side of the window: what you send, how you compress it, how you cache it, how you store it. This chapter turns to the output side, because context engineering does not stop when the prompt is assembled. The tokens the model writes back are context too. They cost output money (the expensive half of the bill, from Chapter 2), they get appended to the transcript and re-sent on the next turn, and when the model writes a method name that does not exist, that mistake becomes part of the context the next step reasons over. Controlling how the model generates is a context lever, and the place all of that control happens is a vector of numbers called the logits.

The model does not choose a word

It is tempting to picture the model reading the prompt and then, somehow, deciding on the next word. That is not what happens, and the gap between the picture and the reality is where all the control lives. Chapter 14 followed a token through attention; here we pick up at the very last step, the output head. After all the attention layers have run, the model holds one vector, and it multiplies that vector by a big matrix whose rows are the vocabulary tokens. The result is one real number per token in the vocabulary. Those numbers are the logits. A logit is a raw, unbounded score: higher means the model prefers that token here, but the numbers are not probabilities, they do not sum to one, and some are negative.

To turn logits into something you can sample from, you push them through the softmax function. Softmax does two things at once: it makes every number positive (by exponentiating) and it makes them sum to one (by dividing by the total), so the output is a genuine probability distribution over the whole vocabulary. Written out, the probability of token $i$ is

$$p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$

where $z_i$ is the logit for token $i$ and the sum in the denominator runs over every token $j$ in the vocabulary. The exponential is what makes softmax interesting: because $e^z$ grows fast, a token whose logit is a little higher than the rest gets a lot more probability, not a little more. That is the model expressing confidence. Two logits close together become two similar probabilities; two logits far apart become one near-certainty and one near-zero.

Remember. The model emits logits, one real number per vocabulary token. Softmax turns that vector into a probability distribution. Sampling draws one token from that distribution. Everything a decoder lets you tune is a change to the logits or to the distribution before the draw. Change nothing and take the argmax and you get greedy decoding: the single highest-logit token, every time, fully deterministic.

We will work with a toy the whole way, because the real vocabulary has on the order of 100,000 tokens and you cannot read a 100,000-long vector. Imagine the prompt is The weather today is and the model has scored eight continuations with these logits:

sunny 3.2   cloudy 2.1   warm 1.7   rainy 1.4   cold 0.9   fine 0.5   nice 0.3   banana -4.0

Softmax turns that into sunny 51.4%, cloudy 17.1%, warm 11.5%, rainy 8.5%, cold 5.2%, fine 3.5%, nice 2.8%, banana 0.0%. Notice that sunny, whose logit is only 1.1 above cloudy, ends up with three times the probability, and banana, four logits below the pack, rounds to zero. That is the exponential at work. Everything below reshapes this vector.

Temperature: the determinism dial

The single most important knob divides every logit by a number $T$ called the temperature, before softmax:

$$p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$$

Think about what dividing does. When $T$ is small (say 0.5), you are dividing by less than one, which stretches the logits apart, so the gaps grow, the exponential exaggerates them further, and the distribution sharpens onto its favourite. In the limit $T \to 0$ the top token gets all the probability and you are back to greedy decoding. When $T$ is large (say 1.5 or 3.0) you are dividing by more than one, which squeezes the logits together, so the gaps shrink and the distribution flattens toward uniform, handing real probability to tokens that were almost dead. At exactly $T = 1$ you divide by one and nothing changes; that is the raw distribution.

So temperature is a determinism dial. Turn it down and the model is predictable, repetitive, and reproducible; the same prompt tends to the same answer. Turn it up and the model is varied, surprising, and sometimes wrong; the same prompt wanders. There is no "correct" temperature. A classifier that must return the same label every time wants it near zero; a brainstorming tool that must not repeat itself wants it high. The important thing to understand is that this one number is the reason two calls with an identical prompt can return different text, and turning it to zero is how you (mostly) stop that.

Truncation: top-k, top-p, min-p

Temperature reshapes the whole distribution but never forbids anything: even at $T = 0.5$, banana keeps a sliver of probability, and one draw in ten thousand will pick it. Truncation methods fix that by cutting the tail off entirely, setting the losing tokens to zero probability so they can never be sampled no matter how many times you draw. There are three common ways to decide where the tail begins.

Top-k is the bluntest: keep the $k$ highest-logit tokens, throw the rest away, and renormalize over the survivors. With $k = 3$ on our example you keep sunny, cloudy, warm and the other five get exactly zero. It is simple and it guarantees the junk is gone, but $k$ is a fixed count that does not know whether the model is confident or unsure: three tokens is too few when the model is genuinely torn among ten good options and too many when only one word fits.

Top-p, also called nucleus sampling, fixes that by keeping a fixed probability mass instead of a fixed count. Sort the tokens by probability, walk down the list adding them up, and stop as soon as the running total first reaches $p$. Everything you have collected is the "nucleus"; everything below is dropped. With $p = 0.9$ the nucleus grows or shrinks to hold 90% of the mass, so it is small when one token dominates and wide when the model is spreading its bets. That adaptivity is why top-p is the most common default.

Min-p is a newer, relative threshold: keep every token whose probability is at least min_p times the top token's probability. If the best token has 60% and min_p is 0.1, you keep everything above 6%. When the model is confident the top token towers over the rest and the nucleus collapses to almost one token; when the model is unsure the top token is low, the bar drops, and the nucleus stays wide. It reacts to the shape of the distribution rather than a fixed count or a fixed mass.

Don't be confused. Temperature and truncation answer different questions and are used together, not instead of each other. Temperature asks "how much should I exaggerate or flatten the model's preferences?" and touches every token. Truncation asks "which tokens are even allowed to be drawn?" and hard-zeros a set. A typical sampler applies truncation first (decide the candidate set), then temperature within it (decide how sharply to prefer inside that set). The one guarantee only truncation can make is this token will never appear, which is exactly the guarantee that matters when the wrong token would be a bug.

logit_bias: editing the scores by hand

The knobs above are global: they reshape or trim the distribution but they do not target a specific token. logit_bias does. It adds a per-token number to the raw logits before anything else, so you can push one exact token up or down. Assign a token a bias of +8 and you can drag it from the bottom of the pack to the winner; assign it -inf (negative infinity) and, after softmax, its probability is exactly zero, which is how you ban a token outright. Because the bias is applied to logits, it flows through temperature and truncation untouched: a banned token is banned no matter what else you set.

This is the sharpest tool of the four and the one providers are most cautious about, because a large enough positive bias overrides the model's judgment entirely: you can force any word to win. It is genuinely useful for narrow jobs (ban the three tokens that start a refusal preamble; forbid a stop token until a minimum length) and genuinely dangerous as a way to steer meaning. Keep that tension in mind, because it explains a design decision we will hit shortly: Anthropic's API does not offer logit_bias at all.

The demo

The script below carries the eight-token toy and implements every knob above as a few lines of NumPy: softmax, temperature, top_k, top_p, min_p, logit_bias. Then it runs the honest test, a seeded 4000-draw experiment, so you can watch the same logits produce wildly different behaviour as you turn the dials, and confirm that a truncated token really does show up 0.0% of the time.

"""The decoder's dials, from scratch: logits -> probabilities -> a token.

A language model does not emit words. On every step it emits one real number
per vocabulary token, a *logit*, and everything a decoder lets you tune
(temperature, top-k, top-p, min-p, logit bias, banned strings) is a
transformation applied to that logit vector before a token is drawn. This lab
builds the whole pipeline in NumPy on an 8-word toy vocabulary so every knob is
a few lines you can read, and it ends by proving, with a seeded 4000-sample
experiment, that temperature is the dial that trades determinism for diversity.

Nothing here needs a network or an API key. numpy only.
"""

import numpy as np

# A toy next-token distribution. Imagine the prompt is "The weather today is"
# and the model has scored these eight continuations. Logits are raw scores:
# unbounded, not probabilities, higher = more preferred.
VOCAB = ["sunny", "cloudy", "warm", "cold", "rainy", "fine", "nice", "banana"]
LOGITS = np.array([3.2, 2.1, 1.7, 0.9, 1.4, 0.5, 0.3, -4.0])


def softmax(logits):
    """Turn a logit vector into a probability distribution.

    Subtract the max first (a standard trick) so exp() never overflows; it
    does not change the result because softmax is shift-invariant."""
    z = logits - np.max(logits)
    e = np.exp(z)
    return e / e.sum()


def show(dist, note=""):
    """Print a distribution as sorted percentages, highest first."""
    order = np.argsort(-dist)
    parts = [f"{VOCAB[i]} {dist[i] * 100:4.1f}%" for i in order if dist[i] > 1e-6]
    print(("  " + note).ljust(26) + "  ".join(parts))


def temperature(logits, t):
    """Divide logits by t BEFORE softmax. t<1 sharpens (more greedy), t>1
    flattens (more random), t->0 becomes argmax, t=1 leaves it unchanged."""
    return softmax(logits / t)


def top_k(logits, k):
    """Keep only the k highest-logit tokens; set the rest to -inf so softmax
    gives them exactly zero probability, then renormalize over the survivors."""
    masked = np.full_like(logits, -np.inf)
    keep = np.argsort(-logits)[:k]
    masked[keep] = logits[keep]
    return softmax(masked)


def top_p(logits, p):
    """Nucleus sampling: sort by probability, keep the smallest prefix whose
    cumulative probability first reaches p, drop the long tail, renormalize.
    The size of the kept set adapts to how peaked the distribution is."""
    probs = softmax(logits)
    order = np.argsort(-probs)
    cum = np.cumsum(probs[order])
    # keep everything up to and including the token that crosses p
    cutoff = np.searchsorted(cum, p) + 1
    keep = order[:cutoff]
    masked = np.full_like(logits, -np.inf)
    masked[keep] = logits[keep]
    return softmax(masked)


def min_p(logits, floor):
    """Keep tokens whose probability is at least floor * (top token's
    probability). A relative threshold: when the model is confident the nucleus
    shrinks to almost one token, when it is unsure the nucleus stays wide."""
    probs = softmax(logits)
    thresh = floor * probs.max()
    masked = np.where(probs >= thresh, logits, -np.inf)
    return softmax(masked)


def logit_bias(logits, bias):
    """Add a per-token bias to the raw logits, exactly like OpenAI's
    logit_bias. -inf (or a large negative) bans a token outright; a positive
    value makes it more likely. This is applied to logits, not probabilities,
    so its effect passes through every other transform above."""
    return softmax(logits + bias)


def demo_distributions():
    print("=== 1. Temperature reshapes the distribution ===")
    show(softmax(LOGITS), "raw (t=1.0):")
    for t in (0.5, 0.7, 1.5, 3.0):
        show(temperature(LOGITS, t), f"t={t}:")
    print("  (t<1 concentrates mass on 'sunny'; t>1 spreads it toward the tail)\n")

    print("=== 2. Truncation: top-k, top-p, min-p keep a subset ===")
    show(top_k(LOGITS, 3), "top-k=3:")
    show(top_p(LOGITS, 0.9), "top-p=0.90:")
    show(top_p(LOGITS, 0.5), "top-p=0.50:")
    show(min_p(LOGITS, 0.1), "min-p=0.10:")
    print("  (each drops the tail so it can never be sampled; note how the")
    print("   top-p set size changes with the threshold)\n")

    print("=== 3. logit_bias edits the scores directly ===")
    ban = np.zeros_like(LOGITS)
    ban[0] = -np.inf                      # ban "sunny", the favourite
    show(logit_bias(LOGITS, ban), "ban 'sunny':")
    boost = np.zeros_like(LOGITS)
    boost[7] = 8.0                        # push "banana" from -4.0 to +4.0
    show(logit_bias(LOGITS, boost), "boost 'banana' +8:")
    print("  (banning reassigns 'sunny's mass to the rest; a big enough boost")
    print("   can make any token win, which is why providers gate this knob)\n")


def demo_sampling():
    """Prove the point empirically: draw 4000 tokens under three settings and
    count outcomes. Same logits, same seed, different dials."""
    print("=== 4. 4000 seeded draws: the dial controls diversity ===")
    rng = np.random.default_rng(0)
    N = 4000
    settings = [
        ("greedy (argmax)", lambda: int(np.argmax(LOGITS))),
        ("t=0.7", lambda: rng.choice(len(VOCAB), p=temperature(LOGITS, 0.7))),
        ("t=1.0", lambda: rng.choice(len(VOCAB), p=softmax(LOGITS))),
        ("t=1.5", lambda: rng.choice(len(VOCAB), p=temperature(LOGITS, 1.5))),
        ("top-k=3, t=1.0", lambda: rng.choice(len(VOCAB), p=top_k(LOGITS, 3))),
    ]
    print(f"  {'setting':<18}" + "".join(f"{w:>8}" for w in VOCAB))
    for name, draw in settings:
        counts = np.zeros(len(VOCAB), dtype=int)
        for _ in range(N):
            counts[draw()] += 1
        pct = counts / N * 100
        print(f"  {name:<18}" + "".join(f"{p:7.1f}%" for p in pct))
    print("""
  Greedy always returns 'sunny': zero diversity, fully reproducible. Raise
  the temperature and mass leaks to 'cloudy', 'warm', 'rainy'. top-k=3
  keeps the three leaders and gives the tail exactly 0.0%, which is the
  guarantee greedy and temperature alone cannot make: a truncated token
  can never appear, no matter how many times you sample.""")


if __name__ == "__main__":
    demo_distributions()
    demo_sampling()

Running it:

=== 1. Temperature reshapes the distribution ===
  raw (t=1.0):            sunny 51.4%  cloudy 17.1%  warm 11.5%  rainy  8.5%  cold  5.2%  fine  3.5%  nice  2.8%  banana  0.0%
  t=0.5:                  sunny 83.0%  cloudy  9.2%  warm  4.1%  rainy  2.3%  cold  0.8%  fine  0.4%  nice  0.3%
  t=0.7:                  sunny 67.8%  cloudy 14.1%  warm  7.9%  rainy  5.2%  cold  2.5%  fine  1.4%  nice  1.1%  banana  0.0%
  t=1.5:                  sunny 37.3%  cloudy 17.9%  warm 13.7%  rainy 11.2%  cold  8.0%  fine  6.2%  nice  5.4%  banana  0.3%
  t=3.0:                  sunny 23.9%  cloudy 16.5%  warm 14.5%  rainy 13.1%  cold 11.1%  fine  9.7%  nice  9.1%  banana  2.2%
  (t<1 concentrates mass on 'sunny'; t>1 spreads it toward the tail)

=== 2. Truncation: top-k, top-p, min-p keep a subset ===
  top-k=3:                sunny 64.3%  cloudy 21.4%  warm 14.3%
  top-p=0.90:             sunny 54.9%  cloudy 18.3%  warm 12.2%  rainy  9.1%  cold  5.5%
  top-p=0.50:             sunny 100.0%
  min-p=0.10:             sunny 54.9%  cloudy 18.3%  warm 12.2%  rainy  9.1%  cold  5.5%
  (each drops the tail so it can never be sampled; note how the
   top-p set size changes with the threshold)

=== 3. logit_bias edits the scores directly ===
  ban 'sunny':            cloudy 35.2%  warm 23.6%  rainy 17.5%  cold 10.6%  fine  7.1%  nice  5.8%  banana  0.1%
  boost 'banana' +8:      banana 53.4%  sunny 24.0%  cloudy  8.0%  warm  5.4%  rainy  4.0%  cold  2.4%  fine  1.6%  nice  1.3%
  (banning reassigns 'sunny's mass to the rest; a big enough boost
   can make any token win, which is why providers gate this knob)

=== 4. 4000 seeded draws: the dial controls diversity ===
  setting              sunny  cloudy    warm    cold   rainy    fine    nice  banana
  greedy (argmax)     100.0%    0.0%    0.0%    0.0%    0.0%    0.0%    0.0%    0.0%
  t=0.7                68.1%   14.0%    8.3%    2.1%    5.0%    1.4%    1.1%    0.0%
  t=1.0                50.9%   17.1%   11.6%    5.6%    8.4%    3.5%    2.9%    0.1%
  t=1.5                36.4%   18.7%   13.6%    8.1%   11.2%    6.0%    5.7%    0.3%
  top-k=3, t=1.0       63.3%   22.5%   14.1%    0.0%    0.0%    0.0%    0.0%    0.0%

Read the last block, because it is the proof. Greedy returns sunny 100% of the time: zero diversity, perfectly reproducible, and this is what you want when you need the same answer twice. Raise the temperature to 0.7, then 1.0, then 1.5, and watch the mass bleed out of sunny and into cloudy, warm, rainy: the dial is doing exactly one thing, trading determinism for spread. And the last row is the guarantee no temperature setting can give: under top-k=3, the four truncated tokens are drawn 0.0% of the time across four thousand samples, not "rarely", but never, because they were removed from the distribution before the draw. When a wrong token is a bug rather than a stylistic quibble, that hard zero is the only thing that will do.

What Claude actually exposes, and what it took away

Now cross from the toy to the real API, because the surprising part is not which knobs exist but which ones were removed. Every knob above is standard across providers, and on older Claude models (Opus 4.6 and earlier, Sonnet 4.6) you can still pass temperature, top_p, and top_k. But on the frontier models this book targets, the picture is different and worth stating plainly:

• temperature, top_p, and top_k are gone on the newest models. On claude-opus-4-8, claude-opus-4-7, claude-sonnet-5, and claude-fable-5, passing any of the three is not ignored, it is a hard 400 error. The sampling dials were deliberately taken away.
• logit_bias was never offered. Unlike OpenAI's API, Anthropic has no per-token bias parameter, so the "ban a token" and "boost a token" moves from the demo have no direct API on the Claude side.
• What remains is stop_sequences (strings that halt generation, reported back as stop_reason: "stop_sequence"), the effort parameter (low through max, which governs how much the model thinks and how many tokens it spends), and constrained decoding through structured outputs and strict tools, which the next section unpacks.

Here is the 400 you get if you reach for the old dial on a current model (this is illustrative, the shape to expect, since the Anthropic SDK is not installed on this box):

from anthropic import Anthropic

client = Anthropic()
client.messages.create(
    model="claude-opus-4-8",
    max_tokens=64,
    temperature=0.7,           # removed on Opus 4.8 / 4.7, Sonnet 5, Fable 5
    messages=[{"role": "user", "content": "Say something."}],
)

anthropic.BadRequestError: Error code: 400 - {'type': 'error', 'error':
{'type': 'invalid_request_error', 'message': 'temperature: Extra inputs are
not permitted'}}

Why would a provider remove the most famous knob in the field? Because on a strong, reasoning-capable model, temperature is a crude instrument for what people actually want. If you want variety, a prompt that asks for variety ("propose four distinct directions") gives you diverse and coherent options, whereas a high temperature gives you noise sprinkled uniformly across every token, including the ones that should have been certain. If you want determinism, note that temperature = 0 never guaranteed identical outputs anyway (floating point, batching, and load all perturb it), so it was always a soft promise. And if you want the output to obey a shape, the right tool is not a global distribution tweak but a hard constraint on the token set, which is the next section. The removal is a bet that prompting plus constrained decoding beats a temperature slider on a model this capable. Whether you agree or not, the practical consequence is concrete: you cannot turn down the temperature in Claude Code, because the model underneath will not accept the parameter. Its determinism comes from the effort setting, the prompt, and the schemas its tools declare, not from a sampling dial.

Constrained decoding: a grammar that masks the logits

If Anthropic removed the sampling dials, what does it give you when the output must have a shape, valid JSON, a value from a fixed set, a call that matches a tool's schema? The answer is constrained decoding, and it is the same logit-masking the demo did in top_k, only the mask comes from a grammar instead of a rank cutoff.

The mechanism is worth seeing clearly because it reappears, in a stronger form, in Chapter 39. Suppose you require the output to match a JSON schema with a field "status" whose value must be one of "active", "pending", or "closed". As the model decodes, a component on the server tracks where in the grammar you are. Right after it emits "status": ", only three continuations keep the output valid, so the server builds the set of tokens that begin one of those three strings and sets every other logit to negative infinity. The model samples, but only from tokens that keep the JSON on a legal path. It cannot emit "activ and then e_but_wrong, because after activ the only allowed next token is e". The grammar walks the model, token by token, exactly the way top_k walked our toy, except the allowed set is recomputed at every step from "what would still be valid here".

Anthropic exposes two doors to this. Structured outputs (output_config with a json_schema format) constrain the whole response to satisfy a schema. Strict tool use (strict: true on a tool definition) constrains a tool call's arguments to satisfy the tool's input_schema exactly. Both are guarantees, not requests: the model does not "try to" produce valid JSON, it is unable to produce invalid JSON, because the illegal tokens were masked before each draw. That is why structured outputs eliminate the parse-failure retries that a "please respond in JSON" instruction still suffers.

Don't be confused. "Please respond in JSON" is a prompt; structured outputs is a constraint. The prompt raises the probability that the output parses; the constraint makes non-parsing output impossible. The difference is the same as the difference between temperature (reshape the odds) and truncation (forbid the token) from earlier in this chapter. On the frontier models, constrained decoding is the sanctioned replacement for the logit_bias-style control the API declines to give you: you do not ban tokens one at a time, you declare the legal shape and let the grammar ban everything that violates it.

There is one more thing to notice, and it is the whole reason the next two chapters exist. In structured outputs the constraint comes from a static grammar you wrote down in advance (the JSON schema). But the most valuable constraint in code generation cannot be written down in advance: the set of method names that are valid after account. depends on the type of account, which depends on the whole repository. To mask logits to that set, you need a constraint computed live from the code, and computing facts about code live is exactly what a language server does. The next chapter opens up the tool that turns a language server into an agent capability (Serena); the one after wires the language server's answers into the sampling loop, giving you constrained decoding where the grammar is the codebase itself.

Using the real tool: commands and before/after proof

The from-scratch masking above is exactly what you invoke when you ask the Anthropic API for a constrained shape. Here are the two controls that survived on the frontier models, with the commands to use them and the before/after that proves they do what the demo did.

Structured outputs (the grammar mask). Instead of asking for JSON and hoping, declare the schema and let the server mask every token that would break it. This is illustrative (the SDK and a key are not on this box), but it is the exact shape:

from anthropic import Anthropic

client = Anthropic()
resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=256,
    output_config={
        "format": {
            "type": "json_schema",
            "schema": {
                "type": "object",
                "properties": {
                    "sentiment": {"type": "string",
                                  "enum": ["positive", "negative", "neutral"]},
                    "confidence": {"type": "number"},
                },
                "required": ["sentiment", "confidence"],
                "additionalProperties": False,
            },
        }
    },
    messages=[{"role": "user", "content": "Review: the build broke again."}],
)
print(resp.content[0].text)

{"sentiment": "negative", "confidence": 0.82}

The value of "sentiment" is guaranteed to be one of the three enum strings, not because the model was asked nicely but because after "sentiment": " the grammar masked every token that did not begin positive, negative, or neutral, the same three-way mask the toy applied by rank. The before is a "respond in JSON" prompt that fails to parse some fraction of the time and forces a retry; the after never fails to parse, so the retry path (and its tokens) disappears.

Stop sequences (halt on a string). The one output-shaping knob that is not a grammar. Give the API a list of strings, and generation stops the moment the model would emit one, with stop_reason reported as "stop_sequence":

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    stop_sequences=["\n\nQ:"],       # stop before the model invents the next Q
    messages=[{"role": "user", "content": "Q: capital of France?\nA:"}],
)
print(resp.stop_reason)              # -> "stop_sequence"

stop_reason: stop_sequence

That is a targeted way to cap output tokens (the lever from Chapter 4) without a fixed max_tokens guess: you stop on the structure, not on a length.

The on-box proof. The API snippets above are labelled illustrative because they need a network call. The measurement we actually ran is the NumPy demo, which showed a truncated token appearing 0.0% of the time across 4000 draws. That hard zero is the on-box evidence for the claim that a masked logit is not "unlikely" but impossible, and structured outputs and strict tools are that same mask applied on Anthropic's servers with a grammar deciding the allowed set. To reproduce the API side with exact numbers, run the two snippets with a key set; the shapes are what you will see.

Further reading

• The Curious Case of Neural Text Degeneration (Holtzman et al., 2020). The paper that introduced nucleus (top-p) sampling and showed why pure greedy and pure sampling both fail on open-ended generation. The origin of the truncation half of this chapter.
• Anthropic structured outputs (docs.claude.com). Anthropic's own reference for output_config.format and strict: true, including the JSON-schema subset they support and the one-time schema-compilation cost. The production form of the constrained-decoding section.
• Outlines / llguidance / XGrammar. Open-source libraries that implement grammar-constrained decoding by compiling a schema or grammar into a per-step token mask. Read any one of them to see the from-scratch mask of this chapter at production scale and speed.
• Anthropic migration guide, sampling parameters. The note that temperature, top_p, and top_k return a 400 on Opus 4.8 / 4.7, Sonnet 5, and Fable 5, and the recommendation to steer with prompting and effort instead. The source for the "what was taken away" section.

Takeaways

• The model emits logits (one real number per token); softmax turns them into a distribution; sampling draws one token. Every decoding control is a transform applied before the draw. Greedy (argmax) is the fully deterministic special case.
• Temperature trades determinism for diversity by sharpening ($T<1$) or flattening ($T>1$) the whole distribution. Truncation (top-k, top-p, min-p) removes tail tokens so they can never be sampled; the demo confirmed a truncated token appears 0.0% of the time across 4000 draws. logit_bias edits one token's score directly, up to and including a -inf ban.
• On the frontier Claude models (claude-opus-4-8 and newer), temperature, top_p, and top_k are removed (a 400 error), and logit_bias was never offered. Steering is done through prompting, the effort parameter, stop sequences, and constrained decoding.
• Constrained decoding (structured outputs, strict tools) is logit-masking driven by a grammar: it makes invalid output impossible, not merely unlikely, which is why it replaces both logit_bias and "please respond in JSON".
• The most useful constraint in code generation cannot be written as a static grammar, because the legal set (a type's members) depends on the whole repository. Computing that live is a language server's job, which is where the next two chapters go.

👉 Constrained decoding masks the logits to a grammar you declare in advance. To mask them to a set computed live from the codebase, you first need a tool that turns a language server into an agent capability. The next chapter opens up exactly that tool, Serena, and shows how it works internally and how it manages memory. Continue to The language server as context engine.