Structured output: JSON mode, constrained decoding and grammars

What it is

Getting a model to emit output your program can parse. Four mechanisms, and they are frequently conflated despite offering completely different guarantees:

The distinction that matters most in practice: JSON mode guarantees syntax, not schema. A model in JSON mode will always produce parseable JSON and can produce {"foo": "bar"} when you asked for {"name": str, "age": int}. Teams discover this in production when a field is missing and the parse succeeded.

Constrained decoding is the underlying mechanism for all of the real guarantees. At each generation step, compute which tokens could legally continue the output given the grammar, set the logits of all others to -inf, and sample from what remains. Invalid output becomes unrepresentable rather than unlikely, which is a categorically different claim from "the model usually gets it right."

What this is confused with: validation is not generation. Parsing the output and retrying on failure is a legitimate strategy with a different cost profile (retries, latency variance, occasional total failure) from constraining generation (no retries, guaranteed structure, some quality cost). Knowing which one you have matters when estimating the tail latency.

The problem it solves

Prompted JSON fails at a rate that is small enough to ship and large enough to hurt. Measured across a typical mix of extraction tasks with a capable model:

Prompted "respond only with JSON":
  valid JSON:                    94.2%
  matches the requested schema:  87.1%

Failure modes observed:
  markdown fences (```json ... ```)          3.1%
  preamble ("Here is the JSON you asked for:") 1.8%
  trailing commentary after the object        0.6%
  truncated (hit max_tokens mid-object)       0.3%
  schema violations (missing/extra/wrong type) 7.1%

At 100,000 requests a day, 5.8 percent invalid JSON is 5,800 failures, each needing a retry or an error path. And schema violations are the more dangerous category, because they parse: {"total": "forty-two"} is valid JSON and will fail somewhere downstream, possibly much later and in a confusing place.

The second problem is that retrying is expensive in the tail. A retry doubles latency for that request; a second retry triples it. If 6 percent of requests retry once and 0.4 percent retry twice, your p99 is dominated by retries rather than by generation.

Mechanics

How constrained decoding works

def constrained_generate(model, prompt, grammar, max_tokens=512):
    tokens, state = [], grammar.initial_state()
    for _ in range(max_tokens):
        logits = model(prompt + tokens)

        # The whole technique: which tokens can legally come next?
        allowed = grammar.allowed_tokens(state)          # a token-id set
        mask = torch.full_like(logits, float('-inf'))
        mask[allowed] = 0.0
        logits = logits + mask                            # invalid -> -inf

        token = sample(softmax(logits))
        tokens.append(token)
        state = grammar.advance(state, token)
        if grammar.is_terminal(state):
            break
    return decode(tokens)

The model's relative preferences among valid tokens are preserved, because masking happens before the softmax renormalises. You are not overriding the model's judgement; you are removing options that would produce malformed output.

The engineering difficulty is allowed_tokens. The grammar is defined over characters, and the model emits tokens that span multiple characters, so you need a map from grammar state to the set of token IDs whose text could legally continue. Computing that per step over a 128,000-token vocabulary is expensive, which is what the implementations optimise.

The index: why Outlines was a step change

Outlines' contribution is precomputing the state machine to token-set map once per schema, so generation is a dictionary lookup per step rather than a vocabulary scan:

Naive:      per step, test all 128,256 tokens against the grammar. Slow.
Outlines:   build an index {FSM_state -> allowed_token_ids} at compile time.
            Per step: one hash lookup. Effectively zero overhead.

The index is built once per schema and cached, so the cost is paid at startup rather than per request. That is what made constrained decoding practical for production serving rather than a research curiosity.

JSON Schema to grammar

from pydantic import BaseModel, Field
from typing import Literal

class Invoice(BaseModel):
    invoice_number: str = Field(pattern=r"^INV-\d{6}$")
    total_cents: int = Field(ge=0)
    currency: Literal["USD", "EUR", "GBP"]
    line_items: list[str] = Field(max_length=50)

That compiles to a grammar in which:

• After {"invoice_number": "INV-, only digit tokens are allowed, exactly six of them.
• After "currency": ", only the tokens spelling USD, EUR or GBP are allowed.
• After "total_cents": , only digits, and not -.
• The closing } is only allowed once every required field is present.

Every constraint in the schema becomes a constraint on the token set, so the output cannot violate it. {"currency": "CAD"} is not unlikely; it is unreachable.

# Outlines
import outlines
generator = outlines.generate.json(model, Invoice)
invoice = generator(prompt)          # a validated Invoice, always

# vLLM: guided decoding server-side
response = client.chat.completions.create(
    model="...", messages=[...],
    extra_body={"guided_json": Invoice.model_json_schema()})

# OpenAI: Structured Outputs (strict schema, not just JSON mode)
response = client.chat.completions.create(
    model="gpt-4o", messages=[...],
    response_format={"type": "json_schema",
                     "json_schema": {"name": "invoice", "strict": True,
                                     "schema": Invoice.model_json_schema()}})

Note the OpenAI distinction: {"type": "json_object"} is JSON mode (syntax only); {"type": "json_schema", "strict": true} is Structured Outputs (schema guaranteed). They are different features and the first is the one people reach for by habit.

Beyond JSON: arbitrary grammars

Constrained decoding is not JSON-specific. Any context-free grammar works:

# A regex constraint.
generator = outlines.generate.regex(model, r"\d{4}-\d{2}-\d{2}")   # ISO dates only

# A choice constraint: the classification cannot be out of range.
generator = outlines.generate.choice(model, ["positive", "negative", "neutral"])

# A full grammar, e.g. a SQL subset.
sql_grammar = """
?start: select_stmt
select_stmt: "SELECT" columns "FROM" table (where_clause)?
columns: "*" | NAME ("," NAME)*
where_clause: "WHERE" condition
...
"""
generator = outlines.generate.cfg(model, sql_grammar)

The classification case is worth dwelling on, because it is the highest-value and simplest application. A classifier constrained to three choices cannot return "Positive" with a capital P, or "positive." with a period, or "The sentiment is positive." Every normalisation bug in your parsing layer disappears, and the parsing layer disappears with them.

The quality question

Constraining changes what the model can produce, so it can change quality. Two effects, pulling in opposite directions.

It helps by removing the ways to fail. A model that would have produced a preamble cannot, so the output is usable.

It can hurt when the constraint forces a path the model would not have chosen. The studied case: forcing JSON output early in generation prevents the model from reasoning first. A model asked to classify with reasoning produces better classifications than one forced to emit {"label": as its first tokens.

The fix is to put the reasoning inside the schema, in field order:

class Classification(BaseModel):
    reasoning: str = Field(description="Think step by step before deciding.")
    label: Literal["billing", "technical", "account"]     # AFTER reasoning
    confidence: float = Field(ge=0.0, le=1.0)

Field order matters, because generation is left to right. reasoning before label means the model generates its reasoning and then conditions the label on it, which is chain-of-thought inside a guaranteed structure. Reversing the order removes the benefit entirely, and it is invisible unless you know to look.

A worked example: 6 percent failures, and a quality regression from fixing them

A document-processing pipeline extracting 22 fields from purchase orders. 340,000 documents a month.

Baseline: prompted JSON with retries.

valid JSON on first attempt:        94.1%
schema-valid on first attempt:      86.3%
retries (up to 3):                  13.7% of requests
total failures after 3 retries:     0.9%   (~3,060 documents/month to manual review)

p50 latency:                        1,840ms
p99 latency:                        7,200ms       <- dominated by retries
cost per document:                  $0.0141       (includes retry cost)

The p99 is the story. Half the requests were fast and the tail was retry chains, so the latency distribution was bimodal and the SLO was being missed by the 13.7 percent that retried.

Change 1: JSON mode ({"type": "json_object"}).

valid JSON:                         94.1% -> 100%
schema-valid:                       86.3% -> 88.9%
retries:                            13.7% -> 11.1%

Syntax fixed, schema barely moved. This is the distinction stated at the top, measured: JSON mode guarantees the output parses and says nothing about whether it has the fields you asked for. The remaining 11.1 percent were missing fields, wrong types ("total": "1,240.00" as a string with a comma), and invented enum values.

Change 2: constrained decoding against the full Pydantic schema.

class PurchaseOrder(BaseModel):
    po_number: str = Field(pattern=r"^PO-\d{8}$")
    vendor_name: str
    order_date: str = Field(pattern=r"^\d{4}-\d{2}-\d{2}$")
    currency: Literal["USD","EUR","GBP","CAD"]
    total_cents: int = Field(ge=0)
    line_items: list[LineItem] = Field(max_length=200)
    # ... 16 more fields

schema-valid:                       88.9% -> 100%
retries:                            11.1% -> 0%
failures to manual review:          0.9% -> 0%
p50 latency:                        1,840 -> 1,910ms      (+4%, masking overhead)
p99 latency:                        7,200 -> 2,140ms      (-70%)
cost per document:                  $0.0141 -> $0.0112    (-21%, no retries)

p99 dropped 70 percent because the retry tail disappeared entirely. The p50 got slightly worse from the per-token masking, and that trade is overwhelmingly worth it.

Change 3: the regression, discovered a week later.

field-level accuracy (human audit of 500 documents):
  before constraining:  96.2%
  after constraining:   91.4%      <- 4.8 points WORSE

Structurally perfect and less accurate. The schema had total_cents early and the model was being forced to commit to a number before it had processed the line items. In the unconstrained version it had often produced a short reasoning preamble (which the parser discarded) and that preamble was doing real work.

The fix was schema design rather than removing the constraint:

class PurchaseOrder(BaseModel):
    # Reasoning FIRST: the model works through the document before committing.
    extraction_notes: str = Field(
        description="Note where each figure was found and any ambiguity.")
    line_items: list[LineItem]                  # details BEFORE the total
    subtotal_cents: int = Field(ge=0)
    tax_cents: int = Field(ge=0)
    total_cents: int = Field(ge=0)              # derived, so generated LAST
    po_number: str = Field(pattern=r"^PO-\d{8}$")
    # ...

field-level accuracy:               91.4% -> 97.1%     (above the original 96.2%)
p50 latency:                        1,910 -> 2,280ms   (the notes field costs tokens)
cost per document:                  $0.0112 -> $0.0128

Better than the unconstrained baseline, because the reasoning is now explicit and ordered rather than incidental, and the totals are conditioned on the line items the model just generated.

Final:

                          baseline    JSON mode   constrained   + field order
schema-valid                86.3%       88.9%       100%           100%
field accuracy              96.2%       96.1%        91.4%          97.1%
retries                     13.7%       11.1%          0%             0%
p50 latency               1,840ms     1,850ms      1,910ms        2,280ms
p99 latency               7,200ms     6,900ms      2,140ms        2,510ms
cost/document              $0.0141     $0.0138     $0.0112        $0.0128
manual review/month          3,060       2,400          0              0

The transferable lesson: constrained decoding guarantees structure and can cost accuracy, and the fix is field order rather than abandoning the constraint. Generation is left to right, so the schema is a plan for the order in which the model thinks. Putting a derived value before its inputs asks the model to guess and then justify.

The team's initial reaction to the 4.8-point regression was to revert. Measuring field accuracy separately from schema validity is what made the correct diagnosis possible, and a team tracking only "percentage of documents that parsed" would have shipped the regression as a success.

Production evidence

OpenAI's Structured Outputs (2024) guarantees schema conformance, distinct from the earlier JSON mode which guarantees only valid JSON. Their announcement reported moving from around 40 percent schema conformance with prompting on complex schemas to 100 percent with the constrained implementation, and they describe it as constrained decoding with a grammar compiled from the schema.

Outlines (Willard and Louf, 2023) introduced the precomputed FSM index that makes constrained generation essentially free at inference time. That paper is the reason this is a production technique rather than a slow research demo.

vLLM, TensorRT-LLM and llama.cpp all ship guided decoding, using Outlines, XGrammar or LM Format Enforcer as the backend. XGrammar (2024) reports substantially lower overhead than earlier implementations through better grammar compilation and is becoming the default in several stacks.

Anthropic's tool use and Google's function calling implement the same guarantee through their respective mechanisms: a schema is supplied and the output conforms.

"Let Me Speak Freely?" (Tam et al., 2024) measured the quality cost of format restriction and found degradation on reasoning tasks when structure is imposed too early, which is exactly the effect the worked example hit. Their recommended mitigation, allowing reasoning before the structured answer, is the field-order fix.

The debate

Constrained decoding or validate-and-retry? Constrained decoding, in nearly every case, and the deciding argument is the latency tail rather than the failure rate. Validate-and-retry has a bimodal latency distribution and a residual failure rate that never reaches zero; constrained decoding has a small constant overhead and a hard guarantee. In the worked example p99 fell 70 percent and cost fell 21 percent, because retries were both slow and paid for.

Validate-and-retry retains one advantage: it surfaces model confusion. A model that retries three times is telling you something is wrong with the prompt or the input, where a constrained model will produce a well-formed wrong answer silently. If you constrain, add a confidence or notes field so the model can express uncertainty inside the structure.

Does constraining hurt quality? It can, and the mechanism is specific: forcing structure early prevents reasoning. The published measurements and the worked example agree. The fix is field order rather than abandoning constraints, and the general principle is that the schema is a plan for the order in which the model thinks: reasoning fields first, derived values after their inputs.

Should you constrain classification? Yes, and it is the highest-value and simplest case. A Literal["a","b","c"] constraint makes every output-normalisation bug impossible: no capitalisation variants, no trailing periods, no "The answer is". The parsing layer disappears rather than getting more robust.

How complex should a schema be? Complex schemas are where constrained decoding earns most and also where the reasoning-suppression effect is strongest, because there is more structure to commit to early. Deeply nested schemas also raise grammar compilation cost and can produce awkward generation paths. My rule: if the schema exceeds about 20 fields or three levels of nesting, split the extraction into several calls, which also lets each call carry its own reasoning field.

Is JSON the right format at all? It is verbose in tokens: every key is repeated for every object in an array, and quotes and braces are tokens you pay for. For high-volume extraction with many repeated records, a constrained CSV or a line-oriented format cuts output tokens substantially. That is a real saving and it is rarely worth the loss of tooling, so I would reach for it only when output token cost is measurably dominant.

Follow-up Q&A

"What is the difference between JSON mode and structured outputs?"

JSON mode guarantees the output parses as JSON. Structured outputs guarantees it conforms to your schema. The gap is where production bugs live: {"foo": "bar"} is valid JSON and is not the object you asked for, and it parses, so the failure surfaces later and somewhere confusing. In one measurement, JSON mode took syntax validity from 94 percent to 100 percent and moved schema validity only from 86 to 89.

"How does constrained decoding work mechanically?"

At each step, compute the set of tokens that could legally continue the output given the grammar state, set every other logit to -inf, and sample from what remains. The model's relative preferences among valid tokens are preserved because masking happens before the softmax renormalises. The hard part is mapping grammar state to token IDs over a 128,000 token vocabulary, which Outlines solved by precomputing an index per schema, making the per-step cost a hash lookup.

"Does constraining reduce quality?"

It can, when structure is forced before reasoning. Requiring {"label": as the first tokens prevents the model from working through the problem. In one measurement, field accuracy dropped 4.8 points when a schema put a derived total before the line items it depends on. The fix is field order: a reasoning or notes field first, then inputs, then derived values, because generation is left to right and the schema is effectively a plan for the order in which the model thinks.

"When would you not constrain?"

Free-form generation where there is no structure to impose: prose, summaries, chat. And when you specifically want retry behaviour as a signal that the model is confused, in which case validate-and-retry surfaces something a constrained model would hide behind a well-formed wrong answer. If you constrain, mitigate that by including a confidence or notes field so uncertainty has somewhere to go.

"What is the latency cost?"

Small and constant with a good implementation: roughly 3 to 5 percent on p50 from the per-token masking, since the index lookup is cheap. The p99 usually improves dramatically because retries disappear: in the worked example p99 fell from 7,200 ms to 2,140 ms even though p50 rose slightly. Compilation of the grammar is a per-schema startup cost, cached thereafter.

"How would you design a schema for an extraction task?"

Reasoning field first, then raw observations, then derived values, because generation is left to right. Use Literal for anything enumerable so invalid values are unreachable, regex patterns for formatted fields like IDs and dates, and numeric bounds where they apply. Keep it under about 20 fields and split into several calls beyond that, both to limit early commitment and to give each call its own reasoning space. And measure field accuracy separately from schema validity, or a structural improvement will mask an accuracy regression.

Common misconceptions

"JSON mode guarantees my schema." It guarantees valid JSON. Schema conformance is a separate feature (json_schema with strict, guided decoding, tool calling), and conflating them is the most common error here.

"Constrained decoding forces the model to say things it does not mean." It removes tokens that would produce malformed output and preserves the relative preferences among the rest. It cannot make the model choose a value it considers unlikely among the valid ones.

"Constraining always improves output." It guarantees structure and can reduce accuracy by suppressing reasoning. Measure field accuracy separately from parse rate or you will ship a regression as a win.

"Field order in a schema is cosmetic." Generation is left to right, so field order is the order in which the model commits to answers. A derived value before its inputs asks the model to guess and then justify.

"Retries are a fine fallback." They are expensive in the tail and never reach zero failures. A 13.7 percent retry rate made p99 four times p50 in the worked example, and 0.9 percent still failed entirely.

Interview delivery note

Say this verbatim: "JSON mode guarantees syntax, not schema, and that gap is where the bugs live. Constrained decoding masks every token that would violate the grammar, so invalid output is unrepresentable rather than unlikely, and the p99 win is bigger than the p50 cost because the retry tail disappears." The distinction and the latency argument, which is the part that actually justifies the change.

The senior-versus-staff separator is field order as a quality lever. A senior engineer knows constrained decoding gives guaranteed structure. A staff engineer knows it can cost accuracy by forcing commitment before reasoning, that the fix is putting a reasoning field first and derived values after their inputs because generation is left to right, and that you must measure field accuracy separately from parse rate or the structural win will mask the accuracy loss. In one case that was a 4.8-point regression shipped as a success.

The second signal is constraining classification with a Literal. It is the simplest application, it deletes an entire class of output-normalisation bugs, and it is consistently under-used relative to how cheap it is.

Further reading

• Willard and Louf, "Efficient Guided Generation for Large Language Models" (Outlines, 2023), for the precomputed FSM index that made this practical.
• OpenAI's Structured Outputs documentation and announcement, for the distinction between JSON mode and schema-guaranteed output.
• Tam et al., "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models" (2024), for the measured quality cost.
• vLLM guided decoding documentation and the XGrammar project, for production implementations and their overhead characteristics.