A language model returns {"refund_amount": 49.99, "reason": "damaged item",} and your parser dies on the trailing comma at 2 a.m. Or it wraps the object in a markdown fence. Or it prepends “Sure, here’s the JSON you asked for:” and now JSON.parse throws on the very first character. Every team shipping agents has hit some version of this, which is why companies building on tool-calling models ask about it directly in interviews.
The question underneath is easy to state and hard to answer well: how do you get a model to produce output a downstream system can consume every single time, not most of the time? At an enterprise agent company the failure mode isn’t academic. If one in five hundred tool calls is malformed and you run millions a day, that’s thousands of broken customer interactions before lunch.
Prompt-and-parse is where everyone starts and where it breaks
The first instinct is to ask nicely. Put “Respond only with valid JSON matching this shape” in the system prompt, add an example, then call json.loads on whatever comes back. This works in the demo and falls over in production, and the reasons are concrete.
Models drift toward being helpful. They add a sentence of explanation before or after the object. They emit ```json fences because that’s what the training data looked like. On longer generations they lose track of an open bracket, switch to single quotes, forget to escape a quote inside a string, or invent a field you never asked for. Bigger models fail less often, but “less often” is not a guarantee, and you can’t run a payment flow on a 99% parse rate.
An interviewer wants you to name the actual mechanism. The model samples one token at a time from a probability distribution, and nothing in plain prompting stops it from sampling a token that makes the string un-parseable. Any real fix has to act at that level.
What constrained decoding actually does
Constrained decoding is the current answer, and if you can explain how it works you’re ahead of most candidates. At each generation step the model produces logits over the whole vocabulary. A grammar engine sits between the logits and the sampler and masks every token that would break the target schema, setting its probability to zero. The model samples only from tokens that keep the output valid. Emitting a trailing comma or an unquoted key becomes structurally impossible, because those tokens are never on the table.
The schema compiles into a state machine, a finite-state machine for regular structures and a pushdown automaton for nested JSON, and the engine tracks which state you’re in as tokens stream out. Given a JSON Schema, it knows that right after an opening brace the only legal next tokens are a quote or a closing brace, and it enforces that token by token.
This is the machinery behind OpenAI’s Structured Outputs with strict: true, Anthropic’s tool-use argument enforcement, and the open-source engines. XGrammar is the default backend in vLLM, SGLang, and TensorRT-LLM, and pushes per-token overhead down into the tens of microseconds. Outlines pioneered the finite-state-machine approach but can spend a long time compiling complicated schemas, sometimes tens of seconds, which bites if schemas are generated at request time rather than fixed ahead of time. llguidance and a handful of others cover similar ground. Here’s the detail that earns points: constrained decoding guarantees the output matches the grammar, and that is the only thing it guarantees.
Valid does not mean correct
This is what separates people who’ve shipped this from people who’ve read about it. Force a model to emit a value from an enum and it will emit one of those values, even when the right answer isn’t in the set. Constrain a field to an integer and you get an integer, not necessarily the right integer. Grammar constraints move the failure from “crashes the parser” to “passes the parser carrying a wrong value,” which is harder to catch because nothing throws.
There’s a quieter cost too. Forcing structure can degrade the reasoning that produced the content. A model that would have thought in prose and then answered sometimes does worse when every token has to fit the schema from the first character. The usual mitigation is a free-text scratchpad field before the constrained fields, so the model reasons first and structures second and you don’t trade answer quality for format safety. A candidate who proposes constrained decoding without mentioning that it can hurt quality has missed something.
Function calling has its own failure surface
Tool calling is structured output with higher stakes, because the output triggers an action. The model has to pick the right tool and then fill its argument schema correctly. The Berkeley Function Calling Leaderboard is the reference here, and what it checks matters: whether the model calls the correct function, whether the arguments are well-formed, whether it handles parallel calls, and whether it correctly decides to call nothing when no tool applies.
Grammar-constrained decoding helps a lot with the well-formed part. Once tool arguments are forced to match their JSON Schema, the whole class of “unexecutable because the JSON is broken” disappears, and a small constrained model can beat a much larger unconstrained one on argument validity. What it doesn’t fix: calling the wrong tool, inventing a plausible but wrong argument value, or firing three tools when it should have answered directly. Those are model-quality and prompt-design problems, and no decoding trick reaches them.
Where each approach actually helps
| Approach | What it guarantees | Typical cost | Where it still breaks |
|---|---|---|---|
| Prompt-and-parse (ask for JSON in the prompt) | Nothing enforced; relies on the model complying | Near zero | Markdown fences, prose, trailing commas, drift on long outputs |
| JSON mode (model told to emit a JSON object) | Output is syntactically valid JSON | Low | Valid JSON that ignores your schema and wrong fields |
| Constrained decoding / grammar (XGrammar, Outlines, OpenAI strict) | Output matches the exact JSON Schema | Per-token masking plus schema compile or warmup | Valid-but-wrong values; can dent answer quality |
| Function / tool calling with strict arguments | Tool arguments match their declared schema | Same as constrained decoding | Wrong tool chosen; hallucinated argument values |
| Post-hoc validation (Pydantic, Zod) | Business rules a grammar can’t express | Cheap check, plus retry cost when it fails | Retry loops and added latency; needs a fail-closed path |
Validation, repair, and knowing when to give up
Even with constrained decoding you validate the output again on your side, because your real constraints are usually richer than the grammar. Pydantic on the Python side and Zod on the TypeScript side are the common tools, and they check what a grammar can’t: that refund_amount is positive, that a date is in the future, that an ID points at something real. Validation catching a bad value is the normal case, not the exception.
When validation fails, the reflex is to reask: hand the error back to the model and let it try again. This works, and it has teeth. Each retry is another full generation, so latency and cost climb, and a model that got it wrong once can loop on the same mistake. Set a hard retry cap, usually one or two, and decide up front what happens when you run out. For a low-stakes summary you might return partial output; for a refund you fail closed and route to a human. Naming that fail-closed decision is often what the interviewer is really digging for.
Guardrails are the layer above this, and it helps to keep the words straight, because candidates blur them. Structured output is about format. Guardrails are about content: is the response on-topic, does it leak PII, does it stay inside policy. Tools like NeMo Guardrails and Guardrails AI run validators over inputs and outputs and can block or rewrite a response. A model can produce perfectly valid JSON that says something you’d never want sent to a customer, and only a content check stops that.
Asked to design the reliability stack for an agent that issues refunds or changes account settings, the strong answer layers these instead of picking one. Constrain the decoding so the shape is guaranteed, validate the values against business rules you don’t trust the model to enforce, run content guardrails on anything customer-facing, and define the fail-closed path for when a value can’t be trusted. The interesting engineering is deciding which checks earn their latency, because every layer is another few hundred milliseconds a customer waits.
Some questions that come up in these loops:
- “Your agent returns valid JSON but the numbers are sometimes wrong. Where do you look first?”
- “Walk me through what constrained decoding does at the logit level.”
- “A schema is generated per request and compilation is slow. How do you keep p99 latency sane?”
- “When would you choose not to force a strict schema on a model?”
That last one is the tell. The engineers who’ve lived with this know the schema is the easy half. Getting a model to emit well-formed output is close to a solved problem; getting it to emit the right output is an eval and prompt question that no decoder will hand you.
