Structured Outputs
AI Engineer path · lesson 59 of 124Make a model emit something a program can parse. Step the five layers from prompting to finetuning, watch a grammar mask the logit vector so an invalid token becomes unreachable rather than unlikely, and see how far LinkedIn's defensive parser carried them.
Sure! Here is the JSON you asked for:
```json
{ "name": "Ada", "city": "London", }
```- • A prose preamble before the object
- • A markdown fence wrapping it
- • A trailing comma, which is not valid JSON
JSON.parse. That is the character of this problem: the content was right and the request still failed.Key insight
Reported rates of correctly generated JSON objects range anywhere from 0% to the high 90s depending on the application and the model. A range that wide is not a detail to tune later — it is the difference between a feature and an outage, so it belongs in your design rather than your backlog.
How strong is this guarantee?
Sort each technique by whether it can still return invalid JSON.
Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.
Best effort — can still fail
Drop items hereValid by construction
Drop items hereTest your knowledge
Which task inherently requires structured output rather than merely preferring it?
The short version
A form versus a blank sheet. Asking politely for an address gets you an address in whatever shape the writer felt like. A form with boxes gets you the same information in the same place every time — constrained sampling is the box, and everything above it is asking more politely.
In more depth
Structured Outputs
A language model emits text. A program needs a contract.
Two situations that genuinely need it
Five layers, by what they guarantee
| # | Layer | Kind | Guarantees |
|---|---|---|---|
| 1 | Prompting | bandage | Nothing |
| 2 | Post-processing | bandage | Nothing, but recovers most failures |
| 3 | Test time compute | bandage | Nothing, but raises the odds |
| 4 | Constrained sampling | treatment | Format is structurally guaranteed |
| 5 | Finetuning | treatment | Strongest and most general |
The first three make failure rarer. Only the last two change what the model can emit.
Constrained sampling
Mask the logit vector to the tokens a grammar permits in the current state, then sample. Because masking precedes the softmax, illegal tokens have probability exactly zero — temperature cannot resurrect them. Costs: a grammar per format (poor generality), and grammar verification can increase generation latency. It guarantees the container, never the contents.
Post-processing works better than it sounds
LinkedIn's defensive YAML parser took correct outputs from 90% to 99.99% (Bottaro and Ramgopal, 2020), because model format errors are highly repetitive. They chose YAML over JSON because it is less verbose — fewer output tokens.
JSON mode
OpenAI introduced it first. It typically guarantees valid JSON, not the content, and the output can still be truncated by max token length — and truncated JSON is not valid JSON.
The classification shortcut
If the output is one label from a fixed set, append a classifier head (feature-based transfer). The class is guaranteed by the architecture.
Interview Questions
References & Further Reading
Love this visualization?
Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.
100% free • No credit card required