Structured Output from LLMs

Agentic systems tend to fall apart at the parsing layer before they fall apart anywhere else. A router needs a category; the model returns a paragraph containing the category. An orchestrator needs a task list; the model returns prose with bullets that sometimes use hyphens and sometimes use asterisks. An evaluator needs a score between zero and one; the model returns "around 0.8 or so." Each of these responses is coherent English and useless as input to a program.

Structured output solves this specific problem. The model is constrained, at decoding time, to produce output that conforms to a declared schema. The harness receives a typed object rather than a string, and the surrounding code can treat the call as an ordinary function. OpenAI shipped a production-grade version in 2024 (OpenAI, 2024); Anthropic and Google expose equivalent capabilities. Every pattern covered later in this series assumes structured output is available. Building without it means accepting unbounded parsing failures.

A ladder of reliability

The reference guide for this series names four tiers for output parsing, ordered by reliability (agentic-patterns.md):

• Free text plus regex. Fragile. Breaks on paraphrase, on a model that starts its response with "Sure, here's" one day and "I'd say" the next.
• XML extraction. Tolerant of minor variation, but still ad hoc. Works well enough for content authoring; poorly for machine-to-machine handoff.
• JSON mode. The model is constrained to emit valid JSON. Parses reliably, but the shape of the JSON is up to the model.
• Structured output. The model is constrained to match a declared schema. Full type safety. The only suitable choice for agentic systems with more than one step.

Only the last tier is appropriate for the patterns in this series. Every article below this level reduces to a discussion of how to recover from a parser that sometimes breaks.

The shape of a schema call

flowchart LR
    IN([Prompt + schema]) --> LLM[LLM with constrained decoding]
    LLM --> OBJ[Typed object matching schema]
    OBJ --> CODE[Program logic: routing, planning, evaluation]
    OBJ -.fields.-> F1[category: Literal]
    OBJ -.fields.-> F2[confidence: float]
    OBJ -.fields.-> F3[reasoning: str]

The schema lives alongside the prompt. The decoder enforces it. The output is an object whose fields are immediately usable. The diagram elides validation retry and partial streaming; both are features of production SDKs but do not change the shape of the interface.

The excerpt below shows structured output without a framework. The schema is a Pydantic model. The Responses API accepts it directly and returns an instance of that model.

from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Literal

client = OpenAI()

class TicketCategory(BaseModel):
    category: Literal["billing", "technical", "general", "urgent"]
    confidence: float = Field(ge=0.0, le=1.0)
    reasoning: str = Field(description="Brief explanation of the classification")

def classify_ticket(text: str) -> TicketCategory:
    response = client.responses.parse(
        model="gpt-4o-mini",
        instructions="Classify the ticket into exactly one category.",
        input=text,
        text_format=TicketCategory,
    )
    return response.output[0].content[0].parsed

The call returns a TicketCategory instance. The category field is a Literal and can drive a match statement without validation. The confidence field is a float in the declared range. The reasoning field carries an explanation the downstream system can log or surface.

The LangChain version is shorter. The with_structured_output binding wraps any chat model with a schema, and the invocation returns an instance directly.

from langchain.chat_models import init_chat_model
from pydantic import BaseModel, Field
from typing import Literal

class TicketCategory(BaseModel):
    category: Literal["billing", "technical", "general", "urgent"]
    confidence: float = Field(ge=0.0, le=1.0)
    reasoning: str

model = init_chat_model("gpt-4o-mini").with_structured_output(TicketCategory)

def classify_ticket(text: str) -> TicketCategory:
    return model.invoke([
        ("system", "Classify the ticket into exactly one category."),
        ("user", text),
    ])

Both excerpts produce the same typed object. Full runnable versions live at github.com/subodhjena/agentic-patterns under examples/02_structured_output.py, including classification, planning, evaluation, and nested schemas.

What the schema does not capture

Structured output reliably produces parseable objects. The failure modes live at the layer above.

The schema was wrong. A schema that allows a field the business logic cannot handle, or omits a field the model was likely to produce, will produce outputs that parse but do not match reality. Treat schema evolution as first-class, not as a one-time declaration.

The prompt and the schema disagreed. A prompt asking the model to "explain in detail" alongside a schema with a one-sentence summary field produces either truncated reasoning or a misused field. Prompt and schema must be designed together.

The model refused. Safety filters can cause an empty or refusal response even when a schema is declared. Production code must handle this as a distinct failure from parse errors.

The schema was too strict. An enum that does not cover the actual taxonomy of inputs forces the model to pick the nearest label silently. A Literal over three categories is fine; a Literal over fifty is usually wrong. When the space is large, use a free-text field plus a separate validator.

Cross-field invariants are invisible to the schema. A schema can declare that score is a float between zero and one but cannot declare that score should correlate with weaknesses. Validate cross-field invariants after parsing. The schema is not a full contract; it is a syntactic guarantee.

When the trade is worth making

Structured output loses a small amount of latitude for the model in exchange for a large reduction in surrounding complexity.

Axis	Free text	JSON mode	Structured output
Parse reliability	Low, regex-dependent	Medium, schema discipline required	High, enforced by decoder
Developer overhead	High, growing parsers	Medium, per-call prompt engineering	Low, schema declared once
Model expressiveness	Full	Full within JSON	Constrained to schema
Latency per call	Baseline	Baseline	Slightly higher in some providers
Evaluability	Hard to compare outputs	Easier, but noisy	Easy, field-level comparison
Composability with code	Low	Medium	High

For any step beyond a single free-text response, this is the right trade. Structured output is worth reaching for even in simple cases: entity extraction, intent detection, form filling, and field-level answers all benefit from the contract, not just the more elaborate patterns downstream.

Where this becomes load-bearing

Routing, covered later in the Workflows stage, is the simplest pattern that depends on structured output. It uses a Literal category field to dispatch. Orchestrator-workers depends on the model producing a structured task list. Evaluator-optimizer depends on a structured evaluation with a score and a verdict that drives the iteration. The agent-computer interface article covers tool definitions, which are the dual of structured output: the schema describes the arguments the model can emit rather than the response it must produce.

Every one of these patterns treats the LLM call as a typed function. That habit of thought is the genuine contribution of structured output; the SDK feature is the afterthought.

References

1. OpenAI. Introducing structured outputs in the API. August 2024.
2. Anthropic. Tool use and JSON mode. 2024.
3. Pydantic. Pydantic documentation: models. 2024.
4. LangChain. Structured outputs. 2024.
5. JSON Schema. JSON Schema specification. 2020.