Reliable Structured Outputs from LLMs Using JSON Schema

You've spent an afternoon crafting a prompt that asks for JSON, and the model returns a perfectly formatted markdown code block. Or it wraps your array in an extra object. Or it decides to rename the key firstName to first_name mid-stream. The fix isn't more prompt engineering — it's enforcing a contract at the API level.

Why structured outputs matter

Freeform text generation is the default for LLMs, and it's the wrong default for any system that needs to act on the output. Parsing natural language responses with regex is brittle. Changing a prompt slightly can change the output format of every downstream consumer. You end up with validation logic that rivals the complexity of the generation itself.

Structured outputs solve this by requesting machine-parseable JSON as part of the generation contract. When you need to call a function, insert into a database, or return a response from an API endpoint, the model should emit exactly the shape you define. No post-processing to strip markdown fences, no fallback parsing when the model decides to add commentary before the JSON.

JSON Schema as a constraint language

JSON Schema is a declarative language for validating the structure of JSON documents. It defines what fields are required, what types they accept, and what values are allowed via enums, patterns, or nested schemas. Several model providers now expose this as a native constraint during generation.

The simplest schema enforces presence and types. Here's a user profile schema that requires name (a string), age (an integer), and email (a string matching a basic pattern):

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["name", "age", "email"],
  "properties": {
    "name": { "type": "string" },
    "age": { "type": "integer", "minimum": 0 },
    "email": { "type": "string", "format": "email" }
  },
  "additionalProperties": false
}

The additionalProperties: false flag is critical — it tells the model that extra keys are not allowed, which some providers enforce during generation and others only validate after the fact.

Comparing provider implementations

Each provider implements structured outputs differently. The table below captures the key differences for the models you're most likely to use in production.

OpenAI's structured outputs with strict: true are the most developer-friendly option. You pass a JSON Schema directly in the API call, and the model guarantees the response adheres to it, including avoiding token generation outside the schema. This is not the same as the earlier json_object mode, which only guaranteed valid JSON — not valid structure.

from openai import OpenAI
 
client = OpenAI()
 
schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "tags": {"type": "array", "items": {"type": "string"}},
        "word_count": {"type": "integer"},
    },
    "required": ["title", "tags", "word_count"],
    "additionalProperties": False,
}
 
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "draft a blog outline about microservices"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "blog_outline",
            "strict": True,
            "schema": schema
        }
    }
)
 
print(response.choices[0].message.content)

Anthropic's approach uses function calling to enforce structure. You define the expected output as a tool, and the model responds with a tool_use block containing valid JSON. This pattern works well but adds latency because the model must decide to use the tool, then fill its parameters.

Local models benefit from the instructor library, which wraps any LLM provider and uses Pydantic models to validate responses. It doesn't constrain generation at the token level — it validates the output and retries if it fails. This is acceptable for many use cases but doesn't guarantee first-try correctness.

Handling edge cases and failures

Even with structured outputs, things go wrong. The most common failure modes:

Valid JSON, invalid schema. The model generates syntactically correct JSON but includes an extra field or uses a string where an integer was expected. OpenAI's strict mode catches this during generation. For other providers, you need a validation step:

import jsonschema
 
def validate_response(raw_json, schema):
    try:
        jsonschema.validate(instance=raw_json, schema=schema)
        return raw_json
    except jsonschema.ValidationError as e:
        # Log the error and optionally retry
        raise

Markdown-wrapped JSON. Some models emit triple-backtick fences around the JSON, especially when the schema is defined in the system prompt rather than via an API parameter. Always strip these before parsing, or use a parser that tolerates them.

Incomplete responses. On token limits or API errors, the model may truncate the JSON. Detect this by checking that the response ends with the closing brace or bracket of your root schema. A simple check is to attempt json.loads() and catch json.JSONDecodeError.

A robust retry strategy sends back the validation error as a user message:

Your previous response failed validation. The schema requires field "email" but the response contained "email_address". Please try again with the correct field name.

This feedback loop converges quickly for most models, though it adds latency.

Limits and gotchas

Schema complexity directly impacts model accuracy. Deeply nested schemas with $ref references or multiple levels of oneOf/anyOf confuse many models. The model can generate structurally correct JSON while semantically missing the intent. Keep schemas flat where possible.

Some providers enforce the schema only at the API level, not during generation. For example, Gemini's response_mime_type: "application/json" tells the model to emit JSON, but it doesn't constrain which keys appear. You still need validation on the client side.

Token overhead matters. A schema with 50 fields and nested objects adds 200–400 tokens to the request. For high-volume use cases, this adds cost and latency. Consider whether every field needs to be in the schema or if some can be handled with post-processing.

Model version updates can break structured output behavior. OpenAI's strict mode was introduced in a specific model iteration, and older models simply ignore the strict flag. Pin models in production and test after provider updates.

Testing and validating structured outputs

Structured outputs are deterministic enough to unit test. Create mock responses that match your schema and verify your validation logic handles them. For edge cases, craft responses that violate the schema and confirm your retry logic fires.

Integration tests against the actual provider API are more revealing. Use a fixture that calls the model with your schema and asserts the response passes jsonschema.validate(). Run these as part of your CI pipeline to catch provider-side breakage.

import pytest
import jsonschema
from your_app import get_structured_response
 
SCHEMA = {
    "type": "object",
    "properties": {
        "summary": {"type": "string"},
        "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
    },
    "required": ["summary", "sentiment"],
    "additionalProperties": False,
}
 
def test_sentiment_analysis_schema():
    result = get_structured_response("The product is amazing")
    jsonschema.validate(instance=result, schema=SCHEMA)

The same approach works for testing against local models or the instructor library. The key insight: your tests should assert schema compliance, not string equality.

Future directions: evolving standards

The current fragmentation across providers is unsustainable. A community-driven effort to standardize how schemas are passed to LLMs would simplify tooling. Vercel AI SDK and LangChain already abstract over provider differences, but they still need provider-specific adapters.

A portable schema definition — one schema file that works across OpenAI, Anthropic, and Gemini — is the logical next step. The OpenAPI specification already uses JSON Schema for describing API responses; extending this pattern to LLM outputs is natural.

Agent frameworks are also driving adoption. When an agent needs to call multiple tools with complex parameters, structured outputs are the only reliable way to orchestrate the flow. Expect provider support to improve as agent use cases dominate production traffic. For a similar shift in how we configure tooling, see how Tailwind CSS v4 without a config file rethinks defaults — structured outputs are undergoing a similar evolution from optional flag to first-class API.

Key takeaways

• JSON Schema is the portable contract for structured LLM outputs, and you should use additionalProperties: false on every object to prevent extra keys.
• OpenAI's strict mode provides the best enforcement guarantee during generation; for other providers, always validate the output with jsonschema and retry with feedback.
• Keep schemas flat — deeply nested structures degrade generation accuracy and increase token overhead.
• Test schema compliance in CI with integration tests against the actual provider API, not just unit tests with mocks.
• Pin model versions in production and test after provider updates, as structured output behavior can change between releases.