Structured Outputs and Validation

Structured output asks a model to produce data that conforms to a schema, allowing application code to consume fields instead of parsing prose. Schema conformance is necessary but insufficient: the harness must still enforce business rules, authorization, referential integrity, freshness, and safe behavior when output is missing or invalid.

What will you be able to do?

You will define a typed triage decision, validate syntactic and semantic constraints, and design a bounded repair or rejection path.

What is the mental model?

A schema is a form, not a fact checker. It can require a priority field and restrict its values, but it cannot prove that a ticket really is urgent or that an order belongs to the current tenant.

How do you define the SupportOps contract?

domain.py:

python

from enum import Enum
from pydantic import BaseModel, Field


class Priority(str, Enum):
    LOW = "low"
    NORMAL = "normal"
    HIGH = "high"


class Citation(BaseModel):
    source_id: str = Field(min_length=1)
    claim: str = Field(min_length=1, max_length=300)


class TriageDecision(BaseModel):
    category: str = Field(min_length=1, max_length=80)
    priority: Priority
    summary: str = Field(min_length=1, max_length=500)
    citations: list[Citation] = Field(max_length=5)
    needs_human: bool
    proposed_action: str | None = Field(default=None, max_length=300)

Then apply rules that a JSON schema cannot express cleanly:

python

def validate_decision(decision: TriageDecision) -> None:
    if decision.priority is Priority.HIGH and not decision.needs_human:
        raise ValueError("High-priority tickets require human review")
    if decision.proposed_action and not decision.citations:
        raise ValueError("An action proposal requires evidence")

What should happen when validation fails?

Choose deliberately:

Reject immediately when the output indicates unsafe intent.
Retry once with machine-readable validation errors for correctable formatting.
Route to a human when uncertainty or missing evidence cannot be repaired.
Stop after the repair budget; never loop until the output happens to pass.

Record the original output securely, validation errors, repair attempt, and final disposition.

How do you prove it works?

Write deterministic tests before calling a model:

python

import pytest


def test_high_priority_requires_human() -> None:
    decision = TriageDecision(
        category="billing",
        priority="high",
        summary="Possible duplicate charge",
        citations=[{"source_id": "kb-7", "claim": "Escalate duplicates"}],
        needs_human=False,
    )
    with pytest.raises(ValueError, match="human review"):
        validate_decision(decision)

Also test extra fields, overlong strings, unknown enum values, empty citations, and cross-tenant source IDs.

Failure injection: Return valid JSON that references another tenant’s document. Schema validation passes. Authorization-aware semantic validation must fail. This demonstrates why “valid structure” is not “safe action.”

What changes in production?

Version schemas, store the version with each outcome, use backward-compatible consumers, and measure validation/repair rates. A sudden increase often indicates prompt drift, a model change, or corrupted context.

What mistakes should you avoid?

Parsing model prose with a fragile regular expression.
Treating enum selection as ground truth.
Retrying invalid output without a limit.
Making every field optional to avoid validation failures.
Executing an action directly from model output without policy checks.

Check your understanding

What can a schema prove, and what can it not prove?
When should invalid output be repaired versus escalated?
Why must schema versions be stored with outcomes?

Expert extension

Add a versioned migration from TriageDecisionV1 to V2 that introduces confidence evidence without breaking replay of old traces. Write compatibility tests.

Official references

Structured Outputs and Validation

How do you define the SupportOps contract?

domain.py:

python

from enum import Enum
from pydantic import BaseModel, Field


class Priority(str, Enum):
    LOW = "low"
    NORMAL = "normal"
    HIGH = "high"


class Citation(BaseModel):
    source_id: str = Field(min_length=1)
    claim: str = Field(min_length=1, max_length=300)


class TriageDecision(BaseModel):
    category: str = Field(min_length=1, max_length=80)
    priority: Priority
    summary: str = Field(min_length=1, max_length=500)
    citations: list[Citation] = Field(max_length=5)
    needs_human: bool
    proposed_action: str | None = Field(default=None, max_length=300)

Then apply rules that a JSON schema cannot express cleanly:

python

def validate_decision(decision: TriageDecision) -> None:
    if decision.priority is Priority.HIGH and not decision.needs_human:
        raise ValueError("High-priority tickets require human review")
    if decision.proposed_action and not decision.citations:
        raise ValueError("An action proposal requires evidence")

What should happen when validation fails?

Choose deliberately:

Reject immediately when the output indicates unsafe intent.

Retry once with machine-readable validation errors for correctable formatting.

Route to a human when uncertainty or missing evidence cannot be repaired.

Stop after the repair budget; never loop until the output happens to pass.

Record the original output securely, validation errors, repair attempt, and final disposition.

How do you prove it works?

Write deterministic tests before calling a model:

python

import pytest


def test_high_priority_requires_human() -> None:
    decision = TriageDecision(
        category="billing",
        priority="high",
        summary="Possible duplicate charge",
        citations=[{"source_id": "kb-7", "claim": "Escalate duplicates"}],
        needs_human=False,
    )
    with pytest.raises(ValueError, match="human review"):
        validate_decision(decision)

Also test extra fields, overlong strings, unknown enum values, empty citations, and cross-tenant source IDs.

Failure injection: Return valid JSON that references another tenant’s document. Schema validation passes. Authorization-aware semantic validation must fail. This demonstrates why “valid structure” is not “safe action.”