Structured Output

Structured output lets you declare the shape of the answer before the run — “verdict is one of object/no/uncertain, confidence is one of low/medium/high” — instead of describing it in your prompt and parsing it back out of free text. The schema is enforced on our side: it is compiled into the validator’s tool schema (and, on the Fast detection path, into the model’s response schema), so a required field cannot be dropped and an enum value cannot be invented. Every detected event comes back with a structured_output payload that satisfies it.

Declare the schema

Use a Pydantic model if you have one — the class name becomes the schema label:

Schema from a Pydantic model

from typing import Literal, Optional
from pydantic import BaseModel, Field

class Debris(BaseModel):
    verdict: Literal["object", "no", "uncertain"]
    confidence: Literal["low", "medium", "high"] = Field(description="How sure you are")
    blocking_lane: bool
    notes: Optional[str] = None

Or declare the fields directly, with no Pydantic dependency:

Schema from field dicts

schema = [
    {"name": "verdict", "type": "enum", "values": ["object", "no", "uncertain"]},
    {"name": "confidence", "type": "enum", "values": ["low", "medium", "high"]},
    {"name": "blocking_lane", "type": "boolean"},
    {"name": "notes", "type": "text", "required": False},
]

Field types are enum, text, number, boolean, and time_range (a clip-relative mm:ss-mm:ss window). A schema takes at most 12 fields, and an enum at most 64 values. From a Pydantic model, Literal[...] and Enum subclasses become enum, Optional[...] becomes an optional field, and List[...] becomes a repeated field. A time_range has no JSON Schema equivalent, so ask for one explicitly:

windows: list[str] = Field(json_schema_extra={"nomadic_type": "time_range"})

If your query already spells out the contract in prose, let the backend draft the schema and edit what comes back:

schema = client.video.infer_schema("is there debris? answer object/no/uncertain")

Run the analysis

Pass the schema to the analyze() call you already make. It is validated locally first, so a bad field name fails immediately rather than per-video at inference time.

Prompt analysis with a schema

batch = client.analyze(
    ["video_id_1", "video_id_2"],
    "is there debris in the lane?",
    output_schema=Debris,
)

Structured output is enforced through the batch a run creates. It is supported on prompt analyses in the default mode="thinking" (any number of videos), and on Ask-style analysis_type runs (ASK, CUSTOM_AGENT) of two or more videos — for a single video there, pass the id in a list. It is not available in mode="fast", on edge-case agents, or on action segmentation, which produces its own fixed schema.

Read the results

The result is still the same dictionary you get today, with two accessors added.

Typed rows

df = batch.to_dataframe()
df[df.verdict == "object"]

to_dataframe() returns one row per event: the video and time metadata (video_id, event_id, t_start, t_end, label, approval, …) followed by one column per schema field. Pass only_structured=True to drop events that carry no payload. If a schema field shares a name with a metadata column, the schema field wins the column and the metadata moves to event_<name>. The batch remembers the schema it was created with, so every declared field is a column even on a run where nothing filled it — df.verdict gives you an empty column rather than a KeyError.

Typed objects

for row in batch.structured(as_model=Debris):
    print(row.verdict, row.confidence)

structured() returns the raw payload dicts, or instances of your model when you pass as_model=. Events with no payload are skipped. Both accessors also work on a batch you fetch later:

batch = client.video.get_batch_analysis("batch_id")
batch.to_dataframe()

to_dataframe() needs pandas (pip install pandas); structured() and everything else do not.

​Declare the schema

​Run the analysis

​Read the results

Declare the schema

Run the analysis

Read the results