Datasets from Batches

A batch is a single run — one query, one schema, over some videos. A dataset is the thing you hand to someone else: a union of batch runs, materialised server-side into one download containing every batch’s full results.

Curated datasets are admin-only. If your account is not an admin the SDK says so rather than returning a bare 403; use client.video.get_batch_analysis(batch_id) to pull a single batch’s results instead.

Create one

Pass the batch you just ran, a batch id, or a list of either:

batch = client.analyze(video_ids, "is there debris in the lane?", output_schema=Debris)

dataset = client.datasets.create_from_batch(batch, name="debris-q3")
print(dataset.id, dataset.status, dataset.event_counts)

Several batches, plus raw footage

dataset = client.datasets.create_from_batches(
    [batch_a, "b_2f9c...", batch_c],
    name="construction-review",
    video_ids=["vid_1", "vid_2"],   # unanalysed extras, bundled with no events
    description="Q3 construction review set",
    scope="org",                     # visible to your organization
)

create_from_batch waits for the background build to finish by default, so dataset.event_counts is final when it returns. Pass wait=False to get the id immediately and poll later with client.datasets.wait(dataset_id). If a batch’s results cannot be read during the build, the dataset ends in error rather than ready and names the batches in dataset.failed_batches. Its download is still written, so nothing that did read is lost — but a dataset silently missing a whole batch should not present itself as complete. Retry with client.datasets.update(dataset_id, refresh=True).

What lands in it

The grain is batch-scoped, not video-scoped. For each batch, the backend reads exactly the analysis that batch produced for each of its videos. So:

A video’s events come from the batch you named — not from everything that video has ever been analysed for.
A video in two batches of the dataset appears twice, once per batch, each time with that batch’s events. That is deliberate: two batches usually asked different questions, and their rows are not comparable.
video_ids are unanalysed extras. They land in unanalyzed_videos as bare {video_id, filename} records with no events.

Read it back

download = client.datasets.download(dataset.id)          # or download(id, "out.json")

download.to_dataframe()                                   # one row per event, batch_id first
download.structured(as_model=Debris)                      # every structured payload, typed
download.events()                                         # every event, across every batch

Events keep whatever their run produced, so a batch that carried an output_schema brings its structured_output along and those fields become columns. A dataset can mix schema and non-schema batches — batches without one simply contribute no schema columns. If a schema field is itself called batch_id, the schema field keeps the column and the batch id moves to event_batch_id.

Manage it

client.datasets.list(scope="user")                        # or scope="org"
client.datasets.get(dataset_id)                           # live status while building
client.datasets.update(dataset_id, name="renamed", add_batches=[batch_d], wait=True)
client.datasets.update(dataset_id, remove_batches=["b_2f9c..."])
client.datasets.update(dataset_id, refresh=True, wait=True)   # rebuild counts and download

Any change to what is in the dataset rebuilds its download by default, and a rebuild is asynchronous — pass wait=True when you need final totals, or a working download, before reading it. A rename alone never triggers one. refresh=False skips the rebuild for video additions and for removals; it cannot skip one for add_batches, since the backend always reprocesses added batches to compute their event counts. A rename on its own does not rebuild: the dataset is renamed immediately, but the name embedded in an already-written download stays as it was until the next rebuild. Pass refresh=True alongside the rename if the download’s own metadata has to match.

​Create one

​What lands in it

​Read it back

​Manage it

Create one

What lands in it

Read it back

Manage it