> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nomadicml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets from Batches

> Bundle one or more finished batches into a durable, downloadable dataset

A batch is a single run — one query, one schema, over some videos. A **dataset**
is the thing you hand to someone else: a union of batch runs, materialised
server-side into one download containing every batch's full results.

<Note>
  Curated datasets are admin-only. If your account is not an admin the SDK says so
  rather than returning a bare 403; use `client.video.get_batch_analysis(batch_id)`
  to pull a single batch's results instead.
</Note>

## Create one

Pass the batch you just ran, a batch id, or a list of either:

```python theme={null}
batch = client.analyze(video_ids, "is there debris in the lane?", output_schema=Debris)

dataset = client.datasets.create_from_batch(batch, name="debris-q3")
print(dataset.id, dataset.status, dataset.event_counts)
```

```python title="Several batches, plus raw footage" theme={null}
dataset = client.datasets.create_from_batches(
    [batch_a, "b_2f9c...", batch_c],
    name="construction-review",
    video_ids=["vid_1", "vid_2"],   # unanalysed extras, bundled with no events
    description="Q3 construction review set",
    scope="org",                     # visible to your organization
)
```

`create_from_batch` waits for the background build to finish by default, so
`dataset.event_counts` is final when it returns. Pass `wait=False` to get the
id immediately and poll later with `client.datasets.wait(dataset_id)`.

If a batch's results cannot be read during the build, the dataset ends in
`error` rather than `ready` and names the batches in `dataset.failed_batches`.
Its download is still written, so nothing that did read is lost — but a dataset
silently missing a whole batch should not present itself as complete. Retry with
`client.datasets.update(dataset_id, refresh=True)`.

## What lands in it

The grain is **batch-scoped, not video-scoped**. For each batch, the backend
reads exactly the analysis that batch produced for each of its videos. So:

* A video's events come from the batch you named — not from everything that
  video has ever been analysed for.
* A video in two batches of the dataset appears **twice**, once per batch, each
  time with that batch's events. That is deliberate: two batches usually asked
  different questions, and their rows are not comparable.
* `video_ids` are unanalysed extras. They land in `unanalyzed_videos` as bare
  `{video_id, filename}` records with no events.

## Read it back

```python theme={null}
download = client.datasets.download(dataset.id)          # or download(id, "out.json")

download.to_dataframe()                                   # one row per event, batch_id first
download.structured(as_model=Debris)                      # every structured payload, typed
download.events()                                         # every event, across every batch
```

Events keep whatever their run produced, so a batch that carried an
`output_schema` brings its `structured_output` along and those fields become
columns. A dataset can mix schema and non-schema batches — batches without one
simply contribute no schema columns. If a schema field is itself called
`batch_id`, the schema field keeps the column and the batch id moves to
`event_batch_id`.

## Manage it

```python theme={null}
client.datasets.list(scope="user")                        # or scope="org"
client.datasets.get(dataset_id)                           # live status while building
client.datasets.update(dataset_id, name="renamed", add_batches=[batch_d], wait=True)
client.datasets.update(dataset_id, remove_batches=["b_2f9c..."])
client.datasets.update(dataset_id, refresh=True, wait=True)   # rebuild counts and download
```

Any change to *what is in* the dataset rebuilds its download by default, and a
rebuild is asynchronous — pass `wait=True` when you need final totals, or a
working download, before reading it. A rename alone never triggers one.

`refresh=False` skips the rebuild for video additions and for removals; it
cannot skip one for `add_batches`, since the backend always reprocesses added
batches to compute their event counts.

A rename on its own does not rebuild: the dataset is renamed immediately, but
the name embedded in an already-written download stays as it was until the next
rebuild. Pass `refresh=True` alongside the rename if the download's own metadata
has to match.