> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nomadicml.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Indexing videos

> Turn uploaded recordings into timestamped descriptions and robotics subtask annotations with the Nomadic SDK.

Use `client.index()` to generate timestamped descriptions of your recordings. Choose **driving** for road footage or **robotics** for manipulation and subtask annotation.

`index()` submits the recordings and waits for their results. It returns a dictionary keyed by the video IDs you supplied. The default mode is `"driving"`, so specify `mode="robotics"` for robot or handheld manipulation recordings.

> [!NOTE]
>
> Indexing uses your account’s credits. Calling `index()` again submits a new request; it does not resume the previous SDK call. If a call times out, inspect the existing run before submitting again.

## Indexing driving footage

Use `mode="driving"` to generate time-bounded descriptions of road scenes, driving behavior, and interactions. Driving is the default mode.

```python theme={null}
results = client.index(["driving_video_id_1", "driving_video_id_2"], mode="driving")
```

Each completed recording returns a `grounded_caption` list. An illustrative driving annotation:

```json theme={null}
{
  "t_start": 4.0,
  "t_end": 7.5,
  "caption": "The vehicle slows as a pedestrian crosses ahead."
}
```

To annotate every selected recording, leave the filtering options at their defaults. For optional novelty selection, pass a fraction with `top_k`:

```python theme={null}
results = client.index(video_ids, mode="driving", top_k=0.2)
```

This requests the most novel 20% of the recording cohort, with driving IMU heuristic additions and missing-evidence fallbacks. It is not a hard retention or cost limit. `smart_filter=True` is a separate driving-only screening option and cannot be combined with `top_k`.

## Indexing robotics footage

Use `mode="robotics"` for robot or handheld manipulation recordings. Nomadic uses available linked camera views and supported motion signals to ground timestamped subtask annotations.

```python theme={null}
results = client.index(["robotics_video_id_1", "robotics_video_id_2"], mode="robotics")
```

Select one primary video ID per recording; its linked cameras remain part of the recording. When a reference subtask taxonomy is configured for the embodiment and task, predictions include its subtask categories and acting arm. Ground-truth intervals are comparison data, not prediction inputs.

### Inspect subtask annotations

Successful results contain `grounded_caption`: a list of time-bounded annotations. Times are in **seconds relative to the video**.

```python theme={null}
for video_id, result in results.items():
    if result["status"] != "completed":
        print(video_id, result.get("error", "Indexing did not complete"))
        continue
    for subtask in result.get("grounded_caption", []):
        print(
            video_id,
            subtask["t_start"],
            subtask["t_end"],
            subtask.get("category", ""),
            subtask.get("acting_hand", ""),
            subtask["caption"],
        )
```

An illustrative robotics annotation:

```json theme={null}
{
  "t_start": 1.2,
  "t_end": 3.8,
  "category": "pick up object",
  "acting_hand": "left",
  "caption": "The left gripper lifts the object from the table."
}
```

| Field | Meaning |
| - | - |
| `t_start`, `t_end` | Start and end of the annotation in video-relative seconds. |
| `caption` | Natural-language description of what happens during the interval. |
| `category` | Subtask category when a reference taxonomy is available. |
| `acting_hand` | Acting arm or hand when provided by the robotics result. |
| `subtask_taxonomy_id` | Identifier of the reference taxonomy, when provided. |

Category and arm fields are optional; robotics results without a reference taxonomy may contain only timestamps and captions.

## Method

### `client.index(videos, **options)`

Pass a list of video IDs as the first argument. All remaining parameters are optional keyword arguments.

```python theme={null}
client.index(
    videos,
    timeout=3600,
    poll_interval=5,
    mode="driving",
    smart_filter=False,
    top_k=None,
)
```

### Parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `videos` | `Sequence[str \| Mapping]` | Required | A list of video ID strings or upload-result dictionaries containing `video_id`. |
| `mode` | `str` | `"driving"` | `"driving"` or `"robotics"`. |
| `timeout` | `float` | `3600` | Maximum time in seconds for the SDK to wait. Must be positive. |
| `poll_interval` | `float` | `5` | Seconds between status polls. Must be positive. |
| `smart_filter` | `bool` | `False` | Screen existing motion/scene indexes before captioning; driving only. |
| `top_k` | `float \| None` | `None` | Optional novelty-selection fraction from `0` to `1`, supported in driving and robotics modes. Cannot be combined with `smart_filter`. |

Leave filtering options at their defaults to annotate all selected recordings. With `top_k`, the retained fraction can exceed the requested value because of heuristic additions or missing-evidence fallbacks; it is not a hard cost limit.

### Return value

A dictionary keyed by the supplied video IDs. Each value contains a `status` and, on success, a `grounded_caption` list. Failed results include `error`; timeout results also include `error_code` and `run_id`.

## Failures and timeouts

Check each recording’s result independently. A completed result contains `status="completed"`; a failed result contains `status="failed"` and an `error`. Optional processing warnings may appear without failing the whole result.

When the SDK reaches its timeout, unfinished recordings return `error_code="timeout"` and `run_id`. **The server may still be processing them.** Open that run in Workbench rather than immediately submitting another indexing request. Once it finishes, retrieve the annotations with `client.get_batch_analysis(run_id)`.

With optional filtering enabled, a successfully skipped recording can have an empty `grounded_caption` list.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.