Indexing videos

Use client.index() to generate timestamped descriptions of your recordings. Choose driving for road footage or robotics for manipulation and subtask annotation. index() submits the recordings and waits for their results. It returns a dictionary keyed by the video IDs you supplied. The default mode is "driving", so specify mode="robotics" for robot or handheld manipulation recordings.

Indexing uses your account’s credits. Calling index() again submits a new request; it does not resume the previous SDK call. If a call times out, inspect the existing run before submitting again.

Indexing driving footage

Use mode="driving" to generate time-bounded descriptions of road scenes, driving behavior, and interactions. Driving is the default mode.

results = client.index(["driving_video_id_1", "driving_video_id_2"], mode="driving")

Each completed recording returns a grounded_caption list. An illustrative driving annotation:

{
  "t_start": 4.0,
  "t_end": 7.5,
  "caption": "The vehicle slows as a pedestrian crosses ahead."
}

To annotate every selected recording, leave the filtering options at their defaults. For optional novelty selection, pass a fraction with top_k:

results = client.index(video_ids, mode="driving", top_k=0.2)

This requests the most novel 20% of the recording cohort, with driving IMU heuristic additions and missing-evidence fallbacks. It is not a hard retention or cost limit. smart_filter=True is a separate driving-only screening option and cannot be combined with top_k.

Indexing robotics footage

Use mode="robotics" for robot or handheld manipulation recordings. Nomadic uses available linked camera views and supported motion signals to ground timestamped subtask annotations.

results = client.index(["robotics_video_id_1", "robotics_video_id_2"], mode="robotics")

Select one primary video ID per recording; its linked cameras remain part of the recording. When a reference subtask taxonomy is configured for the embodiment and task, predictions include its subtask categories and acting arm. Ground-truth intervals are comparison data, not prediction inputs.

Inspect subtask annotations

Successful results contain grounded_caption: a list of time-bounded annotations. Times are in seconds relative to the video.

for video_id, result in results.items():
    if result["status"] != "completed":
        print(video_id, result.get("error", "Indexing did not complete"))
        continue
    for subtask in result.get("grounded_caption", []):
        print(
            video_id,
            subtask["t_start"],
            subtask["t_end"],
            subtask.get("category", ""),
            subtask.get("acting_hand", ""),
            subtask["caption"],
        )

An illustrative robotics annotation:

{
  "t_start": 1.2,
  "t_end": 3.8,
  "category": "pick up object",
  "acting_hand": "left",
  "caption": "The left gripper lifts the object from the table."
}

Field	Meaning
`t_start`, `t_end`	Start and end of the annotation in video-relative seconds.
`caption`	Natural-language description of what happens during the interval.
`category`	Subtask category when a reference taxonomy is available.
`acting_hand`	Acting arm or hand when provided by the robotics result.
`subtask_taxonomy_id`	Identifier of the reference taxonomy, when provided.

Category and arm fields are optional; robotics results without a reference taxonomy may contain only timestamps and captions.

Method

`client.index(videos, **options)`

Pass a list of video IDs as the first argument. All remaining parameters are optional keyword arguments.

client.index(
    videos,
    timeout=3600,
    poll_interval=5,
    mode="driving",
    smart_filter=False,
    top_k=None,
)

Parameters

Parameter	Type	Default	Description
`videos`	`Sequence[str \| Mapping]`	Required	A list of video ID strings or upload-result dictionaries containing `video_id`.
`mode`	`str`	`"driving"`	`"driving"` or `"robotics"`.
`timeout`	`float`	`3600`	Maximum time in seconds for the SDK to wait. Must be positive.
`poll_interval`	`float`	`5`	Seconds between status polls. Must be positive.
`smart_filter`	`bool`	`False`	Screen existing motion/scene indexes before captioning; driving only.
`top_k`	`float \| None`	`None`	Optional novelty-selection fraction from `0` to `1`, supported in driving and robotics modes. Cannot be combined with `smart_filter`.

Leave filtering options at their defaults to annotate all selected recordings. With top_k, the retained fraction can exceed the requested value because of heuristic additions or missing-evidence fallbacks; it is not a hard cost limit.

Return value

A dictionary keyed by the supplied video IDs. Each value contains a status and, on success, a grounded_caption list. Failed results include error; timeout results also include error_code and run_id.

Failures and timeouts

Check each recording’s result independently. A completed result contains status="completed"; a failed result contains status="failed" and an error. Optional processing warnings may appear without failing the whole result. When the SDK reaches its timeout, unfinished recordings return error_code="timeout" and run_id. The server may still be processing them. Open that run in Workbench rather than immediately submitting another indexing request. Once it finishes, retrieve the annotations with client.get_batch_analysis(run_id). With optional filtering enabled, a successfully skipped recording can have an empty grounded_caption list.

​Indexing driving footage

​Indexing robotics footage

​Inspect subtask annotations

​Method

​client.index(videos, **options)

​Parameters

​Return value

​Failures and timeouts

Indexing driving footage

Indexing robotics footage

Inspect subtask annotations

Method

`client.index(videos, **options)`

Parameters

Return value

Failures and timeouts