client.index() to generate timestamped descriptions of your recordings. Choose driving for road footage or robotics for manipulation and subtask annotation.
index() submits the recordings and waits for their results. It returns a dictionary keyed by the video IDs you supplied. The default mode is "driving", so specify mode="robotics" for robot or handheld manipulation recordings.
Indexing uses your account’s credits. Calling
index() again submits a new request; it does not resume the previous SDK call. If a call times out, inspect the existing run before submitting again.Indexing driving footage
Usemode="driving" to generate time-bounded descriptions of road scenes, driving behavior, and interactions. Driving is the default mode.
grounded_caption list. An illustrative driving annotation:
top_k:
smart_filter=True is a separate driving-only screening option and cannot be combined with top_k.
Indexing robotics footage
Usemode="robotics" for robot or handheld manipulation recordings. Nomadic uses available linked camera views and supported motion signals to ground timestamped subtask annotations.
Supply subtasks for a new task
Usesubtask_definitions to customize how subtasks are annotated for new tasks in
your dataset. Define what each subtask means and where it starts and ends.
Pass a dictionary keyed by video ID. Each value is another dictionary mapping
subtask labels to descriptions. Use None if you have a label but no description.
For named tasks with a known arm layout, saved definitions are shared with all
members of your organization, independently of the source video’s sharing settings.
Without an organization, saved definitions are private to your account.
Inspect subtask annotations
Successful results containgrounded_caption: a list of time-bounded annotations. Times are in seconds relative to the video.
Category and arm fields are optional; robotics results without a reference taxonomy may contain only timestamps and captions.
Method
client.index(videos, **options)
Pass a list of video IDs as the first argument. All remaining parameters are optional keyword arguments.
Parameters
Leave filtering options at their defaults to annotate all selected recordings. With
top_k, the retained fraction can exceed the requested value because of heuristic additions or missing-evidence fallbacks; it is not a hard cost limit.
Return value
A dictionary keyed by the supplied video IDs. Each value contains astatus and, on success, a grounded_caption list. Failed results include error; timeout results also include error_code and run_id.
Failures and timeouts
Check each recording’s result independently. A completed result containsstatus="completed"; a failed result contains status="failed" and an error. Optional processing warnings may appear without failing the whole result.
When the SDK reaches its timeout, unfinished recordings return error_code="timeout" and run_id. The server may still be processing them. Open that run in Workbench rather than immediately submitting another indexing request. Once it finishes, retrieve the annotations with client.get_batch_analysis(run_id).
With optional filtering enabled, a successfully skipped recording can have an empty grounded_caption list.