Skip to main content

​
Nomadic → LeRobot: Sub-Task Annotation for GR00T Fine-Tuning

Consider the steps required for a robot to complete a long-horizon task like “make me a cup of coffee”: grab the pod, insert it, place the cup, press the button, wait, pick up the finished cup. If your training data labels the whole demonstration with that single sentence, the policy never sees where one sub-skill ends and the next begins. “Insert the pod” and “pick up the cup” get the exact same language conditioning as everything in between. That flat labeling throws away the sub-task structure that makes long-horizon tasks learnable, and it makes failures hard to diagnose since you can’t tell which sub-skill broke. Nomadic’s action segmentation finds where each sub-task starts and ends in raw footage and gives each one its own language label, turning a flat “make me a cup of coffee” trajectory into a structured, multi-phase episode a policy can actually learn from. This guide walks through that pipeline end to end:
  1. Fit a segmentation model on your fleet data to adapt sub-task boundaries to your specific robot embodiment and task.
  2. Run automated sub-task annotation on your videos to find sub-task boundaries and language-label each one.
  3. Export to a LeRobot dataset ready for GR00T fine-tuning, with each source video as one episode and sub-tasks written as a per-frame task_index timeline inside it.
  4. Finetune GR00T on the exported dataset.
The exported format is LeRobot v2.1 (meta/info.json, meta/tasks.jsonl, meta/modality.json, per-episode parquet + mp4), the format GR00T fine-tuning expects. Run this walkthrough end to end in Google Colab.

​
Setup

from nomadic import NomadicAI
from nomadic.video import AnalysisType

client = NomadicAI(api_key="your_api_key")

​
1. Fit a segmentation model on your fleet data

With Nomadic, you can easily train a sub-task segmentation model that uses your embodiment’s common motion patterns to learn its sub-tasks. Train from a pre-computed trajectory NPZ (external_data=). Point this at your own fleet’s .npz, either a local path on disk or a gs:// URI that Nomadic can read. NPZ schema. The file must contain these four arrays:
KeyShape / typeDescription
signal(n_channels, T), floatPer-channel trajectory signal
t_sec(T,), floatTimestamp per sample, in seconds
sample_rate_hzfloat scalarSampling rate, e.g. 50.0
embodimentstr"droid_manipulator" or "abc130k_bimanual"
Train a segmentation model from your fleet trajectory NPZ
# Local paths are uploaded to Nomadic-managed storage automatically;
# gs:// URIs are passed through as-is.
trajectory_npz = "./my_fleet_trajectories.npz"

# `preset="robotics"` gives tuned defaults for manipulator-style ~50-60 Hz data.
segmenter_job = client.train_segmenter(
    name="my-manipulator-segmenter",
    external_data=trajectory_npz,
    preset="robotics",
)
Poll until training completes: 
import time

segmenter_status = client.get_segmenter_status(segmenter_job["job_id"])
while segmenter_status["status"] not in {"completed", "failed"}:
    print("segmenter training:", segmenter_status["status"])
    time.sleep(15)
    segmenter_status = client.get_segmenter_status(segmenter_job["job_id"])
get_segmenter_status returns status, segmenter_id (populated once status == "completed"), created_at / completed_at, and error on failure.

​
2. Run automated sub-task annotation

With the model trained, pass its ID into analyze() alongside AnalysisType.ACTION_SEGMENTATION. Sub-task boundaries now come from the trajectory phases the model learned.
Analyze videos with your trained segmentation model
if segmenter_status["status"] != "completed":
    raise RuntimeError(f"Segmenter training failed: {segmenter_status.get('error')}")

segmenter_id = segmenter_status["segmenter_id"]

folder_videos = client.my_videos(folder="YOUR_FOLDER_NAME", scope="org")
videos = [v["video_id"] for v in folder_videos[:10]]

segmentation_result = client.analyze(
    videos,
    analysis_type=AnalysisType.ACTION_SEGMENTATION,
    segmenter_id=segmenter_id,
)

batch_id = segmentation_result["batch_metadata"]["batch_id"]
segmenter_id is accepted by client.analyze(...) for single videos, lists of video IDs, and folder/batch calls when analysis_type=AnalysisType.ACTION_SEGMENTATION.

​
Review the segmented sub-tasks

Each event is a short manipulation sub-task with a natural-language label. Rather than treating each span as its own clip, the export step below stitches these spans back onto the full video as a per-frame sub-task timeline, giving VLAs like GR00T the language conditioning signal they train on without breaking the continuous demonstration apart. 
client.visualize(segmentation_result, width=920)

batch_results = client.get_batch_analysis(batch_id)
for entry in batch_results["results"][:2]:
    print(entry["video_id"], "-", len(entry.get("events", [])), "segments")
    for event in entry.get("events", [])[:5]:
        print("   ", event.get("t_start"), "-", event.get("t_end"), ":", event.get("label"))

​
3. Export to a LeRobot dataset

By default, each source video becomes one LeRobot episode, and each Nomadic-annotated sub-task is included as a separate task_index inside that episode. If you prefer the sub-tasks to be their own episodes, pass episode_mode="per_segment".
Export a LeRobot v2.1 dataset for GR00T fine-tuning
export_result = client.export_lerobot_dataset(
    batch_id=batch_id,
    output_dir="./lerobot_dataset",
    trajectory_tool="manipulator_trajectory",
    camera_key="exterior",
    robot_type="franka",
)
Required parameters:
ParameterTypeDescription
batch_id or resultsstr / dictAn action-segmentation batch ID, or an already-fetched get_batch_analysis() payload (mutually exclusive)
output_dirstrLocal directory to write the dataset into
Returns: output_dir, num_episodes, num_frames, num_tasks, tasks, fps, state_dim, state_names, skipped_segments, warnings. Requires: pip install 'nomadic[lerobot]' (numpy, pandas, pyarrow) and the ffmpeg / ffprobe binaries on PATH. Re-running against the same output_dir wipes and regenerates it. If skipped_segments isn’t empty, those source videos didn’t have a completed manipulator_trajectory artifact to draw proprioception from. Point the pipeline at your own manipulator footage with trajectory imported, or pass trajectory_tool=None to export a video-only dataset (no observation.state / action features).

​
Inspect the exported dataset

The layout matches what GR00T’s fine-tuning pipeline expects: meta/modality.json maps the flattened trajectory channels to named state/action groups, and each episode is a (parquet, mp4) pair. meta/episodes.jsonl’s tasks list and the parquet’s task_index column are where the sub-task annotation actually lives. Check that task_index changes over the course of the episode where you’d expect a sub-task transition. 
import json
from pathlib import Path
import pandas as pd

dataset_root = Path(export_result["output_dir"])

info = json.loads((dataset_root / "meta" / "info.json").read_text())
modality = json.loads((dataset_root / "meta" / "modality.json").read_text())

episode_df = pd.read_parquet(dataset_root / "data" / "chunk-000" / "episode_000000.parquet")
episode_df.head()

​
4. Finetune GR00T

The exported directory is ready to hand to NVIDIA’s fine-tuning workflow:
  1. Copy/mount ./lerobot_dataset where your GR00T fine-tuning environment can read it.
  2. Follow the real-robot fine-tuning guide. The exported meta/modality.json is auto-generated in the same spirit as the new_embodiment_config_defaults.py GR00T generates for its own data-collection pipeline; review and adjust the per-group absolute/rotation_type settings for your embodiment before training.
  3. Launch launch_finetune.py pointed at ./lerobot_dataset.
Happy fine-tuning!