Egocentric Manipulation Episodes for Robot Learning
First-person footage of real hand work turned into robot-ready training episodes - per-frame actions, pixel-accurate masks, and 7-D end-effector state, exported as LeRobot and RLDS.
The challenge
Manipulation policies need demonstration data that pairs what was seen with what the hands did at every timestep. Robot teleoperation produces exactly that, but scales poorly - every sample costs robot time, an operator, and a lab, and the resulting data covers only the environments the lab can stage.
Human first-person video carries the same observation-action structure at real-world variety and speed, but only if three hard problems are solved: the hands and gestures must be extracted precisely enough to serve as an action signal, the footage must be segmented into clean start-to-finish task episodes, and everything that is not the task - breaks, interruptions, camera fumbles - must be kept out of the training set.
Most annotation vendors can draw boxes on video. Almost none can produce a per-frame end-effector state, align it with transcripts and action narration, and ship the result in the episode formats robot-learning frameworks actually load.
Why this data matters
The robotics field is converging on a simple observation: the biggest robot-learning datasets in existence are dwarfed by what humans record of themselves doing manual work. Recent robotics foundation models train on egocentric human video alongside robot data precisely because human demonstrations are the only manipulation data source that scales to real-world diversity.
Egocentric footage is uniquely suited to this: a head-mounted camera sees the workspace, the objects, and the hands interacting with them from nearly the same viewpoint a humanoid or arm-mounted camera would occupy. The gap between 'what the person saw' and 'what the robot will see' is smaller than for any other capture setup.
The bottleneck has never been recording - it is turning hours of raw, messy footage into structured, curated episodes. That conversion is exactly what Payana's pipeline automates.
How we did it
- 1
Wearable capture
A worker wears a head-mounted camera and simply performs the task; voice notes record intent as it happens. No scripting, no staging, no lab - the data inherits the noise and variety of the real workplace, which is what policies need to generalize.
- 2
Episode pipeline
The 7-model GPU stack annotates every second: a vision-language model narrates the action, detection and open-vocabulary models find every object and tool in frame, segmentation traces pixel-accurate masks, and speech becomes time-aligned transcripts.
- 3
Hand & gesture state
Hand-pose estimation extracts a 7-dimensional end-effector state per frame - 3D position, 3D orientation, and gripper aperture - the same action space most manipulation policies are trained in. Gestures like reach, grasp, transport and release become measurable trajectories instead of vague labels.
- 4
Automatic task segmentation
Long recordings of repeated work are split into individual task cycles automatically, so one shift of continuous footage becomes hundreds of discrete training episodes with clean starts and ends.
- 5
Curation
An on-task scoring model flags every second that is not the task - the coffee break, the conversation, the camera adjustment - before it can contaminate an episode.
- 6
Human review
Every episode passes a review workspace with annotated playback and per-second quality timelines. A reviewer approves, corrects, or excludes; every decision is tracked.
Inside the dataset
- Synchronized egocentric frames with per-frame observation-action pairs
- 7-D end-effector state per frame (position, orientation, gripper aperture)
- Object detections and pixel-accurate masks for everything the hands touch
- Action narration from a vision-language model, per second
- Time-aligned speech transcripts carrying worker intent
- Episode boundaries from automatic task-cycle segmentation
Quality control
Robot-learning data has a failure mode most datasets don't: a policy will happily learn from garbage, and you only find out after an expensive training run. So quality control happens before export - automatic curation scores every second for task relevance, human reviewers watch every episode with annotations overlaid, and reviewer overrides are tracked so the dataset's provenance is auditable.
Episodes ship with quality metrics designed to predict policy performance: coverage of the task's step distribution, annotation confidence distributions, and the fraction of frames that passed human review unchanged.
What was delivered
- Observation-action episodes with synchronized frames and per-frame state
- Gesture and hand-object interaction labels on every second
- Off-task segments excluded by curation + human review
- Fixed train/valid/test splits, versioned and reproducible
Exported as LeRobot v3 (loadable directly by the LeRobot library) and RLDS / Open X-Embodiment style episodes for TFDS pipelines, plus Parquet and JSONL for custom loaders.
What the data enables
- Imitation-learning / behavior-cloning training for manipulation policies
- Pre-training corpora for robotics foundation models (human video + robot data co-training)
- Task-decomposition and affordance datasets for hierarchical policies
- Evaluation suites: held-out human demonstrations as ground truth for policy behavior
Want a dataset like this?
Send us your images or video - we annotate, review, and deliver a training-ready dataset in your format.
Book an intro call