In short
- Egocentric video scales human behaviour across many people, objects and rooms. It records what happened from a useful viewpoint, but ordinary video does not contain the robot action that caused it.
- Teleoperation records observations and executable robot actions together. That makes it the most direct supervision for a target embodiment, while tying collection to hardware, calibration, safety and operator time.
- UMI-style grippers collect camera-centric manipulation trajectories without placing a robot in every room. They narrow the gap between human video and teleoperation, but only for tasks and end effectors their representation can express.
- The practical answer is usually a mixture: broad human video for coverage, portable gripper demonstrations for manipulation structure, and robot-native teleoperation for final adaptation and evaluation.
The mug is ordinary: off-white ceramic, one handle, a little heavier than it looks.
Place it on a kitchen counter and ask three people to teach a robot to move it onto a saucer.
The first person wears a camera on the chest. The recording shows a hand approaching, the mug turning slightly as it is lifted, the wrist avoiding a cereal box and the handle ending on the left. The footage contains a real room, a real object and a natural recovery when the first grip slips.
The second person stands behind a teleoperation interface. A robot arm follows the operator’s input. Every camera frame arrives beside joint positions, gripper state and an action command. The demonstration is slower, but the policy does not have to guess what the robot did.
The third person holds a parallel-jaw gripper with a wrist camera. There is no robot arm in the kitchen. The person closes the jaws around the mug and moves it through the task while the device reconstructs the gripper’s trajectory. The demonstration resembles the action space a robot can execute without requiring the robot to travel to the room.
All three files may be sold under the label robot training data. Only one contains native actions for the robot that will eventually move. Another supplies enormous environmental breadth. The third tries to preserve useful parts of both.
A policy needs more than a video of success
A manipulation policy observes a state and chooses an action. In its simplest form:
The observation o_t may include camera images, depth, robot joint angles and gripper state. The goal g may be an instruction such as “place the mug on the saucer with the handle facing left.” The action a_t could be joint commands, an end-effector movement or a gripper command.
A video gives the learner a sequence of observations. Teleoperation gives observations paired with actions. A handheld gripper attempts to recover a compact, camera-centred action trajectory that can later be mapped onto a robot.
Teleoperation · observation + robot state → robot action → next state
UMI-style capture · wrist observation + gripper trajectory → retargeted robot action
This difference is easy to hide inside a dataset count. One thousand hours of video and one thousand hours of teleoperation are not alternative prices for the same evidence. The first may cover thousands of kitchens cheaply. The second may teach one robot precisely how its own elbow and gripper moved.
Teacher one: the person wearing the camera
Egocentric capture moves the camera into the worker’s point of view. A phone, action camera or custom rig records the hands, objects and workspace while the person performs an ordinary task.
Its advantage is not that a human hand perfectly resembles a robot gripper. It is that people already occupy the environments robots are expected to enter. They open unfamiliar cupboards, restock oddly shaped shelves, fold different fabrics and discover that the “same” mug may be ceramic, steel, plastic or still wet from the sink.
That distribution is expensive to reproduce around a robot. Open-AoE illustrates the scale difference. Its 2026 release reports approximately 2,000 hours of manipulation video from more than 500 contributors using over 400 smartphones in natural environments. The project adds text, reconstructed hand pose, camera trajectories and atomic-action segments to make raw video more useful for downstream robot learning. [1]
Video at that scale can teach object affordances, task order, hand–object interaction, scene understanding and what successful execution looks like. It can also pretrain a model before a particular robot has been selected.
But the missing action matters. The pixels do not directly say how far a six-degree-of-freedom robot should move, which joint should bend, or how hard a parallel gripper should close. Human hands articulate differently from robot hands. A chest camera can lose the fingers behind the mug. Pose reconstruction adds useful estimates, not measured robot commands.
This is the embodiment gap: the demonstrator and the deployed machine do not share the same body. A model can learn that the mug should end on the saucer without learning an executable path for a particular arm.
What egocentric video buys
Cheap environmental breadth, natural task variation and early visual pretraining. What it usually does not buy on its own is a native action label for the final robot.
Teacher two: the person driving the robot
In teleoperation, a person controls the robot through a leader arm, VR controllers, a SpaceMouse or another interface. The recording can preserve camera views, joint state, end-effector pose, gripper width and the action issued at every step.
Now the training pair has the form imitation learning wants:
The policy sees what the robot saw and what a successful operator commanded next. If the target robot is the same platform, fewer geometric assumptions sit between demonstration and deployment.
DROID shows what a serious distributed teleoperation effort can contain. Its authors collected 76,000 trajectories—about 350 hours—across 564 scenes and 86 tasks with 50 collectors. They used a common portable setup built around a Franka Panda arm, stereo cameras, a wrist camera and a VR teleoperation interface. [2]
The numbers also expose the operational burden. The collection took twelve months. The robot, cameras and controller had to reach each scene. The system needed calibration, safe workspaces and people trained to recover from awkward arm configurations. Moving the robot increases diversity; moving a camera is still easier.
Teleoperation also carries an embodiment choice into the dataset. Joint actions from one arm are not automatically valid for another arm with different reach, limits or control frequency. Larger mixtures can help: Open X-Embodiment assembled data from 22 robot embodiments and reported positive transfer across platforms. It also had to standardise many datasets and action spaces before that mixture became useful. [3]
Direct supervision is valuable because it is specific. That specificity is also the constraint.
What teleoperation buys
Measured observations and executable robot actions on a known platform. What it costs is hardware time, operator time, calibration, safety and a collection process coupled to the chosen embodiment.
Teacher three: the gripper without the robot
The Universal Manipulation Interface begins with a peculiar object: a handheld parallel-jaw gripper carrying a wrist-mounted GoPro. A person holds the tool, grasps the object and performs the task directly. Camera motion and gripper width provide a manipulation trajectory without a full robot arm being present.
The original UMI work was designed to make those demonstrations deployable. It uses a relative-trajectory action representation, checks whether motions fit the target robot’s kinematic limits and matches observation/action latency at inference time. Its authors demonstrated the same learned cup-arrangement policy on UR5 and Franka arms. [4]
For one cup-arrangement experiment, the UMI project reports 111 demonstrations per hour, compared with 35 per hour using SpaceMouse teleoperation and 231 per hour for a bare human hand. That is a little more than three times the teleoperation rate for that setup—not a universal rate for every task, operator or UMI implementation. [4]
The reason is logistical. A portable gripper can visit a restaurant, workshop or apartment without transporting a robot, clearing its safety envelope and recalibrating the arm. It also constrains the human demonstration into something closer to a parallel-jaw robot action than ordinary hand video.
UMI does not erase embodiment. A two-jaw gripper cannot demonstrate the finger coordination required by a dexterous hand. A human still supplies motion with an arm whose reach and collision profile differ from the deployed robot. Trajectory reconstruction, camera calibration, time synchronisation and kinematic feasibility remain part of the pipeline.
It is best understood as a bridge: richer action structure than ordinary video, greater location diversity than robot-bound teleoperation.
The three datasets preserve different evidence
| Question | Egocentric video | Teleoperation | UMI-style gripper |
|---|---|---|---|
| What is measured? | Human-centred visual observation; optional IMU, depth and reconstructed pose | Robot observations, state and native action commands | Wrist-centred video, gripper width and reconstructed gripper trajectory |
| Does it contain a direct robot action? | Usually no | Yes, for the recorded platform | A retargetable end-effector representation, not every joint command |
| Environment diversity | Highest potential | Limited by moving and operating the robot | High for tasks compatible with the handheld tool |
| Embodiment dependence | Human-to-robot gap | High unless actions are normalised or retargeted | Lower for compatible parallel-jaw systems, not zero |
| Best role | Pretraining, task understanding, affordances, coverage | Final policy learning, adaptation and robot-native evaluation | Portable manipulation demonstrations and broader action coverage |
| Common hidden failure | Hands leave frame or pose estimates drift | Streams desynchronise or operator artefacts enter the policy | Trajectory reconstruction or calibration fails silently |
The comparison describes typical collection designs. Actual deliverables depend on the rig, sensors, action representation and annotation ordered.
The cheapest hour can be the most expensive dataset
Data budgets are often compared per recorded hour. That denominator is too forgiving. A better unit is the accepted, trainable episode that covers a condition the model needs.
A hundred inexpensive chest-camera hours can be poor value if every task hides the hand behind an object. Ten hours of teleoperation can be poor value if the camera and action clocks drift. A portable gripper campaign can be poor value if the target task needs tactile force or dexterous fingers the device never records.
Coverage is also multidimensional:
Suppose a home-manipulation programme needs 12 tasks, 8 room types, 15 object variants, 6 operators and 5 repetitions. A full Cartesian product contains:
No sensible pilot records every combination. The arithmetic reveals why “collect 1,000 hours” is not a specification. A programme must decide which factors deserve coverage, which can be sampled and which failures need deliberate oversampling.
A strong programme uses the methods in sequence
The choice does not have to be exclusive. Each method can answer a different uncertainty.
Use egocentric capture across many people, rooms and objects to learn what varies and where success breaks.
Use portable grippers where end-effector paths and grasp timing matter but a robot cannot visit every environment.
Collect teleoperation on the target platform for its reach, latency, collisions, gripper and control interface.
Test the deployed policy in objects and environments excluded by person, session and location—not random clips from the same recording day.
This funnel spends the most embodiment-specific collection where it is most valuable. Human video discovers the breadth of the world. Portable demonstrations bring parts of that breadth into an action representation. Teleoperation teaches and validates the final machine.
Quality is a clock problem before it is a model problem
All three methods fail if their streams disagree about time.
A camera at 30 frames per second produces one frame every 33.3 milliseconds. If an action stream is offset by 83 milliseconds, the action can be paired with an image roughly two and a half frames away:
During a slow reach, that may be tolerable. During contact, a few frames can separate an open gripper from a closed one. The policy learns from the wrong cause and effect even though the video looks correct to a reviewer.
A deliverable therefore needs more than media files. At minimum it needs monotonic timestamps, a declared clock source, calibration records where geometry matters, per-episode task and environment metadata, consent references, QC status, failure reasons and a versioned annotation guide.
Annotation should preserve uncertainty. If a hand pose is estimated rather than measured, the record should say so. If an episode fails because the object left frame, rejection is data about the protocol—not something to conceal by trimming the clip.
How an enterprise should choose
Start with the intended action space, not the fashionable dataset name.
If the policy cannot execute the target robot safely → collect teleoperation on that embodiment.
If manipulation trajectories are needed across many real rooms → evaluate a UMI-style programme.
If the task depends on force, slip or dexterous fingers → add tactile or measured hand-pose capture; none of the three basic streams is sufficient alone.
Then ask operational questions. Which cameras and control streams will arrive? What coordinate frame defines the action? How are devices synchronised? Can a demonstration be replayed? What counts as success? Which sessions are held out? Who owns the footage and the derived trajectories? Can the contributor’s consent support model training and commercial deployment?
The right answer should fit in a written delivery specification. Perit’s Physical Intelligence programme separates egocentric, teleoperation and UMI-style gripper capture because each promises different evidence. The sample episodes show the file lists and metadata before a programme is scoped.
A useful pilot should deliberately try to break the proposed method. Put the chest camera near a high cupboard. Give the teleoperator a task near the arm’s reach limit. Ask the gripper demonstrator to manipulate an object whose geometry exposes the tool’s constraints. Ten to thirty episodes can reveal a mismatch before thousands reproduce it. Perit describes that pilot logic in Order 5% First.
The mug did not have one teacher
The chest camera taught the model what kitchens look like when nobody prepared them for a robot.
The teleoperator taught one machine which action followed each observation.
The handheld gripper carried a robot-like trajectory into a room where no robot stood.
None of the demonstrations was the complete truth of the task. Each preserved a different relationship between the world, the body and the action.
That is the procurement lesson hiding inside the ordinary mug. Buying “robot data” without specifying that relationship is like buying a map without asking whether it contains roads, elevation or property lines. The file may be accurate. It may still omit the variable the journey depends on.
The best training programme does not choose the most impressive capture rig. It chooses the cheapest evidence the model can use—then spends its expensive robot time only where the robot’s own body has to become part of the lesson.
References
- Zishuo Li et al., “Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning”, July 2026.
- Alexander Khazatsky et al., “DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset”, Robotics: Science and Systems 2024.
- Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic Learning Datasets and RT-X Models”, 2023/2024.
- Cheng Chi et al., “Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots”, Robotics: Science and Systems 2024.
- Perit, Physical Intelligence collection lines and downloadable sample episodes, accessed 15 November 2026.