Physical Intelligence · Egocentric

What the hands see, at the moment they see it.

Monocular first-person video from a chest or head mount, recorded by people doing everyday tasks in the rooms where those tasks actually happen — for imitation learning, hand–object understanding and embodied pretraining.

What it is

What gets captured, and why it is worth having.

Environment
Apartment kitchen, mid-morning window light
Captured with
GoPro on a chest mount, RGB + on-device audio
Task
Empty a cupboard shelf, wipe it, put the containers back
Apartment kitchen, mid-morning window light · GoPro on a chest mount, RGB + on-device audio

A camera on the chest or forehead, a task, and a room. The person is not an actor: they are cleaning a stove, folding clothes, restocking a shelf, and the footage carries everything a staged capture loses — hesitation, reach, re-grasp, the way a cupboard door actually swings.

Because the rig is a phone or an action camera, the pool is anyone who can follow a protocol. That is what makes egocentric the fastest way to put thousands of diverse hours in front of a model before a single robot exists.

What it trains
  • Imitation learning from human video
  • Hand–object interaction and affordance models
  • Task segmentation and long-horizon activity understanding
  • Embodied pretraining ahead of robot data
The spec

How it is captured, and what you receive.

How it is captured
Primary rig
GoPro-class action camera on a chest or head mount
Phone rig
iPhone or Android in a chest harness or head strap, IMU logged
Lens
Wide or linear field of view to the protocol, horizon lock off
Sensors
RGB, timestamped IMU on phone rigs, on-device mono audio
Sync
Timestamps to the frame; multi-rig sessions clapped and aligned
Custom
Stereo or wider-field rigs built to your spec
What you receive
Video
H.264 in MP4
Resolution
4K or 1080p
Frame rate
29.97 / 30 fps
Audio
Mono, on-device
IMU
Accelerometer and gyroscope, timestamped to frames on phone rigs
Episode
2–15 minutes, one task each
Environments and tasks

Where it is recorded, and what people are asked to do.

Tasks are recorded where they actually happen — not in a studio dressed to look like a kitchen. Hover a room to see the kind of setting the protocol calls for.

Kitchens

Indian, European and Brazilian home kitchens, lived-in and cluttered, plus small commercial kitchens and pantries.

Cooking preparationWashing upCupboards and drawersAppliancesPouring and scooping

Living rooms and bedrooms

Real Western homes in Spain, Germany, France and Italy, and apartments in India, Brazil, Japan and Korea. Sofas, wardrobes, shelves, laundry.

Tidying and foldingMaking bedsShelving and storageLaundryFurniture interaction

Offices

Open-plan and small offices: desks, pantries, meeting rooms, filing and cable clutter.

Desk tasksDocument handlingCables and peripheralsPantry tasksDrawers and storage

Retail and small shops

Kirana stores, groceries, pharmacies and supermarket aisles, with real stock and real customers out of focus.

Shelf restockingPicking to a listBagging and countersInventory checksCrates and boxes

Warehouses and light industrial

Racking aisles, packing benches, courier depots and small production lines with pallets, trolleys and parts bins.

Pallet to shelfParcel sortingPacking and tapingParts movementInspection

Workshops and labs

Mechanical and electronics workshops, repair benches and robotics labs where a follower rig can be set up.

Tool handlingAssembly and fasteningPegboards and racksBench tidyingTeleoperation setups

Agriculture and outdoor

Polytunnels, packing sheds, market stalls and yards, in daylight that changes from one episode to the next.

Sorting produceCrates and traysPruning and pickingWeighing and baggingLoading

Client-defined and confidential

Your site, or a space recreated to your drawings, run by an NDA'd crew with limited-visibility workflows and no clip leaving your region.

Proprietary workflowsRecreated spacesStealth programsOn-site with your rigYour robot, our operators
What ships

Deliverables, and the labels you can add.

Deliverables
  • Raw video per episode, original device capture kept
  • Task and environment metadata per episode
  • QC status and reviewer chain per episode
  • Consent reference on every file
  • Quality report per batch: accepted, rejected, and why
  • Weekly drops to your bucket, region pinned at kickoff
Annotation add-ons
  • Task and sub-task segmentation
  • Natural-language captions
  • 3D hand pose estimation
  • Object and interaction labels

Labelled on the same bench as our speech work, to a written guide, behind the same calibration test.

How a pilot runs

Ten to thirty episodes, then the volume.

  1. 01day 0
    Brief

    The task family, the environments, the rig, the sensors, the episode count. One page, agreed before anything is bought or anyone is recruited.

  2. 02week 1
    SOP and kit

    A written capture protocol — framing, lighting, where a task starts and ends, what counts as a failed take — and the kit list. Operators are trained and tested on it before a single episode is paid for.

  3. 03week 2
    Pilot episodes

    A small batch, ten to thirty episodes, delivered with QC status and a quality report. This is where the protocol breaks — on purpose, and cheaply.

  4. 04ongoing
    Review and scale

    You review the pilot, we fix the protocol, then the run scales on a weekly delivery cadence with the same report attached to every batch.

Turnaround
48 hrs
Written spec and quote
5 days
Sample episode in your format
2 weeks
Pilot batch with its QA report
Weekly
Delivery cadence once the pilot clears
Before an episode counts

What QC checks, every time.

  • Framing

    The task stays in frame; hands and workspace inside the protocol's bounds for the whole episode.

  • Exposure and focus

    No blown highlights or focus hunting beyond the tolerance set for the room.

  • Motion

    No camera whip, no dropped frames; IMU continuous where the rig has one.

  • Sync

    Every stream aligned within one frame; timestamps monotonic across cameras, state and action logs.

  • Task compliance

    Start and end states match the SOP; no skipped or reordered steps unless the protocol allows it.

  • Duration

    Inside the episode window for the task family; overruns are trimmed to the protocol, not the clip.

  • Metadata

    Environment, task, rig, operator id, session and consent reference present and valid on every episode.

  • Privacy

    No bystander faces, screens or documents without consent; blurred where the protocol flags it.

  • Calibration

    Intrinsics and extrinsics on file per rig and session for depth, stereo and multi-camera capture.

  • Labels

    Where annotation is ordered, agreement against the gold set above the threshold before the batch ships.