{"id":881,"date":"2026-09-30T09:31:56","date_gmt":"2026-09-30T09:31:56","guid":{"rendered":"https:\/\/perit.ai\/blogs\/?p=881"},"modified":"2026-09-30T18:44:03","modified_gmt":"2026-09-30T18:44:03","slug":"teleoperation-explained","status":"publish","type":"post","link":"https:\/\/perit.ai\/blogs\/teleoperation-explained\/","title":{"rendered":"Teleoperation Explained"},"content":{"rendered":"\n\r\n<div class=\"pp-post\" style=\"max-width: 50rem;margin: 0 auto;font-family: Verdana, Geneva, Tahoma, sans-serif;font-size: 1rem;line-height: 1.68;color: #000\">\r\n<div class=\"meta\" style=\"font-size: 0.92rem;color: #333;margin-bottom: 1.6rem;padding-bottom: 0.65rem;border-bottom: 1px solid #aaa\">A hand closing around a mug in one frame of reference, a gripper closing around it in another, and the split-second of bookkeeping that keeps them the same demonstration<\/div>\r\n<p class=\"lede\" style=\"margin: 0 0 1rem;font-size: 1.08rem\">An operator reaches for a mug on a kitchen counter. A two-fingered gripper follows the movement, across the room or over a network connection. The operator controls the action. To train a policy from this episode, the recording must include the joint positions, gripper state and action commands, with timestamps that match them to the camera frames.<\/p>\r\n<p style=\"margin: 0 0 1rem\">Video shows the mug being picked up. A teleoperation episode also records the robot commands that produced the grasp.<\/p>\r\n<h2 id=\"video-shows-what-happened-teleoperation-records-what-was-done\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: bold\">Video shows what happened; teleoperation records what was done<\/h2>\r\n<p style=\"margin: 0 0 1rem\">Perit\u2019s egocentric video work records people completing tasks from a first-person view. There is no robot state or command stream, but the footage can show what successful execution looks like. UMI gripper capture adds gripper-relative motion without requiring a full robot in the loop.<\/p>\r\n<p style=\"margin: 0 0 1rem\">In teleoperation, a person controls a robot or a robot-equivalent end-effector directly. The recorded actions are commands a policy can learn to predict. Perit delivers these demonstrations episode by episode, with synchronised camera streams.<\/p>\r\n<h2 id=\"what-one-episode-actually-contains\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: bold\">What one episode actually contains<\/h2>\r\n<p style=\"margin: 0 0 1rem\">Each episode contains several streams recorded together:<\/p>\r\n<ul style=\"margin: 0.35rem 0 1.2rem 1.35rem;padding: 0\">\r\n<li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem\"><strong>Camera streams<\/strong>: one or more views of the workspace, synchronised to each other and to everything below.<\/li>\r\n<li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem\"><strong>State<\/strong>: the robot or end-effector&#8217;s own position and configuration at each moment: joint angles, gripper pose, whatever the rig reports about itself.<\/li>\r\n<li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem\"><strong>Action<\/strong>: the command sent by the operator&#8217;s teleoperation input at that same moment, which is what a policy is ultimately trying to learn to predict.<\/li>\r\n<li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem\"><strong>Depth, where the rig supports it<\/strong>: RGB-D, stereo or LiDAR capture, aligned per episode, adding a 3D geometry layer to the 2D camera view.<\/li>\r\n<li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem\"><strong>Tactile, where the rig supports it<\/strong>: a contact and pressure stream synchronised alongside the video, for tasks where force matters as much as position.<\/li>\r\n<\/ul>\r\n<p style=\"margin: 0 0 1rem\">A camera frame needs a matching action timestamp. A pressure spike needs a matching view of the contact. Without that alignment, the policy cannot reliably connect the command, the state and the result.<\/p>\r\n<h2 id=\"the-one-frame-rule\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: bold\">The one-frame rule<\/h2>\r\n<p style=\"margin: 0 0 1rem\">Perit\u2019s Sync check requires streams to align within one frame, with monotonic timestamps across cameras, state and action logs. Converting that frame into milliseconds makes the tolerance easier to judge.<\/p>\r\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;border: 1px solid #999;border-left: 4px solid #000\"><span class=\"eq\" style=\"display: block;text-align: center;font-family: &apos;Times New Roman&apos;, Georgia, serif;font-size: 1.12rem;margin: 0.4rem 0\">at 30 fps: 1 frame = 1\/30 s \u2248 33.3 ms<\/span> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &apos;Times New Roman&apos;, Georgia, serif;font-size: 1.12rem;margin: 0.4rem 0\">at 60 fps: 1 frame = 1\/60 s \u2248 16.7 ms<\/span><\/div>\r\n<p style=\"margin: 0 0 1rem\">At 33 milliseconds, that budget is roughly six times tighter than the approximately 200 milliseconds a person takes to react to a visual stimulus. The comparison is only a reference for scale. The rig needs tight internal timing so a policy can associate a gripper command with the frame in which it happened.<\/p>\r\n<p style=\"margin: 0 0 1rem\">Consider a pick-and-place task with roughly a two-second window for a successful gripper closure. The fingers must reach the object before closing and close soon enough to secure it.<\/p>\r\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;border: 1px solid #999;border-left: 4px solid #000\"><span class=\"eq\" style=\"display: block;text-align: center;font-family: &apos;Times New Roman&apos;, Georgia, serif;font-size: 1.12rem;margin: 0.4rem 0\">150 ms drift \u00f7 2,000 ms window = 7.5% of the action window misattributed<\/span><\/div>\r\n<p style=\"margin: 0 0 1rem\">A 150-millisecond drift shifts the action label by 7.5% of that two-second window. An unbuffered USB camera or an unsynchronised logging thread can introduce it. The recording may look normal while repeatedly pairing the command with an earlier or later frame.<\/p>\r\n<figure class=\"viz\" style=\"margin: 2.2rem 0\">\r\n<div class=\"viz-card\" style=\"border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb;padding: 14px 14px 8px;overflow: auto\">Three parallel timelines show camera, state and action ticks at regular intervals. A shaded vertical band marks a one-frame tolerance window around a gripper-close event; one action tick falls inside the band and is marked as passing, while a second, offset example falls outside the band and is marked as failing sync. Camera, state and action streams aligned within one frame A gripper-close action either lands inside the \u00b11-frame band or fails sync \u00b11 frame Camera State Action PassFail: outside tolerancetime \u2192<\/div>\r\n<figcaption class=\"viz-caption\" style=\"margin-top: 0.65rem;font-size: 0.9rem;line-height: 1.55;color: #52514e\">Figure 1. A gripper-close action passes the sync check when it falls within one frame of the matching camera and state ticks. Outside that band, the episode cannot reliably identify which frame belongs to the action.<\/figcaption>\r\n<\/figure>\r\n<h2 id=\"calibration-turns-a-camera-into-a-ruler\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: bold\">Checking camera calibration<\/h2>\r\n<p style=\"margin: 0 0 1rem\">Depth and multi-camera rigs also need calibration records for each rig and session. Intrinsics describe the camera\u2019s focal length, optical centre and lens distortion, allowing a pixel to be mapped to a ray in 3D space. Extrinsics locate and orient that camera relative to the rig or world, so the ray can be expressed in a shared coordinate frame.<\/p>\r\n<p style=\"margin: 0 0 1rem\">Moving a rig, bumping a camera or adjusting a mount can invalidate the previous extrinsics. The video may still look normal while the resulting 3D coordinates carry a consistent offset. Check and log calibration for each session.<\/p>\r\n<h2 id=\"where-the-episodes-come-from\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: bold\">Where the episodes come from<\/h2>\r\n<p style=\"margin: 0 0 1rem\">Perit records teleoperation in mechanical and electronics workshops, repair benches and robotics labs, as well as kitchens, living rooms, offices, retail floors, warehouses and agricultural sites. Its embodied-data programme spans 2,500-plus screened contributors, more than 120 homes, shops and worksites, and over 40 cities. Published capacity across all embodied modalities is 200 capture hours a day.<\/p>\r\n<p style=\"margin: 0 0 1rem\">Teams can use their own teleoperation hardware, including ALOHA-class arms. Bring Your Own Rig supplies crews, rooms and QA around the team\u2019s hardware and protocol.<\/p>\r\n<div class=\"case\" style=\"margin: 1.45rem 0;padding: 0.9rem 1rem;border-left: 4px solid #1f5fb8\">\r\n<p style=\"margin: 0 0 1rem;margin-bottom: 0\"><strong>A BYOR scenario:<\/strong> A robotics team has a working leader-follower arm pair and a capture stack tuned to their own action space, but no pipeline for recruiting operators, running sessions across varied real-world sites, or QA-checking sync and calibration at volume. Rather than replicate that infrastructure, they ship the rig and protocol; Perit supplies the crews, the kitchens and workshops to run it in, and the same sync, calibration and privacy checks applied to every other episode on the bench.<\/p>\r\n<\/div>\r\n<h2 id=\"the-pilot-for-a-rig-nobody-has-run-before\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: bold\">The pilot for a rig nobody has run before<\/h2>\r\n<p style=\"margin: 0 0 1rem\">A new rig goes through the same four-phase pilot used for Perit\u2019s physical-intelligence work. Before committing volume, the pilot checks whether the SOP produces clean sync, the task suits the arm, and calibration holds across a full session.<\/p>\r\n<div class=\"gate\" style=\"display: flex;flex-wrap: wrap;gap: 0.75rem;margin: 2rem 0\" aria-label=\"The four-phase embodied data pilot process\">\r\n<div class=\"gate-step\" style=\"flex: 1 1 11rem;min-width: 11rem;padding: 0.95rem 1rem 1rem;border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb\"><span class=\"step-no\" style=\"display: block;margin-bottom: 0.4rem;color: #2a78d6;font-size: 0.78rem;font-weight: bold;letter-spacing: 0.08em;text-transform: uppercase\">Step 01<\/span><strong style=\"display: block;margin-bottom: 0.35rem;font-size: 0.92rem;line-height: 1.4;color: #0b0b0b\">Day 0: Brief<\/strong><span class=\"step-desc\" style=\"display: block;font-size: 0.84rem;line-height: 1.5;color: #52514e\">Task family, environment, rig, sensors and episode count agreed on one page.<\/span><\/div>\r\n<div class=\"gate-step\" style=\"flex: 1 1 11rem;min-width: 11rem;padding: 0.95rem 1rem 1rem;border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb\"><span class=\"step-no\" style=\"display: block;margin-bottom: 0.4rem;color: #2a78d6;font-size: 0.78rem;font-weight: bold;letter-spacing: 0.08em;text-transform: uppercase\">Step 02<\/span><strong style=\"display: block;margin-bottom: 0.35rem;font-size: 0.92rem;line-height: 1.4;color: #0b0b0b\">Week 1: SOP and kit<\/strong><span class=\"step-desc\" style=\"display: block;font-size: 0.84rem;line-height: 1.5;color: #52514e\">Capture protocol and kit list written; operators trained and tested before payment.<\/span><\/div>\r\n<div class=\"gate-step\" style=\"flex: 1 1 11rem;min-width: 11rem;padding: 0.95rem 1rem 1rem;border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb\"><span class=\"step-no\" style=\"display: block;margin-bottom: 0.4rem;color: #2a78d6;font-size: 0.78rem;font-weight: bold;letter-spacing: 0.08em;text-transform: uppercase\">Step 03<\/span><strong style=\"display: block;margin-bottom: 0.35rem;font-size: 0.92rem;line-height: 1.4;color: #0b0b0b\">Week 2: Pilot episodes<\/strong><span class=\"step-desc\" style=\"display: block;font-size: 0.84rem;line-height: 1.5;color: #52514e\">Ten to thirty episodes delivered with QC status and a quality report.<\/span><\/div>\r\n<div class=\"gate-step\" style=\"flex: 1 1 11rem;min-width: 11rem;padding: 0.95rem 1rem 1rem;border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb\"><span class=\"step-no\" style=\"display: block;margin-bottom: 0.4rem;color: #2a78d6;font-size: 0.78rem;font-weight: bold;letter-spacing: 0.08em;text-transform: uppercase\">Step 04<\/span><strong style=\"display: block;margin-bottom: 0.35rem;font-size: 0.92rem;line-height: 1.4;color: #0b0b0b\">Ongoing: Review and scale<\/strong><span class=\"step-desc\" style=\"display: block;font-size: 0.84rem;line-height: 1.5;color: #52514e\">Protocol refined against pilot findings, then weekly delivery.<\/span><\/div>\r\n<\/div>\r\n<p style=\"margin: 0 0 1rem\">A useful brief names the object class, starting and ending states, failure definition and required streams. \u201cPick and place\u201d alone leaves too much open. Ten to thirty pilot episodes give the team a chance to check timing across a real operator\u2019s session and fix drift before the main order.<\/p>\r\n<div class=\"table-wrap\" style=\"max-width: 100%;overflow: auto;margin: 1.5rem 0\">\r\n<table style=\"width: 100%;min-width: 640px;border-collapse: collapse;font-size: 0.9rem;line-height: 1.5\">\r\n<thead>\r\n<tr>\r\n<th style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: bold;color: #0b0b0b\">Modality<\/th>\r\n<th style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: bold;color: #0b0b0b\">What&#8217;s synchronised<\/th>\r\n<th style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: bold;color: #0b0b0b\">Failure mode if sync breaks<\/th>\r\n<\/tr>\r\n<\/thead>\r\n<tbody>\r\n<tr>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Teleoperation<\/td>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Camera, robot\/end-effector state, action commands<\/td>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Policy learns the wrong instant to act; grasp timing drifts systematically<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Depth<\/td>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">RGB video and depth stream, per episode<\/td>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">3D geometry misaligned with the visual frame it&#8217;s supposed to describe<\/td>\r\n<\/tr>\r\n<tr>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Tactile<\/td>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Video and pressure stream<\/td>\r\n<td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Contact events attributed to the wrong moment in the task<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n<div class=\"quote-block\" style=\"margin: 1.6rem 0;padding: 0.15rem 0 0.15rem 1rem;border-left: 3px solid #000;font-size: 1.05rem;font-style: italic\">The video is the part you can watch. The synchronisation is the part that decides whether a policy can actually learn anything from it.<\/div>\r\n<h2 id=\"metadata-is-what-makes-an-episode-debuggable-later\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: bold\">Tracing failures with episode metadata<\/h2>\r\n<p style=\"margin: 0 0 1rem\">Every episode needs valid environment, task, rig, operator ID, session and consent references. These records become useful when a trained policy behaves unexpectedly and the team needs to trace the examples behind it.<\/p>\r\n<p style=\"margin: 0 0 1rem\">If a policy closes its gripper consistently early, operator and session IDs let you inspect the relevant slice of episodes. You can compare sessions, check for calibration drift and look for differences in grip technique. Without those records, the investigation starts with thousands of episodes and little way to narrow the search.<\/p>\r\n<h2 id=\"what-ships-at-the-end\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: bold\">What ships at the end<\/h2>\r\n<p style=\"margin: 0 0 1rem\">The ten checks cover framing, exposure and focus, motion, sync, task compliance, duration, metadata, privacy, calibration, and label agreement where annotation was ordered. Failed episodes are re-shot or rejected without billing. For teleoperation, sync and calibration determine whether the policy can connect the recorded command to the right physical state.<\/p>\r\n<div class=\"note\" style=\"margin: 1.5rem 0;padding: 0.85rem 1rem;border: 1px solid #999;font-size: 0.92rem\"><strong style=\"display: block;margin-bottom: 0.25rem\">Episode requirements<\/strong> Teleoperation records robot commands alongside camera and state streams. Check that they stay synchronised within one frame, and verify calibration for every session.<\/div>\r\n<div class=\"cta\" style=\"margin: 2.2rem 0 0;padding: 1.15rem 1.2rem;border: 2px solid #000\">\r\n<h2 id=\"tell-us-the-task-the-room-and-the-rig\" style=\"font-size: 1.28rem;line-height: 1.35;font-weight: bold;margin: 0 0 0.65rem\">Tell us the task, the room and the rig<\/h2>\r\n<p style=\"margin: 0 0 1rem;margin-bottom: 0\">Perit runs teleoperation, depth and tactile capture in kitchens, workshops, warehouses and client-defined sites across more than 40 cities. Use Perit\u2019s rig or bring your own. <a style=\"color: #0000ee;text-decoration: underline\" href=\"https:\/\/perit.ai\/physical-intelligence\" target=\"_blank\" rel=\"noopener\">See the full physical intelligence lineup<\/a> or send a one-page brief to get a pilot scoped within 48 hours.<\/p>\r\n<\/div>\r\n<footer class=\"post-note\" style=\"margin-top: 3rem;padding-top: 1rem;border-top: 1px solid #aaa;font-size: 0.88rem;line-height: 1.5;color: #222\" aria-label=\"Notes on the examples and figures\">\r\n<p style=\"margin: 0 0 1rem\">The opening mug-and-gripper scene and the pick-and-place timing example are illustrative, constructed to walk through the synchronisation mechanics rather than describing a specific recorded episode. Human visual reaction time is cited as an order-of-magnitude comparison, not a precise benchmark. Figures on contributor count, site count, city count and daily capture capacity, and the ten-point QC checklist, reflect Perit&#8217;s published physical intelligence programme at the time of writing.<\/p>\r\n<\/footer><\/div>\r\n\n","protected":false},"excerpt":{"rendered":"<p>What a teleoperation episode records, how its streams stay synchronised, and why calibration and session metadata matter for robot learning.<\/p>\n","protected":false},"author":2,"featured_media":884,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,23],"tags":[],"class_list":["post-881","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-featured","category-robotics"],"_links":{"self":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/881","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/comments?post=881"}],"version-history":[{"count":5,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/881\/revisions"}],"predecessor-version":[{"id":926,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/881\/revisions\/926"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/media\/884"}],"wp:attachment":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/media?parent=881"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/categories?post=881"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/tags?post=881"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}