Design a teleoperation system

The single most important thing to get right about teleoperation is that it is two products wearing one body. The first is the obvious one, a remote human sees through a robot's cameras and drives its arms, so a fleet can sell task completion its autonomy cannot yet deliver. The second is quieter and worth more. A teleoperation session, recorded properly, is a demonstration, synchronized observations and actions of exactly the kind imitation learning consumes. Every robot company doing manipulation at scale runs some version of this system, and the ones that treat recording as an afterthought are paying operators by the hour without collecting the asset.

Framed that way, the design has two hard problems. The first is closing a control loop across the public internet, video flowing down and commands flowing up fast enough that a human can guide a gripper onto a mug handle, which is a latency problem before it is anything else. The second is making sure that loop is never load-bearing for safety, because the network will vanish mid-motion and the robot has to handle that moment entirely on its own. Under those two sit the ordinary but load-bearing pieces, session signaling, operator matching, a multi-camera video pipeline, and the recording path that feeds robot generalization.

The named systems and numbers in this article come from the teams that published them. ALOHA for the leader-follower rig, the DROID collection for the VR-controller setup, UMI for the handheld gripper, and 1X's published material on NEO and Redwood for what a commercial fleet's version of this loop looks like.

Scope and requirements

Functionally, an operator sees a robot's camera views, drives it with an input device, and hands control back when the tricky part is done. The robot executes commands, keeps itself safe, and records the whole exchange. The fleet side assigns operators to robots that need help, and for home robots the owner schedules and approves sessions. That is a small surface, and as in every design here the difficulty is not feature count but the non-functional pressure behind it.

Four constraints shape the design. Latency is the product. The operator's video needs to be under a couple hundred milliseconds glass to glass on a good link, because manipulation through a half-second-old picture feels like steering a car by looking out the back window. Safety cannot depend on the network. The link will drop mid-motion, and the robot must degrade to a safe state using only what is on board. Recording is a first-class requirement, not telemetry. An unrecorded session is operator payroll spent without buying data, so the recorder gets the same engineering attention as the control path. And privacy is structural for home robots, because the product is a camera on legs inside someone's house. Below the line I would push the learning stack itself, which is its own article in robot generalization, along with navigation autonomy and the robot hardware.

The control loop split

Three nested loops run at three different rates, and the design is mostly the discipline of keeping them apart. The innermost loop is the joint servo, current and position control on the motor drivers at kilohertz rates, and it never leaves the robot. The middle loop is robot-local control at roughly 100 to 500 Hz, trajectory smoothing, workspace and velocity limits, force control on contact, balance if the robot has legs, and the watchdog that decides what happens when commands stop arriving. The outer loop is the human, tens of hertz of intent round-tripping through the public internet.

The human sends intent, not torques. Depending on the input device that intent is a stream of joint targets, end-effector poses, or base velocities, plus gripper commands, and the robot's local controller turns it into smooth motion. The split matches what each side is good at. A person is superb at perception and strategy, where should the gripper go, which object is the right one, and hopeless at millisecond-scale corrections through a laggy link, so contact-rich stabilization, compliance, and disturbance rejection stay local. ALOHA makes the rates concrete, its teleoperation and recording both run at 50 Hz, and that is over a wire on a desk. A remote system ships the same 50 Hz intent stream over a network it does not control, which is why everything faster than 50 Hz has to already be on the robot.

The rule that falls out is worth stating plainly. Any loop whose failure can hurt someone must close on the robot. The network carries suggestions. The robot's own controller is the only thing that ever commands an actuator, and it applies its limits to every command regardless of who sent it.

The operator's commands are suggestions that stop at the teleop agent. Only the local controller commands actuators, the recorder sees every frame and every command, and the hardware e-stop path never touches software.

The latency budget, end to end

Walk the downlink one stage at a time, because the budget only makes sense as a sum. Sensor exposure and readout at 30 fps means a frame is already up to 33 ms old when it exists. Hardware encode adds 5 to 15 ms. The one-way network path adds 20 to 60 ms on decent continental routes, and more if the operator is on another continent. The receiver's jitter buffer, which smooths out packet arrival variance, adds 30 to 100 ms and is the one knob you actively tune, shrinking it when the network is steady and growing it when it is not. Decode and render add another 10 to 20 ms. Add it up and a well-built pipeline lands between 100 and 250 ms glass to glass, which is the range WebRTC was built for, while the chunked HTTP streaming of a video streaming service lives in whole seconds by design.

Then comes the human, and the human is slow. Simple reaction time is around a quarter second, and deciding what to do takes longer. The uplink back is cheap, a command is tens of bytes and pays one one-way trip. So the full loop from something happening in front of the robot to the actuator responding to the operator's reaction is roughly half a second on a good day. The design consequence is the one from the previous section, the remote human is a planner, not a stabilizer. If a grasp needs a 50 ms correction, that correction must come from local force control, because no transport gets a human there in time.

Latency is also not one number, it is a per-device property you measure. The UMI paper measured its camera, proprioception, and execution latencies separately, found values ranging from single-digit milliseconds to hundreds, and built latency matching into deployment, aligning observations to a common time and commanding actions slightly ahead to compensate. A teleoperation system needs the same bookkeeping, and the cheapest latency win it has is not a codec, it is assigning operators near the robot's region, because 60 ms of geography is free to remove and impossible to compress.

The video downlink, and why WebRTC over UDP

Manipulation needs multiple views, so start with the bandwidth math. ALOHA streams four Logitech C922x webcams at 480x640, two on the wrists and two static. The DROID rig carries two Zed 2 stereo cameras plus a wrist-mounted Zed Mini. Call it three useful views for a remote operator. A 720p30 H.264 stream of a scene with motion runs 1.5 to 2.5 Mbps, so a session costs 5 to 8 Mbps of downlink per robot, and the operator station composites the views with the wrist camera large, because the wrist view is where grasps succeed or fail.

Now the transport, and why TCP streaming is a non-starter. TCP promises ordered, reliable delivery, and both halves of that promise are wrong for this workload. Ordered means one lost packet blocks every packet behind it, head-of-line blocking, so a single loss stalls the whole picture. Reliable means the loss is retransmitted, and the retransmitted bytes belong to a frame the operator no longer wants, because a newer frame already exists. Under congestion TCP fills queues and latency compounds exactly when the operator most needs the picture. The TCP-based streaming stack, RTMP ingest and HLS or DASH delivery, buffers seconds on purpose to ride out variance, which is the right call for a movie and a disqualifying one for a control loop.

WebRTC is the assembly of choices that fixes this, and it is an open standard maintained with support from Apple, Google, Microsoft, and Mozilla, shipped in every modern browser, which means the operator station can be a web page instead of an installed client. Media travels as SRTP, encrypted RTP over UDP, so a lost packet is simply gone and the stream continues with the next frame, with recovery handled selectively, retransmit only what is still useful, otherwise conceal and request a refresh. Its congestion control adapts the encoder bitrate to keep queues short rather than filling them. The command uplink uses the WebRTC data channel, which you configure to drop stale messages instead of retransmitting them, because a 50 Hz command stream has the same property as the video, the newest message obsoletes every older one. The robot enforces this anyway by sequence number, trusting no transport.

The session stack

Before any video flows, the two ends have to find each other, and WebRTC deliberately leaves that to you. Every robot holds a persistent authenticated WebSocket to a signaling service, the same connection shape as the chat system, and an operator's browser connects to the same service to exchange SDP, the session description that carries codecs, resolutions, and encryption parameters. Then ICE, the connectivity framework, gathers candidate network paths for both ends. A STUN server tells each peer its public address so they can attempt a direct UDP path through their NATs, and a TURN server relays traffic as the fallback when NAT rules make a direct path impossible, which MDN describes bluntly as overhead to be used only when there is no alternative.

That fallback has a price tag worth computing. A relayed session pushes its full 6 Mbps through servers you pay for, so 1,000 concurrently relayed robots is roughly 6 Gbps of TURN egress, all day, every day. Your direct-connection rate is a cost metric that deserves a dashboard, and robots on well-configured networks should relay rarely.

Operator assignment is a matching problem, the same shape as dispatch in rideshare with round-trip time playing the role of distance. A robot that needs help enqueues a request carrying a context snapshot, what task it was on, why it stopped, and a still from each camera, so the operator arrives oriented instead of spending the first thirty seconds scrubbing video. The matcher filters by hard constraints first, certification for that robot model and task family, legal jurisdiction and language for home sessions, then ranks by network proximity to keep the latency budget honest, then balances queue fairness so no robot starves. Handoff between operators is a first-class operation, the robot holds pose, the new operator gets the context snapshot, and control transfers only through an explicit re-arm, never implicitly.

When the link degrades, and when it drops

Degradation comes before the drop, and the response is a ladder. As the congestion controller reports a shrinking budget, the video downshifts bitrate first, then resolution, then frame rate, because a sharp slow picture beats a smeared fast one for manipulation. The command channel is protected at all costs, it is a few kilobits and it is the product. Every command carries a sequence number and a robot-clock timestamp, and the robot silently ignores anything stale or out of order, so a burst of delayed packets arriving all at once cannot make the arm replay the past.

The drop is where the design earns its keep. The robot runs a watchdog, a deadman timer fed by valid commands. When commands stop, the robot first holds pose, still balancing, still gravity-compensating, because brief gaps are normal. When silence persists past a threshold, it executes a safe-stop, a controlled deceleration to a gravity-compensated hold, brakes engaged if the platform has them. The important word is controlled. An instant freeze is itself a dynamic event, locking joints mid-swing whips the payload and, for a legged robot, balance is an active process that must keep running, so the safe state is produced by local control, not by the absence of control.

Two tempting alternatives fail. Full autonomy fallback, letting the onboard policy finish the task when the link dies, fails because the policy is by definition the thing that was not good enough, that is why a human was driving, and a silent takeover also corrupts the recording, since steps labeled as human demonstrations would actually be policy actions. The defensible middle ground is letting local control complete a motion primitive the operator already committed, a grasp already closing, under local guards, then holding. Resuming by replaying buffered commands fails harder. Commands queued during the outage were issued against a world that no longer exists. On reconnect the operator sees a fresh state snapshot and must explicitly re-arm, and the command queue from before the drop is discarded, always.

# Runs on the robot at 100 Hz. The network feeds this loop from outside,
# it is never inside it.
CMD_FRESH_MS = 100      # a newest command older than this is not applied
SAFESTOP_MS  = 250      # sustained silence triggers the controlled stop

def control_tick(now_ms):
    global silence_ms, last_seq, armed
    cmd = latest_command()          # highest sequence number received so far
    fresh = cmd and cmd.seq > last_seq and now_ms - cmd.recv_ms < CMD_FRESH_MS

    if armed and fresh:
        target = clamp_workspace(cmd.target)    # limits apply to every command
        target = clamp_velocity(target, q_now)  # no operator can override them
        last_seq, silence_ms = cmd.seq, 0
        setpoint = interpolate(q_now, target)   # smooth 50 Hz intent to 100 Hz
    else:
        silence_ms += 10
        if silence_ms >= SAFESTOP_MS:
            armed = False                       # resume needs an explicit re-arm
            setpoint = safe_stop(q_now)         # decelerate, gravity-comp hold
        else:
            setpoint = hold(q_now)              # short gap: hold pose, stay balanced
    servo(setpoint)                             # kilohertz loops live below this

Input devices, and what each buys

Leader-follower arms. ALOHA is the reference design, a bimanual rig built within a $20k budget. The operator backdrives a smaller WidowX leader arm, about $3,300, and a larger ViperX 6-DoF follower mirrors it through direct joint-space mapping, leader joint angles become follower joint targets, synchronized at 50 Hz along with the recording. The authors chose joint-space over task-space mapping because inverse kinematics fails frequently in this setting and the direct mapping avoids that and its latency. The payoff is fidelity, the human feels the kinematics of the robot through their own hands, and ten minutes of demonstrations per task was enough for the ACT policy to hit 80 to 90 percent success on six fine-manipulation tasks. The cost is that the leader is robot-specific hardware, one rig per robot model, and it wants the operator at a desk built for it.

VR controllers. The DROID collection drove a Franka Panda 7-DoF arm with an Oculus Quest 2 headset and controllers, and the scale it reached is the argument, 76k trajectories, 350 hours, 564 scenes, and 86 tasks gathered by 50 collectors across 13 institutions over 12 months. A tracked controller reports a 6-DoF hand pose in task space, so the robot solves IK to follow it, the opposite trade from ALOHA. What you get is cheap, portable, robot-agnostic hardware that any operator anywhere can pick up, which is exactly the shape a remote fleet product needs. What you give up is the kinesthetic feel and the guarantee that every commanded pose is reachable.

Handheld grippers, no robot at all. UMI is a hand-held parallel-jaw gripper, $73 of 3D printing plus $298 of GoPro camera and accessories, with a 155-degree fisheye lens, side mirrors that give implicit stereo, and the camera's IMU feeding ORB-SLAM3 to recover the trajectory at 6.1 mm and 3.5 degrees of error. A person just does the task while holding it, and relative-trajectory actions plus latency matching make the resulting policies deployable across robot platforms. Throughput is the headline, in the paper's comparison the gripper collects demonstrations more than three times faster than teleoperation, at nearly half the speed of a bare human hand. Sunday Robotics pushes the same idea further with a Skill Capture Glove built with the same geometry and sensor layout as their robot's hand, converting glove recordings into robot data with a claimed 90 percent success rate, on the argument that teleoperation alone scales too slowly.

The trade-off triangle is fidelity, reach, and throughput. Leader arms win fidelity for one robot model. VR controllers win reach, any operator, any robot, over a network. Handheld devices win throughput but take the robot out of the loop entirely, which means they are not teleoperation, there is no operator product, no paid task completed, and no on-policy data of the robot correcting its own mistakes. A fleet business needs the middle option for its product, and a data engine happily uses all three.

Every session is training data

The recorder is the second product, so specify it like one. Record on the robot at capture quality, not the degraded stream the operator happened to see through a congested link, though keeping the operator's rendition too is useful for audit. Record every camera, joint positions and velocities, gripper state, and force-torque if present. Record the action as the operator's raw command, not the smoothed trajectory the robot executed. ALOHA is explicit about this, it records the leader arm's joint positions, the human's input, as the actions, because the policy should learn to produce intent, and the local controller will smooth at deployment just as it did during collection. Record who was in control at every step, human or policy, because episodes are a mix. 1X's Redwood model was trained on teleoperated and autonomous episodes from EVE and NEO, including both successful and failed rollouts, so the flag and the outcome label are what make the mixture usable.

Synchronization is the part that quietly ruins datasets. Stamp everything at the source with one monotonic robot clock, camera frames at capture, joint states at read, commands at receipt. Operator-side events map onto the robot timeline through the measured RTT. The storage math is friendly. Video dominates, three views at recording quality run around 8 Mbps, call it 3.6 GB per robot-hour, while a 50 Hz action stream for 14 joints in float32 is under 3 KB per second, hundreds of times smaller. Buffer episodes on the robot, seal them when the session ends, and upload resumably off-peak, because home Wi-Fi upstream is not a place to stream 8 Mbps continuously while also holding a live session.

Downstream, sealed episodes land in an episode store with lineage, pass through curation that filters corrupt captures, runs redaction for home privacy, and attaches labels, and then feed the training pipeline that robot generalization describes end to end. Failures are kept, labeled, because they carry signal. The payoff for format discipline compounds across the field. DROID ships 1,417 camera viewpoints with intrinsic and extrinsic calibration, which is what makes 13 institutions' data usable as one dataset, and Physical Intelligence trained π0 on dexterous data collected across 8 distinct robot embodiments, which only works when every embodiment records observations and actions in a shape one pipeline can consume.

# One step in an episode, written at 50 Hz on the robot.
step = {
    "seq": 48213,
    "t_robot_ns": 171234567891234,   # one monotonic clock, stamped at capture
    "obs": {
        "images": {                  # refs into per-camera video files
            "wrist_left":  ("cam2.mp4", 9182),
            "wrist_right": ("cam3.mp4", 9182),
            "front":       ("cam0.mp4", 9182),
        },
        "q":  [...],                 # joint positions, radians
        "dq": [...],                 # joint velocities
        "gripper": 0.31,             # normalized width
    },
    "action": {                      # what the HUMAN commanded, pre-smoothing
        "q_target": [...],           # ALOHA records leader joints as the action
        "gripper": 0.20,
    },
    "control_source": "teleop",      # or "policy", flagged on every step
}

# Episode metadata, written once when the episode is sealed.
episode = {
    "episode_id": "...", "robot_id": "...", "operator_id": "...",
    "task": "put the mug in the dishwasher",  # language label
    "success": True,                          # marked before release
    "intervention_spans": [(4210, 5890)],     # where the human took over
    "calibration": {...},                     # camera intrinsics + extrinsics
}

The recorder stamps everything on the robot's clock and seals episodes locally. Upload is queued and resumable, curation filters and redacts, and the trained checkpoints flow back to the same fleet that generated the data.

The operator-to-robot ratio

A fully teleoperated robot is a bad business, you are paying an operator's wage plus the robot plus the bandwidth to do a job the operator could do alone. The economics only work when one operator covers many robots. If autonomy handles a fraction a of task time, the ceiling is roughly one operator per 1/(1−a) robots, so 90 percent autonomy caps you near ten robots per operator and 99 percent near a hundred. The ceiling is not reachable, because interventions arrive in bursts, not on a schedule. Requests queue, the customer's wait time is a product SLA, and you staff below the ceiling to keep response time inside it, the same arrival-process reality every dispatch system lives with.

This is why the two products are one system. 1X's framing when it announced NEO was that owners could schedule a teleoperator for chores the robot does not know yet, the robot learning from the demonstration while the job gets done, and its Redwood model, a 160M-parameter transformer running on NEO's onboard GPU at about 5 Hz, was trained on exactly those teleoperated and autonomous episodes, gathered on robots in 1X offices and employee homes. Every paid session raises a, a higher a raises the ratio, and a better ratio funds more robots generating more sessions. The flywheel only spins if the recorder from the previous section actually captures every session in trainable form.

The honest counterargument is that the flywheel starts too slowly. Sunday Robotics argues you cannot deploy robots at scale without intelligence and cannot build intelligence without deployment data, and their answer was to skip teleoperation for collection entirely, capturing household demonstrations with gloves, reported at roughly $200 apiece, and transforming them into robot data. That resolves the chicken-and-egg for the dataset, and it gives up the operator-in-the-loop product, the paid task completion and the on-policy corrections, which is precisely what the teleop fleet keeps.

Safety, authority, and privacy

The e-stop is not software. A physical button on the robot and a safety-rated stop channel cut motor power or engage brakes through a path that never traverses the network, the OS, or your code, because every layer of the stack is something that can hang, and the stop must work when all of them have. The operator gets a stop button too, and it should be fast, but it rides the same network as everything else, so it is a convenience on top of the guarantee, not the guarantee. The watchdog from earlier is software and handles the network failing. The e-stop is electrical and handles the software failing.

Authority follows the rule this site keeps repeating, most recently in the brokerage design, never trust the client. The operator is an authenticated, certified, well-meaning client, and the robot still treats every command as untrusted input. Workspace limits, velocity and force caps, and joint bounds are enforced locally on every command with no override path from the network side. Sessions carry scoped grants, this operator, this robot, this time window, and every command lands in the recorder with the operator's identity attached, which makes the training log double as the audit log.

Home robots make privacy a design input rather than a policy document. The sensitive resource is the video downlink, a live camera feed from inside a house, and the recordings derived from it. The controls have to be structural. Owners schedule and approve sessions rather than being surprised by them, the robot shows an unmistakable indicator whenever a human is connected, no-go zones exclude rooms from both driving and viewing, and redaction, blurring faces and screens, runs on-device before video ever leaves the robot. 1X's stated safeguards for NEO's expert mode follow this shape, scheduled and approved sessions with blurring and no-go zones. Recording for training is opt-in and separate from consenting to a session, the redaction pipeline runs again before episodes enter the training store, and deletion requests follow the episode-store lineage, because a recording you cannot delete is a liability you chose to build.

Questions and answers

The core ideas as questions with the answers given outright. Each wrong multiple-choice option is marked with why it is wrong, and the ordering ones show the correct sequence.

References

1. Zhao et al., Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ALOHA), The leader-follower rig, joint-space mapping at 50 Hz, four 480x640 cameras, and the $20k budget.
2. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, 76k trajectories over 350 hours from 50 collectors at 13 institutions, driven with a Quest 2 headset and controllers on a Franka Panda.
3. Chi et al., Universal Manipulation Interface (UMI), The handheld gripper, GoPro fisheye and mirror optics, latency matching, and the throughput comparison showing collection more than three times faster than teleoperation.
4. MDN, Introduction to WebRTC protocols, ICE, STUN, TURN, and SDP, including why TURN relay is a last resort.
5. WebRTC.org, the official project site, The open standard for real-time video, voice, and data between peers, supported across all modern browsers.
6. 1X, Redwood AI, The 160M-parameter onboard model trained on teleoperated and autonomous episodes from EVE and NEO.
7. Physical Intelligence, π0, A foundation policy trained on dexterous manipulation data collected across 8 distinct robot embodiments.
8. Sunday Robotics, ACT-1: trained on zero robot data, The counterargument to the teleop flywheel, glove-based capture with a claimed 90 percent conversion to robot data.