Design a robot learning pipeline that generalizes

Fifty teleoperated demonstrations, roughly ten minutes of robot time, are enough to teach a bimanual robot to open a translucent condiment cup or slot a battery with 80 to 90 percent success. That was the ACT result in 2023, and it reset the field's assumptions about how much data one manipulation task needs. The catch sits in what the number does not say. Those success rates hold in the scene where the demonstrations were recorded, with the same table, the same lighting, and the same camera mounts, and moving the setup to a different room can take a working policy to zero. The system under design here is the pipeline that closes that gap, the collection, curation, training, and evaluation machinery that produces a policy whose success survives its first unseen kitchen.

The organizing evidence comes from Physical Intelligence's pi0.5, which trained on mobile-manipulation data from a spread of real homes and was then evaluated on multi-minute cleaning tasks in homes that appear nowhere in its training set. Its success in a new home climbs with the number of distinct training environments, and at around 100 environments it approaches a baseline trained directly on the test environment. Read that curve as the design brief. Generalization is bought with environment count, task variety, and embodiment breadth, which makes the data engine the product and the model checkpoint its output.

The walkthrough follows the data through six decisions, what a demonstration costs and how ALOHA, DROID, Figure, Generalist, and Sunday Robotics pay for it, how pi0 weights a 10,000-hour mixture so rare tasks stay visible, why pretraining on 22 embodiments helps the one robot you ship, where action chunking and flow-matching action experts fit, how to evaluate in scenes the model has never seen, and how a deployed fleet becomes the cheapest collection channel you own.

Scope and requirements

The target is a mobile manipulator that executes multi-step household and light-industrial tasks from a natural language instruction, clearing a table into a dishwasher, folding laundry out of a dryer, wiping a counter, restocking a shelf. The deliverable is not a model, it is the pipeline that produces successive models, and it has six stages walked in order here, demonstration collection, episode storage and curation, mixture construction, cross-embodiment pretraining followed by task post-training, evaluation in held-out scenes, and deployment logging that feeds the next cycle.

Four constraints shape everything downstream. Control runs at tens to hundreds of hertz, pi0 emits actions at up to 50 Hz for laundry folding and Figure's Helix drives a 35 degree-of-freedom upper body at 200 Hz, so whatever the backbone does, something has to produce joint targets on a schedule measured in milliseconds. Inference runs onboard, because homes have unreliable connectivity and a control loop cannot absorb a cloud round trip, which is why Helix runs entirely on embedded low-power GPUs and 1X's Redwood, a deliberately small 160M-parameter model, runs at about 5 Hz on NEO's own compute. Tasks are long, minutes of continuous manipulation with dozens of object interactions, so errors compound and recovery behavior matters as much as nominal skill. And the defining requirement, the one this whole article serves, is that success is measured in environments that appear nowhere in the training set. Locomotion controllers, hand hardware, and safety certification are real problems pushed below the line to protect the time.

Generalization is a data problem

The learning problem is imitation, supervised prediction of action sequences conditioned on camera images, proprioception, and a language instruction. The reference point for how well this works in-distribution is ACT, the policy behind the ALOHA system. A transformer trained as the decoder of a conditional VAE maps four camera views and joint positions to sequences of future joint targets, and with 50 demonstrations per task, roughly ten minutes of data, it reached 80 to 90 percent success on fine bimanual tasks like opening a translucent condiment cup and slotting a battery. Ten minutes of data per task is astonishingly little, and that efficiency is why imitation won the framing battle.

The same result defines the failure mode. Those policies are scene-bound, trained and evaluated on one table with one camera rig, and nothing in 50 demonstrations of one scene tells the model what a different kitchen looks like. The fix is not capacity, it is coverage, and pi0.5 measured the exchange rate. Success in an unseen home climbs with the number of distinct environments in training, and near 100 environments the generalist approaches a specialist trained on the test home itself. Google DeepMind reports the same lever pulled from the other side, Gemini Robotics inherits web-scale visual and semantic breadth from Gemini 2.0 and more than doubles the previous state of the art on their generalization benchmark. Both results say the model already had the parameters, what it lacked was a training distribution containing the right kind of novelty.

Two adjacent framings lose on contact. Reinforcement learning from scratch on real hardware pays for exploration in broken grippers and unproductive robot hours, so RL earns a place refining a policy that imitation already made competent, not as the main engine. And the classical modular stack, perception feeding a planner feeding a motion controller, works where the object catalog and the scene are controlled, which is precisely what a home is not. The open-world cases that break the modular stack, a shirt crumpled in a way no mesh library contains, are the cases the product exists for.

Teleoperation and the price of a demonstration

ALOHA set the reference design for collection. An operator grips two small leader arms and the follower arms mirror the motion, with teleoperation and recording running at 50 Hz and four RGB cameras streaming 480x640 views, all within a $20k budget for the bimanual rig. Episodes run 8 to 14 seconds, and the standard 50 demonstrations per task work out to roughly ten minutes of stored robot time. What teleoperation buys, and the reason it remains the backbone of every serious dataset, is exact action labels in the robot's own action space. Nothing has to be retargeted, inferred, or transferred.

What it costs is human time at a one-to-one exchange rate before overhead, since an hour of demonstrations costs an operator hour plus scene resets, failed takes, and rig maintenance. DROID is the honest public record of what diversity costs at that rate. Building 76k episodes, 350 hours of interaction across 564 scenes and 86 tasks, took 50 data collectors at 13 institutions on three continents 12 months, using a Franka arm on a wheeled height-adjustable mount and a Quest 2 headset for control. Figure trained Helix on roughly 500 hours of teleoperated upper-body data, which its own post describes as under 5 percent the size of earlier VLA datasets. At the industrial extreme, Generalist reports its GEN-0 models pretrain on 270,000 hours of real manipulation data with collection running above 10,000 hours a week across thousands of homes, warehouses, and workplaces. Nobody reaches numbers like that by paying an operator per rig-hour in a lab, which is why the collection channel itself is where the companies diverge.

Sunday Robotics removed the robot from the collection loop entirely. Their Skill Capture Glove shares its geometry and sensor layout with the robot's hand, data collectors wear it while doing chores in their own homes, and press coverage puts the harvest near 10 million episodes from more than 500 households, with a glove costing around $200 to produce against $20k for a teleop rig. Their ACT-1 foundation model trains on what the company calls zero robot data, with a transform step it reports converts glove recordings into robot training data at a 90 percent success rate. Google DeepMind's AutoRT points at a second channel, autonomous collection, where a VLM describes what a robot sees, an LLM proposes tasks, a second LLM screens them against a written constitution, and a fleet of over 20 robots gathered 77k real episodes spanning 6,650 distinct instructions. The trade runs quality against scale. Teleoperation gives the cleanest labels, gloves give real-home diversity with a transfer step, and autonomy gives volume at lower per-episode value.

Dataset composition and weighting

A pretraining corpus is a portfolio, not a pile, and the weighting decisions matter as much as the collection. pi0's pretraining mixture holds 903 million timesteps of internal data spanning 7 robot configurations and 68 tasks, over 10,000 hours in total, and open-source data, OXE, Bridge v2, and DROID together, makes up just 9.1 percent of it. Sampling proportionally from a corpus like that would let the most-collected tasks dominate every batch, because collection volume follows convenience, whichever tasks the rigs were set up for last quarter, rather than the balance the policy needs.

pi0's published correction weights each robot-task combination by n^0.43, where n is that combination's sample count. The exponent compresses the head of the distribution without deleting it. A combination with 209 times the data of another gets only about 10 times the sampling weight, so the rare task stays visible in every epoch instead of drowning. This is the same instinct as an exploration budget in a recommender, spend a controlled fraction of every batch on the tail, because the tail is where generalization lives.

The second composition decision is what non-robot data rides along. pi0.5 co-trains on five streams at once, web-scale captioning, VQA, and object detection data, verbal instructions from humans coaching the robot through long tasks, subtask commands pairing observations with high-level labels, cross-embodiment data from simpler static robots, and roughly 400 hours of mobile-manipulator data matching the deployment platform. Note the proportions. The slice that actually matches the robot being shipped is small, and most of what the final policy knows about objects, layouts, and language it learned from the other streams. The deployment-matched data teaches the hands, everything else teaches the world.

# pi0-style mixture weighting: down-weight over-collected combos
# (counts here are illustrative, the exponent 0.43 is pi0's published value)
import numpy as np

counts = {                      # samples per (robot, task) combination
    ("bimanual_trossen", "fold_laundry"): 9_400_000,
    ("ur5e", "bus_table"):                2_100_000,
    ("mobile_fibocom", "make_coffee"):      310_000,
    ("franka", "open_drawer"):               45_000,
}

alpha = 0.43                    # pi0 weights each combo by n ** 0.43
raw = {k: v ** alpha for k, v in counts.items()}
total = sum(raw.values())
weights = {k: v / total for k, v in raw.items()}

# The 9.4M-sample combo has 209x the data of the 45k one,
# but only 9.9x the sampling weight. Rare combos stay visible
# instead of drowning under the head of the distribution.

Pretrain on cross-embodiment data, post-train on your task

Cross-embodiment pretraining has a public existence proof. Open X-Embodiment pooled 60 datasets from 34 labs into more than 1 million real trajectories covering 22 robot embodiments and 527 skills, normalized into 7-dimensional end-effector actions. Models trained across that pool beat models trained on any single lab's data. RT-1-X outperformed the corresponding single-dataset models by 50 percent in small-data domains, and RT-2-X tripled RT-2's score on emergent-skill evaluations, learning spatial relations from episodes recorded on other people's hardware. Another robot's data does not transfer as trajectories, the kinematics differ, it transfers as representations, and the transfer is worth double-digit success points.

OpenVLA is the reproducible recipe. A 7B model, Llama 2 with a fused SigLIP and DINOv2 vision encoder, trained on 970k OXE trajectories for 15 days on 64 A100s, and it outperforms RT-2-X, a 55B closed model, on most zero-shot control evaluations. RT-2-X keeps an edge only on tasks that need web concepts absent from robot data, which the OpenVLA authors attribute to its larger internet pretraining. The adaptation story matters as much as the headline. LoRA fine-tuning touches 1.4 percent of OpenVLA's parameters and matches full fine-tuning, so bringing the pretrained checkpoint to a new robot is a single-GPU job, not a datacenter one.

The two-phase recipe mirrors language models on purpose. Pretraining on the broad cross-embodiment mixture buys the priors, object recognition across viewpoints, recovery after a slip, the mapping from language to scenes, and post-training on a smaller curated set buys fluency on the tasks the product ships, which is how pi0 reaches its laundry-folding and table-bussing results. The tension to manage is that post-training pulls the policy toward the curated distribution, and pushed too far it erases the breadth pretraining paid for, which is why the evaluation gate later in this design runs on held-out scenes rather than on the post-training tasks alone.

The pipeline end to end. Teleoperation, open cross-embodiment data, and web data meet in the mixture builder, whose n^0.43 weighting feeds pretraining. Post-training specializes the checkpoint, the eval gate runs trials in held-out homes, and fleet rollouts return along the dashed edge, which is the flywheel.

The model, a VLA with an action expert

A vision-language-action model is a VLM whose output modality is motor commands, and the design space has two live axes, how actions are represented and how a slow backbone feeds a fast controller. RT-2 and OpenVLA discretize, each action dimension becomes tokens in the language model's vocabulary and control becomes next-token prediction. That inherits the whole LLM stack, and it pays in rate, which is the limitation pi0's authors targeted when they moved to continuous outputs generated by flow matching so the model could serve chunked control at up to 50 Hz.

The continuous designs attach a separate action module. pi0 adds a 300M-parameter action expert, initialized from scratch, to the 3B PaliGemma backbone, 3.3B parameters in total, and trains it with flow matching, a relative of diffusion, to emit 50-step action chunks. NVIDIA's GR00T N1 draws the same picture with different numbers, 2.2B parameters total with 1.34B in an Eagle-2 VLM running around 10 Hz while a diffusion-transformer head generates 16-step action chunks at 120 Hz. Figure's Helix makes the split fully architectural, System 2 is a 7B VLM ticking at 7 to 9 Hz and System 1 is an 80M cross-attention transformer emitting continuous actions at 200 Hz across a 35-DoF upper body, both onboard. 1X's Redwood marks the small end, 160M parameters with a diffusion policy at roughly 5 Hz driving mobile manipulation on NEO. Every one of these is the same shape, a slow deliberate module that reads the scene and the instruction, and a fast reflex module that turns its output into joint targets.

Chunking is the load-bearing trick on the fast side, and it predates the VLA era. ACT predicts k=100 future actions per forward pass, two seconds of motion at 50 Hz, which divides the effective decision horizon by 100 and with it the compounding of per-step errors, and it also absorbs the pauses in human demonstrations that make single-step prediction ill-posed, since a paused teleoperator generates identical observations followed by different actions. Rather than executing a chunk open-loop, ACT re-queries every step and averages all live chunks' opinions of the current step with exponential weights, which the paper calls temporal ensembling, getting smoothness and reactivity at once.

# ACT-style chunking with temporal ensembling
import math
import numpy as np
from collections import deque

K = 100                      # chunk size from the ACT paper, 2 s at 50 Hz
M = 0.01                     # ensemble decay, w_i = exp(-m * i)
chunks = deque(maxlen=K)     # each entry: (t_predicted_at, actions[K])

def act(policy, obs, t):
    chunks.append((t, policy.predict_chunk(obs)))  # one forward pass
    preds, weights = [], []
    for i, (t0, actions) in enumerate(chunks):     # i = 0 is the oldest chunk
        age = t - t0
        if age < K:
            preds.append(actions[age])             # this chunk's action for "now"
            weights.append(math.exp(-M * i))       # oldest chunk weighted highest
    w = np.array(weights)
    w /= w.sum()
    return (np.array(preds) * w[:, None]).sum(axis=0)  # blended joint target

Serving on the robot, and the train/serve loop

At runtime the loop reads like this on a pi0.5-style stack. Cameras and joint encoders feed the backbone, the model first emits a high-level subtask as literal text, pick up the pillow, put the plate in the sink, and then, conditioned on that subtask, the action expert emits a 50-step chunk covering one second of motion. The executor streams those joint targets to the low-level controllers, re-queries before the chunk runs dry, and blends overlapping chunks. Inference stays on the robot because a control tick does not tolerate a WAN round trip and homes do not guarantee connectivity, which is the explicit reason 1X gives for running Redwood on NEO's onboard GPU.

The training half of the loop is ordinary MLOps with one robotics twist, the logs are gold. Every rollout, autonomous or teleoperated, is an episode in exactly the format the trainer eats, so deployment is also collection. Rollouts land in the episode lake tagged with outcomes and interventions, curation filters them into post-training batches, recurring jobs produce candidate checkpoints into a registry, and a candidate only replaces the fleet policy after clearing the unseen-scene gate. 1X states that Redwood learns from both successful and failed autonomous episodes, and failure rollouts are disproportionately valuable because they document exactly where the current policy's competence ends.

The serve path runs left to right, sensors into the slow VLM for a text subtask, the action expert for a 50-step chunk, and the executor blending chunks into joint targets. The train path runs underneath, logged rollouts with outcomes feed post-training jobs, and the registry deploys the next checkpoint along the dashed edge.

Evaluation in unseen scenes

Evaluation is the pipeline's bottleneck resource, because the only number that matters requires a physical robot in a place it has never been, and every trial burns minutes of robot time plus a human to stage and reset the scene. There is no held-out test set you can rerun in an afternoon. pi0.5's evaluation is the pattern to copy, take the policy to homes that appear nowhere in training and ask for multi-minute jobs, put the dishes in the sink, pick the clothes off the bedroom floor, and publish performance as a function of training-environment count, so the curve, not a single number, is the result. Gemini Robotics likewise reports against a dedicated generalization benchmark rather than a task list, which is what made its claim of more than doubling the prior state of the art meaningful.

Three rubric decisions keep the numbers honest. Hold out entire environments, not object positions, since a policy can memorize a room while appearing to generalize across rearrangements. Score partial progress rather than binary success, because long-horizon tasks fail at some stage, and which stage moved is the information a mixture change needs. And measure instruction following separately from task completion, the distinction pi0.5's out-of-distribution tests draw, because a policy that competently does a well-practiced wrong thing is a language failure, not a manipulation failure, and the fixes live in different parts of the mixture. Simulation earns a place as a cheap regression filter between hardware sessions, but contact-rich manipulation shifts enough between simulation and reality that promotion decisions stay on hardware.

The data flywheel from deployment

Once the policy clears a competence threshold, the fleet becomes the collection channel and the economics invert, because data now arrives as a byproduct of doing the job. AutoRT is the earliest public sketch of fleet-scale collection, over 20 robots across multiple buildings, a VLM describing each scene, an LLM proposing tasks, and a constitution-checking critic filtering them, with 88 percent of generated tasks valid before filtering and 93 percent after, yielding 77k real episodes across 6,650 instructions. 1X runs the production version, training Redwood on teleoperated and autonomous episodes from its EVE and NEO robots gathered in offices and employee homes. And a collection rate like Generalist's 10,000-plus hours a week is an operations achievement before it is a research one, gloves, rigs, and fleets running as a supply chain.

Two details decide whether the flywheel compounds or collapses. Interventions are labels. When a human takes over mid-task, the takeover point marks a state the policy mishandles and the correction shows what it should have done, which makes intervention segments the highest-value data the pipeline will ever see. And autonomous data is biased toward the policy's own habits, it documents what the current model already does well, so left unweighted it narrows the training distribution the same way a recommender that trains only on its own recommendations narrows a catalog. The n^0.43 discipline from the mixture section applies with extra force here. Deployment data joins the portfolio as one more over-represented combination to be down-weighted, not as a firehose to be drunk raw.

Alternatives that work, and non-starters

Several alternatives genuinely work and belong in the portfolio. Synthetic data works as a middle layer. GR00T N1's data pyramid puts web data and human video at the base, simulation and generated video in the middle, and real teleoperation at the top, and NVIDIA reports expanding 88 hours of real GR-1 humanoid teleoperation into roughly 827 hours with generated video. Robot-free human capture works when the hardware is designed for it, and Sunday's glove-first approach, where the company reports iterating on the glove close to 100 times and then designing the robot's hand around it, is the strongest version of the idea. The modular VLM-planner stack also still works for structured, auditable tasks. Gemini Robotics-ER exposes pointing, 3D detection, spatial reasoning, and code generation for exactly that consumer, and a warehouse cell with a fixed object catalog can reasonably stop there.

The non-starters share a shape, each tries to substitute something cheap for environment coverage. More demonstrations of the same scene do not generalize, the pi0.5 curve rises with environment count rather than demo count, and a thousand demos of one kitchen buy polish, not transfer. Simulation-only training for contact-rich manipulation leaves the gap the pyramid exists to bridge, which is why even the synthetic-heavy GR00T recipe anchors on real teleoperation at the top. Model scale without data breadth fails the OpenVLA test, where 7B beat 55B on most control evaluations because of what it was trained on rather than what it was. And real-world reinforcement learning from scratch as the primary engine spends its exploration budget in broken hardware and operator time, which is why every lab and company in this article, without exception, starts from imitation.

Questions and answers

The core ideas as questions with the answers given outright. Each wrong multiple-choice option is marked with why it is wrong, and the ordering ones show the correct sequence.

References

1. Zhao, Kumar, Levine, Finn, Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT / ALOHA), Action chunking with k=100, 50 Hz teleoperation, the $20k bimanual rig, 50 demonstrations per task, and temporal ensembling.
2. Black et al., pi0: A Vision-Language-Action Flow Model for General Robot Control, PaliGemma 3B plus a 300M action expert, flow matching, 50-step chunks at up to 50 Hz, 10,000+ hours across 7 robot configurations and 68 tasks, and the n^0.43 mixture weighting.
3. Physical Intelligence, pi0.5: a VLA with Open-World Generalization, Co-training across five data streams, roughly 400 hours of mobile manipulation data, hierarchical subtask-then-chunk inference, and the scaling curve that flattens near 100 training environments.
4. Kim et al., OpenVLA: An Open-Source Vision-Language-Action Model, 7B Llama 2 backbone with fused SigLIP and DINOv2 encoders, 970k OXE trajectories on 64 A100s for 15 days, outperforming the 55B RT-2-X, LoRA fine-tuning at 1.4 percent of parameters.
5. Open X-Embodiment Collaboration, Robotic Learning Datasets and RT-X Models, 22 embodiments, 60 datasets from 34 labs, 1M+ trajectories and 527 skills, with RT-1-X up 50 percent in small-data domains and RT-2-X at 3x on emergent skills.
6. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, 76k episodes and 350 hours across 564 scenes and 86 tasks, collected by 50 people at 13 institutions over 12 months, the public record of what scene diversity costs.
7. Figure, Helix: A Vision-Language-Action Model for Generalist Humanoid Control, System 2 as a 7B VLM at 7 to 9 Hz, System 1 as an 80M visuomotor transformer at 200 Hz over 35 DoF, trained on about 500 hours of teleoperation, running on embedded GPUs.
8. Generalist AI, GEN-0, 270,000 hours of real manipulation data growing at over 10,000 hours a week, power-law scaling results, and the reported capability phase transition past 7B parameters.