Embodied foundation models: imitation, diffusion policies, and vision-language-action

Why this subject matters now

Five years ago a manipulation policy was usually a task-specific thing: a scripted state machine, or a reinforcement-learning agent trained in simulation for one skill, or a behavior-cloning network that imitated a few hundred demonstrations of a single motion. The policy did not transfer. A model that stacked blocks knew nothing about pouring water, and a model trained on one robot arm was useless on another. What changed is that the recipe which produced foundation models in language and vision, pretrain broadly, then adapt, has been imported into robotics, and it works well enough that a single set of weights now drives many tasks across several robot bodies.

Three empirical facts drove the shift. First, scale in demonstration data helps: policies trained on the pooled Open X-Embodiment corpus, roughly a million real trajectories aggregated from dozens of labs, generalize better than the same architecture trained on any one lab's data. Second, the semantics learned by an internet-pretrained vision-language model transfer to manipulation: a model that has read about apples and knows what a stop sign looks like can be fine-tuned to pick up the apple it has never been trained to grasp, because the hard part, grounding the word in the pixels, is already solved. Third, generative sequence models turned out to be the right tool for the action-prediction problem, because human demonstrations are multimodal, there are many good ways to accomplish a task, and a model that averages them produces a policy that does none of them.

A practitioner today is expected to know why single-step behavior cloning is brittle in a way the theory makes precise, why action chunking and diffusion fix the two different failure modes they address, what a vision-language-action model actually is under the hood, and, most of all, how to evaluate a policy on real hardware without fooling themselves. That last skill is the one most often missing. This page assumes fluency with supervised learning and with the mechanics of transformers and diffusion; the diffusion and large vision models page owns the denoising-diffusion derivation this one builds on, the deep reinforcement learning page owns the RL and imitation-learning background, and the multimodal foundation models page owns the vision-language models that vision-language-action policies are built from.

Core theory

The imitation-learning setup and behavior cloning

Fix a finite-horizon Markov decision process with states \( s \in \mathcal{S} \), actions \( a \in \mathcal{A} \), a transition kernel \( P(s' \mid s, a) \), an initial-state distribution \( \mu_0 \), and a horizon \( T \). A policy \( \pi(a \mid s) \) induces a distribution over length-\( T \) trajectories. Write \( d_\pi^t \) for the marginal distribution of the state at time \( t \) under \( \pi \), and \( d_\pi = \frac{1}{T}\sum_{t=1}^{T} d_\pi^t \) for the time-averaged state distribution. There is an expert \( \pi^\star \) that we can query, and we observe demonstrations, trajectories drawn by rolling out \( \pi^\star \).

Behavior cloning treats this as pure supervised learning. Collect a dataset \( \mathcal{D} = \{(s_i, a_i)\} \) with \( s_i \sim d_{\pi^\star} \) and \( a_i \sim \pi^\star(\cdot \mid s_i) \), and fit a policy by minimizing a surrogate classification or regression loss

This is the algorithm behind ALVINN, Pomerleau's 1988 neural network that steered a van from camera images, and it is still the backbone of most manipulation policies. Its appeal is that it is just fitting a conditional distribution and inherits all the machinery of deep supervised learning. Its defect is the expectation in that objective: the loss is measured under \( d_{\pi^\star} \), the distribution of states the expert visits, but at test time the learned policy \( \hat\pi \) visits states drawn from \( d_{\hat\pi} \), and those two distributions come apart the moment \( \hat\pi \) makes a mistake. This mismatch is the covariate shift that the rest of this section is about.

Covariate shift and why the error is quadratic in the horizon

Define the per-step error of the learned policy as its probability of disagreeing with the expert, measured under the expert's own state distribution:

This \( \epsilon \) is exactly what supervised learning drives down; a validation 0-1 loss of one percent means \( \epsilon = 0.01 \). The question is what this buys us at test time. Following the reduction of Ross and Bagnell (2010), assign a task cost \( C(s, a) \in [0, 1] \) and suppose, worst case, that a single deviation from the expert can be unrecoverable: once the policy takes a wrong action it may enter a region of state space the expert never demonstrated, where we can bound its cost by nothing better than the maximum, \( 1 \) per step, for the rest of the episode.

Model the rollout as a survival process. At each visited state the policy matches the expert with probability \( 1 - \epsilon \) and errs with probability \( \epsilon \). Let \( K \) be the step at which the first error occurs; if the first error is at step \( k \), then steps \( k, k+1, \dots, T \) are all charged the worst-case cost, which is \( T - k + 1 \) bad steps. The probability the first error lands at step \( k \) is \( (1-\epsilon)^{k-1}\epsilon \). The expected number of bad steps is therefore

To see the horizon scaling, take \( \epsilon \) small enough that \( (1-\epsilon)^{k-1}\approx 1 \) over the range that matters. Then the sum collapses to

The quadratic is not an artifact of the worst-case cost assumption; it is the signature of compounding. Each of the \( T \) steps is an opportunity to leave the training distribution, and each departure is paid for over the \( O(T) \) steps that remain. Multiply an \( O(T) \) chance of falling off by an \( O(T) \) tail cost and the total is \( O(T^2) \). Equivalently, Ross and Bagnell's theorem states the bound directly: if the surrogate loss is an upper bound on the 0-1 loss and equals \( \epsilon \), then the expected cost of the cloned policy satisfies

with the quadratic factor \( T^2 \) rather than the naive \( T \) one would hope for from a per-step error \( \epsilon \) accumulated over \( T \) steps. The regret of behavior cloning grows with the square of the horizon.

DAgger and the linear bound

The fix is to measure and reduce error under the distribution the learned policy actually induces, not the expert's. DAgger (Dataset Aggregation), from Ross, Gordon, and Bagnell (2011), does this by iterating. Start with a policy, roll it out to collect the states it visits, query the expert for the correct action at each of those states, aggregate the new labeled pairs into the dataset, and refit. Formally, at iteration \( n \) the policy is trained on the union of all states visited so far, so it is optimized against a distribution that converges to \( d_{\hat\pi} \) rather than \( d_{\pi^\star} \).

Why does this turn the quadratic into a linear bound? Redefine the error under the induced distribution,

which is what DAgger controls, because it trains on exactly those states. Now the compounding argument breaks. When the policy makes an error and enters an off-distribution state, that state is, by construction, in the training distribution on the next iteration, so the policy learns to recover from it rather than diverging. Under the assumption that the task admits recovery, that the expert can return to good behavior from the states the learner visits with a cost-to-go gap bounded by a constant \( u \), each mistake costs \( O(u) \) once rather than \( O(T) \) forever. Summing \( T \) independent single-step costs,

The horizon dependence is linear. The interpretation is precise: the \( T^2 \) of behavior cloning is a product of \( T \) chances to leave the manifold and \( T \) steps of unrecovered cost per departure, and DAgger removes the second factor by making off-manifold states part of training so recovery costs \( O(1) \). This is the central theoretical result of imitation learning, and it explains why the modern stack invests heavily in two things: collecting data that covers the states a deployed policy will actually see (a distributional argument), and architectures that keep \( \epsilon \) itself small (action chunking and diffusion, below). DAgger's practical cost is that it needs an interactive expert, one that can be queried at arbitrary states, which is cheap for a driving simulator but expensive or unsafe for a real robot near a person. Much of the field's engineering is about getting DAgger-like coverage without a live expert.

Action representation: why single actions are brittle

Two failure modes hide inside \( \epsilon \), and they call for different fixes. The first is temporal: a policy that predicts one action per observation is a decision maker at every one of the \( T \) steps, so it has \( T \) opportunities to compound error, and it must also model the fine-grained, high-frequency structure of expert motion, which is exactly the part most polluted by demonstrator jitter and by non-Markovian habits (a human pausing because they blinked, not because the state demanded it). The second is distributional: at a given state there are often several equally good actions, and a model trained with a unimodal regression loss will predict their average, which can be a bad action, this is the multimodality problem, deferred to the diffusion subsection.

Action chunking, introduced with the Action Chunking Transformer (ACT) by Zhao and colleagues in 2023, attacks the temporal failure mode directly. Rather than a policy \( \pi(a_t \mid s_t) \) that emits one action, learn a policy \( \pi(a_{t:t+k} \mid s_t) \) that emits a chunk of \( k \) future actions from a single observation. The robot then executes some or all of that chunk before predicting again. The immediate benefit is a shorter effective decision horizon: an episode of \( T \) physical steps now involves only \( T/k \) policy queries, and it is the number of decisions that drives compounding, so the naive-cloning bound improves from \( O(\epsilon T^2) \) toward \( O\!\big(\epsilon (T/k)^2 \cdot k\big) = O(\epsilon T^2/k) \) in the regime where errors are made per decision and paid per step. Chunking also lets the model absorb non-Markovian demonstrator behavior into a single prediction instead of being forced to explain each idle frame, and it reduces the closed-loop feedback rate, which is the price.

That price is reactivity. A chunk executed open-loop for \( k \) steps cannot respond to a disturbance until it finishes, so the worst-case reaction latency is the chunk duration. ACT resolves the tension with temporal ensembling: instead of executing a chunk and then discarding it, run the policy at every step, so that at any physical time \( t \) several overlapping chunks all contain a prediction for \( a_t \), and combine them with an exponentially weighted average that favors the most recent prediction,

where \( \hat a_t^{(i)} \) is the prediction for \( a_t \) made \( i \) steps ago and \( m \ge 0 \) tunes how fast old predictions decay. The scheme gives a smooth command (an average over many predictions) while still incorporating the newest observation at every step (the \( i = 0 \) term). The parameter \( m \) sets a bias-variance knob between smoothness and reactivity, worked numerically below.

Diffusion policies and multimodal action distributions

The distributional failure mode needs a generative model of actions, not a point predictor. Suppose that from the current state two demonstrated behaviors are equally valid, pushing a T-shaped block to the left of an obstacle or to the right, the canonical pushT example from Chi and colleagues (2023). A network trained to minimize mean-squared error against the demonstrations will predict the mean of the two modes, which steers the block straight into the obstacle, a behavior no demonstrator ever performed. A Gaussian-head policy has the same defect: it can represent one mode or a fat blob covering both, but not two sharp separated modes. What is needed is a model that represents an arbitrary multimodal conditional distribution over actions, and that is exactly what a denoising diffusion model provides.

The diffusion policy models the conditional distribution of an action sequence \( A = a_{t:t+k} \) given an observation \( O \) (a short history of images and proprioception). The diffusion page derives the denoising-diffusion machinery in full; the essential facts reused here are that the forward process noises a clean sample \( A_0 \) into \( A_\tau = \sqrt{\bar\alpha_\tau}\,A_0 + \sqrt{1-\bar\alpha_\tau}\,\varepsilon \) with \( \varepsilon \sim \N(0, I) \), and that a network \( \varepsilon_\theta \) trained to predict the noise learns the score of the data distribution. The only change for policies is that the network is conditioned on the observation, so the training objective is the conditional denoising loss

The derivation is identical to unconditional DDPM with the observation carried through as a conditioning input; the variational bound on \( -\log p_\theta(A_0 \mid O) \) reduces, under the same reparameterization, to this noise-prediction regression. At inference the policy samples an action sequence by reverse diffusion, starting from \( A_N \sim \N(0, I) \) and iterating

for \( \tau = N, \dots, 1 \), which is the standard DDPM ancestral sampler applied to the action tensor. Because the model draws a sample rather than predicting a mean, it commits to one mode of the action distribution per rollout, going left or right around the obstacle but never through it. This is the reason diffusion policies outperform Gaussian and mean-squared-error policies on multimodal manipulation: the loss no longer averages incompatible behaviors.

Two engineering choices make it a policy rather than a generator. First, the network predicts a full chunk \( A = a_{t:t+k} \), so diffusion policies inherit the horizon and smoothness benefits of action chunking for free; the temporal structure of the sequence is modeled jointly by the denoiser. Second, execution is receding-horizon: the policy predicts \( k \) actions but executes only the first \( k_a < k \) of them before re-observing and re-sampling, a model-predictive-control style that keeps the closed loop reactive while still planning a short way ahead. The unexecuted tail of the chunk is discarded. The cost of diffusion is inference latency, each action sequence needs several denoising steps, which is why practical systems use few-step samplers (DDIM, or a small \( N \)) and why the receding horizon executes a handful of actions per expensive sampling call.

The vision-language-action paradigm

A vision-language-action (VLA) model is a pretrained vision-language model that has been fine-tuned to output actions. The bet is that the expensive, data-hungry part of manipulation, perceiving a scene and grounding a natural-language instruction in it, is already solved by a VLM pretrained on internet-scale image-text data, and that adding motor control is comparatively cheap. RT-2, from Brohan and colleagues in 2023, made the bet concrete: take a large VLM (a PaLI-style image-text transformer), and co-fine-tune it on both its original vision-language data and robot trajectories, where each trajectory step is turned into a text string the model can emit. The claim, borne out empirically, is that internet semantics transfer, a VLA can follow instructions referencing objects and concepts it never saw in robot data, because the concept was learned from the web and the fine-tuning only had to attach it to motor output.

The mechanism that lets a language model emit an action is action tokenization: discretize each continuous action dimension into a finite set of bins and represent the resulting integers as tokens the transformer already knows. RT-2 discretizes each of the seven action dimensions (three for end-effector translation, three for rotation, one for the gripper) into 256 uniform bins and maps each bin to a token, so an action becomes a length-7 string of tokens and the policy is literally next-token prediction with a cross-entropy loss. The alternative, a continuous regression head bolted onto the VLM, is what OpenVLA (Kim and colleagues, 2024) and later systems often prefer for high-frequency control, because discretization caps the achievable precision and a continuous head sidesteps the token budget. The tradeoff is worked below: 256 bins over \( [-1, 1] \) fix the action resolution at a bin width of \( 2/256 \approx 0.0078 \), a maximum quantization error of half a bin, which is fine for coarse pick-and-place and too coarse for contact-rich insertion.

The other pillar of the VLA program is cross-embodiment training. Robots differ in their arms, grippers, cameras, and control frequencies, and historically each lab trained on its own robot's data only. The Open X-Embodiment collaboration (2023) pooled data from dozens of labs and robots into a single dataset of roughly a million trajectories in a unified format, and trained RT-X models on the union. The finding was positive transfer: a policy trained on many embodiments outperformed the same policy trained on any single embodiment's data, including on that embodiment's own tasks. The intuition is that manipulation shares structure across bodies, the geometry of reaching, grasping, and placing is largely embodiment-agnostic once expressed in end-effector coordinates, so pooling data amortizes the cost of learning that shared structure. This is the robotics analogue of multilingual transfer in language models, and it is why the field now builds datasets, not just models. The vision-language backbone itself is the subject of the multimodal foundation models page; a VLA is that backbone with an action head and robot data.

Generalist policies and the current model families

Several lines of work now compete to be the general-purpose robot policy, and they differ mainly in how they represent actions and how much they lean on a pretrained language backbone. The RT line from Google, RT-1 (2022, a transformer over discretized actions trained on a large single-embodiment dataset) then RT-2 (2023, the VLA above), established that transformer policies scale with data. Octo (Octo Model Team, 2024) is an open, cross-embodiment transformer policy trained on Open X-Embodiment with a diffusion action head, designed to be fine-tuned onto new robots and sensors with modest data; it is deliberately a smaller, from-scratch policy rather than a fine-tuned VLM. OpenVLA (2024) is the open-weights VLA: a 7-billion-parameter model built on a Llama-2 language backbone with visual encoders, fine-tuned on Open X-Embodiment, and it demonstrated that an open VLA could match or beat the much larger closed RT-2 on a broad task suite. \( \pi_0 \) (Pi-zero, Black and colleagues at Physical Intelligence, 2024) combines a VLM backbone with a flow-matching action head that generates high-frequency continuous action chunks, aiming at dexterous, 50 Hz control that discretized VLAs struggle to reach. The through-line is convergence on a recipe, pretrained multimodal backbone plus a generative action head plus cross-embodiment data, with the open questions being which action representation and how much language.

World models: planning and data augmentation

A parallel program learns a model of the world, a predictive model of how observations and states evolve under actions, and uses it either to plan or to manufacture training data. The Dreamer line (Hafner and colleagues, v1 through v3) learns a compact latent dynamics model from experience and trains a policy entirely inside the learned model, imagining rollouts in latent space rather than acting in the world, which is dramatically more sample-efficient; the deep reinforcement learning page covers model-based RL and the Dreamer objective in more depth. DreamerV3 notably solved a range of tasks with a single set of hyperparameters, evidence that a learned world model plus imagination-based policy learning generalizes across domains.

The newer and more distinctly foundation-model flavored idea is to use a large video generation model as the world model, or even as the policy. A video model trained on internet and robot footage learns the dynamics of the visual world implicitly; conditioned on a current frame and a goal or instruction, it can generate a plausible future video of the task being accomplished, and an inverse-dynamics model or a controller can then extract the actions that would realize that imagined video. This turns the abundant, cheap resource, video, into a source of manipulation supervision, sidestepping the action-labeled data bottleneck. It also serves as data augmentation: a learned dynamics model can generate synthetic rollouts to expand the training distribution, a way to approximate DAgger-style coverage without a live expert. The open question is fidelity, generated futures that are physically implausible teach the wrong dynamics, and the field is actively working out when video-model supervision helps versus when it hallucinates.

Sim-to-real transfer and domain randomization

Real robot data is expensive; simulation is nearly free. The catch is the reality gap: a policy trained in a simulator with idealized dynamics fails on hardware whose friction, mass, latency, and sensor noise differ from the sim. Domain randomization, from Tobin and colleagues (2017) for vision and Peng and colleagues (2018) for dynamics, is the dominant fix. Rather than trying to match the simulator to reality (system identification), randomize the simulator's parameters over a broad distribution during training, so the policy is forced to be robust to any dynamics in that range, and the real robot, if its true parameters fall inside the range, is just one more sample.

The robustness has a clean second-order explanation. Let \( \mu \) be a dynamics parameter (say a friction coefficient) with nominal value \( \mu_0 \), and let \( J(\mu) \) be the return of a fixed policy when the true parameter is \( \mu \). Suppose \( J \) is smooth and peaks at \( \mu_0 \) when the policy is tuned to the nominal model; Taylor expanding to second order,

with \( J'(\mu_0) = 0 \) at the peak and \( J''(\mu_0) < 0 \). A policy trained only on \( \mu_0 \) is optimized for the peak and can have a sharply curved \( J \), a large \( |J''| \), so a real-world offset \( \mu - \mu_0 \) drops performance quadratically. Training on a randomized \( \mu \sim p \) instead maximizes the expected return,

where the linear term vanishes for a symmetric \( p \). Because \( J'' < 0 \), the objective now penalizes curvature in proportion to the randomization variance: the optimizer prefers policies whose \( J(\mu) \) is flat, robust, over sharply peaked ones, even at some cost to peak performance. Domain randomization is, to second order, a regularizer on the sensitivity of return to dynamics. The tradeoff against system identification is direct: sysID measures the true \( \mu \) and trains a specialist for it, achieving the peak but failing if the measurement is wrong or drifts; randomization gives up the peak for a guarantee that covers a whole range. Too wide a randomization range and no single policy can be good across all of it (a conservative, sluggish policy results); too narrow and the real robot falls outside it. The art is calibrating the range.

The most successful sim-to-real systems combine randomization with a teacher-student distillation that exploits privileged information. In the ANYmal legged-locomotion work (Lee and colleagues, 2020; Miki and colleagues, 2022), a teacher policy is trained in simulation with access to privileged state the real robot cannot measure, exact terrain geometry, contact forces, true friction, using reinforcement learning where such ground truth is available for free. A student policy is then trained by supervised imitation of the teacher but restricted to the observations a real robot actually has, a history of proprioception and noisy exteroception. The student learns to infer the privileged information implicitly from its observable history, and it is what deploys. This is imitation learning again, with the teacher as an interactive, always-queryable expert inside the simulator, which neatly sidesteps DAgger's need for a live human expert: the teacher can be queried at any state the student visits, so the student gets DAgger-quality coverage for free. The combination, randomized dynamics plus privileged teacher-student distillation, is what put robust legged locomotion on real hardware in the wild.

Data, the collection bottleneck, and honest evaluation

The binding constraint on the whole enterprise is action-labeled real robot data. Unlike text and images, which exist on the internet in quantities no one can consume, robot demonstrations must be collected, usually by teleoperation, a human driving the robot through the task, which is slow, expensive, and produces one trajectory at a time. DROID (Khazatsky and colleagues, 2024) is a deliberately diverse teleop dataset spanning hundreds of scenes and many operators; BridgeData and BridgeData V2 (Walke and colleagues, 2023) similarly targeted breadth of tasks and environments. These datasets, folded into Open X-Embodiment, are the fuel. The cost structure is why simulation, cross-embodiment pooling, and video-model supervision are pursued so hard: each is a way to get more effective training signal per dollar of human teleoperation time.

The second thing that separates real robotics from a benchmark leader board is evaluation. Simulation benchmarks are seductive and misleading: a policy can overfit to a simulator's quirks and report a success rate that evaporates on hardware, because the sim reward does not capture the failure modes that matter (a grasp that looks successful in sim but slips on a real object). The only trustworthy evaluation is a real-robot A/B: run policy A and policy B on the same physical tasks, many times, and compare success rates with confidence intervals. Here the small-\( N \) problem bites hard. Real trials are expensive, so a lab might run 20 or 50 per condition, and at that scale the point estimate of a success rate is nearly meaningless without an interval. A policy that succeeds 9 times out of 10 and one that succeeds 90 times out of 100 both report 90%, but the first has a 95% confidence interval so wide it is nearly useless, while the second is tight enough to act on. Reporting a bare success rate from 20 trials, as many papers do, hides exactly the uncertainty a reader needs.

The right tool is a binomial confidence interval that behaves well for small \( N \) and extreme rates, where the textbook Wald ("normal approximation") interval fails badly, it can even extend past 0 or 1. The Wilson score interval (Wilson, 1927) is the standard robust choice. For \( s \) successes in \( n \) trials with \( \hat p = s/n \) and z-value \( z \) (1.96 for 95%), the interval is centered at

The \( z^2/2n \) term shrinks the center toward \( 1/2 \), which is what fixes the behavior at extreme rates: a Wilson interval for \( 20/20 \) does not collapse to a single point the way the Wald interval does. The Clopper-Pearson interval, built from the exact binomial tails via the Beta distribution, is the conservative alternative, guaranteed to cover at the nominal level but wider than Wilson. Either is acceptable; a bare point estimate is not. The next problem works both, and the implementation section verifies them in Python against SciPy.

Implementation

Four code blocks follow. The first is a single conditional denoising-diffusion training step for an action-sequence policy, in PyTorch and JAX; it is the loss above turned into runnable tensors. The second is an action-chunking transformer head with temporal ensembling, again in both frameworks. The third computes the behavior-cloning versus DAgger compounding numbers from first principles. The fourth computes and verifies the Wilson and Clopper-Pearson evaluation intervals. The measured shapes assume a batch of observations already encoded to a conditioning vector; the vision encoder is omitted for focus.

Diffusion-policy denoising step

The step samples a random diffusion timestep per example, noises the clean action chunk with the closed-form marginal, predicts the noise conditioned on the observation embedding, and returns the mean-squared error. This is the entire training signal for a diffusion policy; everything else is architecture and the sampler.

import torch
import torch.nn as nn
import torch.nn.functional as F

def make_alpha_bar(num_steps: int) -> torch.Tensor:
    # cosine-ish linear beta schedule; returns cumulative product alpha_bar
    betas = torch.linspace(1e-4, 0.02, num_steps)          # (N,)
    alphas = 1.0 - betas
    return torch.cumprod(alphas, dim=0)                     # (N,)

class EpsNet(nn.Module):
    # predicts noise on an action chunk, conditioned on obs embed + timestep
    def __init__(self, act_dim, horizon, obs_dim, hidden=512):
        super().__init__()
        self.horizon, self.act_dim = horizon, act_dim
        in_dim = act_dim * horizon + obs_dim + 1           # +1 for timestep
        self.net = nn.Sequential(
            nn.Linear(in_dim, hidden), nn.SiLU(),
            nn.Linear(hidden, hidden), nn.SiLU(),
            nn.Linear(hidden, act_dim * horizon),
        )

    def forward(self, a_noisy, t_frac, obs):
        # a_noisy: (B, H, A)  t_frac: (B, 1)  obs: (B, obs_dim)
        B = a_noisy.shape[0]
        x = torch.cat([a_noisy.reshape(B, -1), t_frac, obs], dim=-1)
        return self.net(x).reshape(B, self.horizon, self.act_dim)

def diffusion_policy_loss(model, alpha_bar, actions, obs):
    # actions: (B, H, A) clean chunk   obs: (B, obs_dim)
    B, H, A = actions.shape
    N = alpha_bar.shape[0]
    tau = torch.randint(0, N, (B,), device=actions.device)  # (B,)
    ab = alpha_bar[tau].view(B, 1, 1)                        # (B,1,1)
    eps = torch.randn_like(actions)                         # (B,H,A)
    a_noisy = torch.sqrt(ab) * actions + torch.sqrt(1 - ab) * eps
    t_frac = (tau.float() / N).view(B, 1)                   # (B,1) in [0,1)
    eps_hat = model(a_noisy, t_frac, obs)                   # (B,H,A)
    return F.mse_loss(eps_hat, eps)

if __name__ == "__main__":
    torch.manual_seed(0)
    B, H, A, O, N = 32, 16, 7, 128, 100
    model = EpsNet(A, H, O)
    alpha_bar = make_alpha_bar(N)
    actions = torch.randn(B, H, A)
    obs = torch.randn(B, O)
    loss = diffusion_policy_loss(model, alpha_bar, actions, obs)
    print("diffusion-policy loss:", float(loss))           # ~1.0 at init

import jax, jax.numpy as jnp
import flax.linen as nn
import optax

def make_alpha_bar(num_steps):
    betas = jnp.linspace(1e-4, 0.02, num_steps)            # (N,)
    return jnp.cumprod(1.0 - betas)                        # (N,)

class EpsNet(nn.Module):
    act_dim: int
    horizon: int
    hidden: int = 512
    @nn.compact
    def __call__(self, a_noisy, t_frac, obs):
        # a_noisy: (B, H, A)  t_frac: (B, 1)  obs: (B, obs_dim)
        B = a_noisy.shape[0]
        x = jnp.concatenate([a_noisy.reshape(B, -1), t_frac, obs], axis=-1)
        x = nn.silu(nn.Dense(self.hidden)(x))
        x = nn.silu(nn.Dense(self.hidden)(x))
        x = nn.Dense(self.act_dim * self.horizon)(x)
        return x.reshape(B, self.horizon, self.act_dim)

def diffusion_policy_loss(params, apply_fn, alpha_bar, actions, obs, key):
    # actions: (B, H, A)  obs: (B, obs_dim)
    B, H, A = actions.shape
    N = alpha_bar.shape[0]
    k1, k2 = jax.random.split(key)
    tau = jax.random.randint(k1, (B,), 0, N)               # (B,)
    ab = alpha_bar[tau].reshape(B, 1, 1)                    # (B,1,1)
    eps = jax.random.normal(k2, actions.shape)             # (B,H,A)
    a_noisy = jnp.sqrt(ab) * actions + jnp.sqrt(1 - ab) * eps
    t_frac = (tau.astype(jnp.float32) / N).reshape(B, 1)   # (B,1)
    eps_hat = apply_fn(params, a_noisy, t_frac, obs)       # (B,H,A)
    return jnp.mean((eps_hat - eps) ** 2)

if __name__ == "__main__":
    B, H, A, O, N = 32, 16, 7, 128, 100
    key = jax.random.PRNGKey(0)
    model = EpsNet(act_dim=A, horizon=H)
    alpha_bar = make_alpha_bar(N)
    actions = jax.random.normal(key, (B, H, A))
    obs = jax.random.normal(key, (B, O))
    params = model.init(key, actions, jnp.zeros((B, 1)), obs)
    loss = diffusion_policy_loss(params, model.apply, alpha_bar,
                                 actions, obs, key)
    print("diffusion-policy loss:", float(loss))           # ~1.0 at init

Action-chunking transformer head with temporal ensembling

The head maps an observation embedding to a chunk of \( k \) actions in one forward pass (the deterministic ACT-style head; the full model adds a transformer encoder-decoder and a CVAE latent, omitted here). The temporal-ensemble buffer stores the overlapping predictions and returns the exponentially weighted command for the current step.

import torch
import torch.nn as nn
import numpy as np

class ChunkHead(nn.Module):
    # obs embedding -> (horizon, act_dim) chunk in one shot
    def __init__(self, obs_dim, act_dim, horizon, hidden=512):
        super().__init__()
        self.horizon, self.act_dim = horizon, act_dim
        self.net = nn.Sequential(
            nn.Linear(obs_dim, hidden), nn.SiLU(),
            nn.Linear(hidden, hidden), nn.SiLU(),
            nn.Linear(hidden, horizon * act_dim),
        )
    def forward(self, obs):                                 # obs: (B, obs_dim)
        B = obs.shape[0]
        return self.net(obs).reshape(B, self.horizon, self.act_dim)

class TemporalEnsemble:
    # combine overlapping chunk predictions with weights exp(-m i)
    def __init__(self, horizon, act_dim, m=0.1):
        self.H, self.A, self.m = horizon, act_dim, m
        self.buf = np.full((horizon, horizon, act_dim), np.nan)  # (age,slot,A)
    def step(self, chunk):                                  # chunk: (H, A) np
        self.buf = np.roll(self.buf, 1, axis=0)            # age everything
        self.buf[0] = chunk                                 # newest at age 0
        preds, ages = [], []
        for age in range(self.H):
            slot = self.buf[age, age]                       # prediction for now
            if not np.isnan(slot).any():
                preds.append(slot); ages.append(age)
        preds = np.stack(preds); ages = np.array(ages)
        w = np.exp(-self.m * ages); w = w / w.sum()
        return (w[:, None] * preds).sum(axis=0)             # (A,) command

if __name__ == "__main__":
    torch.manual_seed(0)
    H, A = 8, 7
    head = ChunkHead(128, A, H)
    te = TemporalEnsemble(H, A, m=0.1)
    for _ in range(12):
        obs = torch.randn(1, 128)
        chunk = head(obs).detach().numpy()[0]              # (H, A)
        cmd = te.step(chunk)                                # (A,)
    print("command dim:", cmd.shape, "example:", np.round(cmd[:3], 3))

import jax, jax.numpy as jnp
import flax.linen as nn
import numpy as np

class ChunkHead(nn.Module):
    act_dim: int
    horizon: int
    hidden: int = 512
    @nn.compact
    def __call__(self, obs):                                # obs: (B, obs_dim)
        B = obs.shape[0]
        x = nn.silu(nn.Dense(self.hidden)(obs))
        x = nn.silu(nn.Dense(self.hidden)(x))
        x = nn.Dense(self.horizon * self.act_dim)(x)
        return x.reshape(B, self.horizon, self.act_dim)

class TemporalEnsemble:
    # combine overlapping chunk predictions with weights exp(-m i)
    def __init__(self, horizon, act_dim, m=0.1):
        self.H, self.A, self.m = horizon, act_dim, m
        self.buf = np.full((horizon, horizon, act_dim), np.nan)
    def step(self, chunk):                                  # chunk: (H, A) np
        self.buf = np.roll(self.buf, 1, axis=0)
        self.buf[0] = chunk
        preds, ages = [], []
        for age in range(self.H):
            slot = self.buf[age, age]
            if not np.isnan(slot).any():
                preds.append(slot); ages.append(age)
        preds = np.stack(preds); ages = np.array(ages)
        w = np.exp(-self.m * ages); w = w / w.sum()
        return (w[:, None] * preds).sum(axis=0)             # (A,) command

if __name__ == "__main__":
    H, A = 8, 7
    key = jax.random.PRNGKey(0)
    head = ChunkHead(act_dim=A, horizon=H)
    params = head.init(key, jnp.zeros((1, 128)))
    te = TemporalEnsemble(H, A, m=0.1)
    for i in range(12):
        obs = jax.random.normal(jax.random.PRNGKey(i), (1, 128))
        chunk = np.asarray(head.apply(params, obs))[0]      # (H, A)
        cmd = te.step(chunk)                                # (A,)
    print("command dim:", cmd.shape, "example:", np.round(cmd[:3], 3))

Behavior-cloning versus DAgger compounding

This computes the exact expected bad-step count of behavior cloning as the survival sum, the small-error quadratic approximation, and the linear DAgger figure, reproducing the numbers used in Problem 1.

import numpy as np

def bc_expected_bad_steps(eps: float, T: int) -> float:
    # first error at step k (1-indexed) -> steps k..T are bad -> T-k+1 bad
    k = np.arange(1, T + 1)
    p_first = (1 - eps) ** (k - 1) * eps        # P(first error at k)
    bad = (T - k + 1)                            # bad steps if first error at k
    return float(np.sum(p_first * bad))          # no-error trajectories add 0

for eps, T in [(0.01, 500), (0.05, 200), (0.02, 400)]:
    bc = bc_expected_bad_steps(eps, T)
    quad = eps * T * (T + 1) / 2                  # small-eps approx  O(eps T^2)
    dagger = eps * T                              # linear bound     O(eps T)
    print(f"eps={eps} T={T}: BC={bc:7.2f}  quad~{quad:8.2f}  "
          f"DAgger={dagger:6.2f}  ratio={bc / dagger:5.1f}")
# eps=0.01 T=500: BC= 401.65  quad~ 1252.50  DAgger=  5.00  ratio= 80.3
# eps=0.05 T=200: BC= 181.00  quad~ 1005.00  DAgger= 10.00  ratio= 18.1
# eps=0.02 T=400: BC= 351.02  quad~ 1604.00  DAgger=  8.00  ratio= 43.9

Real-robot evaluation intervals

This computes the Wilson and Clopper-Pearson intervals, the two-proportion test, and the power calculation from Problem 6, and it checks the Wilson center against the SciPy exact interval. Run on this machine with NumPy 2.2.6 and SciPy 1.15.3 (the older NumPy 1.21 on the host mis-linked LAPACK, so factorizations were unreliable; these computations use only elementary functions and the Beta quantile, which are unaffected, but the fresh stack is used regardless).

import numpy as np
from scipy import stats

def wilson(s, n, z=1.96):
    p = s / n
    denom = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denom
    half = (z / denom) * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return center - half, center + half

def clopper_pearson(s, n):
    lo = stats.beta.ppf(0.025, s, n - s + 1) if s > 0 else 0.0
    hi = stats.beta.ppf(0.975, s + 1, n - s) if s < n else 1.0
    return lo, hi

for s, n in [(42, 50), (9, 10), (90, 100)]:
    wl, wh = wilson(s, n)
    cl, ch = clopper_pearson(s, n)
    print(f"{s}/{n}={s/n:.0%}: Wilson=[{wl:.3f},{wh:.3f}] w={wh-wl:.3f}  "
          f"CP=[{cl:.3f},{ch:.3f}]")
# 42/50=84%: Wilson=[0.715,0.917] w=0.202  CP=[0.709,0.928]
# 9/10=90%:  Wilson=[0.596,0.982] w=0.386  CP=[0.555,0.997]
# 90/100=90%: Wilson=[0.826,0.945] w=0.119  CP=[0.824,0.951]

# two-policy A/B: A=42/50, B=36/50
sA, nA, sB, nB = 42, 50, 36, 50
pool = (sA + sB) / (nA + nB)
se = np.sqrt(pool * (1 - pool) * (1 / nA + 1 / nB))
z = (sA / nA - sB / nB) / se
p_two = 2 * (1 - stats.norm.cdf(abs(z)))
_, p_fisher = stats.fisher_exact([[sA, nA - sA], [sB, nB - sB]])
print(f"A/B: z={z:.2f} p_z={p_two:.3f} p_fisher={p_fisher:.3f}")
# A/B: z=1.45 p_z=0.148 p_fisher=0.227   -> NOT significant at 50/arm

# trials per arm to detect 84% vs 72% at 80% power, alpha=0.05
za, zb, p1, p2 = 1.96, 0.84, 0.84, 0.72
pbar = (p1 + p2) / 2
n_arm = ((za * np.sqrt(2 * pbar * (1 - pbar))
          + zb * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2) / (p1 - p2) ** 2
print("trials per arm:", int(np.ceil(n_arm)))              # 186

How it is done in practice

A deployed manipulation policy in 2025 is usually one of two shapes. It is either a fine-tuned vision-language-action model, a multi-billion parameter transformer with a pretrained vision-language backbone and an action head, running at a few Hertz and driving coarse-to-medium manipulation from natural-language instructions; or it is a smaller, purpose-built diffusion or chunking policy, tens of millions of parameters, trained on a few hundred to a few thousand demonstrations of a target task family and running at tens of Hertz for dexterous, contact-rich control. The two are converging: \( \pi_0 \) and its successors put a flow-matching or diffusion action head on a VLM backbone to get both the semantics of the large model and the high-frequency continuous control of the small one.

The engineering that separates a demo from a product lives in the unglamorous parts. Observation and action normalization has to be computed per-dataset and applied consistently, because cross-embodiment training mixes robots with wildly different action scales, and a policy that saw the wrong normalization statistics fails silently. Inference latency is a first-class constraint: a diffusion policy that needs 50 denoising steps is unusable for closed-loop control unless distilled to a few-step sampler or run with a receding horizon that amortizes each expensive sample over several executed actions. The vision encoder dominates compute and is often frozen or shared. And the control stack underneath the policy, a real-time loop that clamps commands, enforces joint limits, and falls back to a safe behavior when the policy stalls, is what keeps a learned policy from destroying hardware. As a rough sense of the compute available for the research-scale experiments and distillation runs this page's methods imply, the H100 80GB HBM3 in this repository sustains about 745 TFLOP/s of bf16 matmul at 4096-dimension and roughly 3.0 TB/s of memory bandwidth (from the repository's classes/data/h100.json benchmark), which comfortably trains the small diffusion and chunking policies and fine-tunes a 7B OpenVLA-scale model with parameter-efficient adapters.

Data pipelines are built around the RLDS/TFDS format that Open X-Embodiment standardized, and the practical work of adding a new robot is mostly writing a loader that maps its raw teleop logs into that schema with correct action-space conventions. Evaluation is run as a fixed protocol, a defined set of tasks, object placements, and success criteria, executed enough times to compute the confidence intervals above, with A/B comparisons rather than absolute numbers because absolute success rates drift with lighting, object wear, and operator setup. The labs that publish trustworthy results are the ones that report intervals and run enough trials; the small-\( N \) trap in Problem 6 is the single most common way robot papers overclaim.

  teleop / sim rollout            pretraining                fine-tune + deploy
  ------------------            -----------             ------------------
  human demos  ----\         internet image-text          target-robot demos
  privileged   ----+---> [ VLM backbone ] --co-train--> [ VLA / diffusion head ]
   teacher (sim)  /        (semantics, grounding)              |
  cross-embodiment                                     receding-horizon exec
  (Open X, DROID)                                        @ 5-50 Hz on hardware
                                                                |
                                                     real-robot A/B + Wilson CI
          

The current research frontier

The most active question is the action representation. Discretized tokenization (RT-2, OpenVLA) is simple and reuses the language model's machinery but caps precision and control frequency; continuous diffusion and flow-matching heads (Octo, \( \pi_0 \), Physical Intelligence's line) reach higher frequency and finer control but add inference cost and complicate the clean next-token-prediction story. Groups at Google DeepMind, Physical Intelligence, Berkeley, Stanford, and the open Octo and LeRobot communities are pushing different points on this tradeoff, and it is not settled which wins for which task regime.

The RDT line from Tsinghua is a clean case study because one group has now built both ends of the tradeoff. RDT-1B (2024) is a 1.2B-parameter diffusion foundation model for bimanual manipulation, pretrained on multi-robot data and fine-tuned on dual-arm demonstrations. Its successor RDT-2 (2025) moves to the autoregressive side. The RDT2-VQ variant adapts the Qwen2.5-VL-7B-Instruct vision-language backbone into a VLA that predicts 24-step relative action chunks through a lightweight residual vector-quantized action tokenizer, trains on large-scale UMI bimanual data, and aims at zero-shot deployment on unseen embodiments for simple open-vocabulary skills. A residual quantizer with chunked decoding recovers much of the precision that plain per-dimension binning gives up, which narrows the gap the diffusion heads were built to close. On the tokenized side, OpenVLA-OFT showed that the fine-tuning recipe matters as much as the architecture, with parallel decoding, action chunking, and a continuous action head lifting both control frequency and success well above the original autoregressive OpenVLA recipe.

Above the representation question sits a race between generalist systems. Physical Intelligence followed \( \pi_0 \) with \( \pi_{0.5} \), which co-trains on heterogeneous data sources so the policy generalizes to homes it never saw during training. Google DeepMind's Gemini Robotics attaches action output to the Gemini multimodal backbone, and NVIDIA's GR00T N1 targets humanoids with a dual-system design that pairs a slow vision-language module with a fast diffusion action expert. Alongside these imitation-first systems, Berkeley's SERL and HIL-SERL line showed that sample-efficient reinforcement learning with human-in-the-loop corrections can reach near-perfect success on contact-rich tasks within a few hours of real-robot training, a route past the ceiling that demonstration data imposes on the imitation-only stack.

The second frontier is data. Teleoperation does not scale, and the field is exploring four escapes. Cross-embodiment pooling (Open X-Embodiment, RT-X) extracts more from existing data by sharing across robots. Cheaper collection hardware attacks the cost directly, the UMI handheld gripper decouples demonstration from any particular robot so in-the-wild bimanual data can be gathered at walking pace, Mobile ALOHA extends low-cost whole-body teleoperation to mobile manipulation, and RDT-2 training on large-scale UMI data is an early sign that device-collected data can carry a foundation model on its own. Simulation with domain randomization and teacher-student distillation, mature for locomotion (ANYmal), is being pushed toward manipulation, where contact-rich dynamics are much harder to simulate faithfully. And learning from video, using internet and human video as a source of dynamics and even action supervision through inverse models, is the highest-ceiling and least-proven direction, with work at Meta AI, Google DeepMind, and several academic groups. World models in the Dreamer lineage, and large video-generation models repurposed as world models, sit at the intersection of the data and planning questions.

The third frontier is evaluation and reliability. There is growing recognition, argued forcefully by several groups after a run of irreproducible robot results, that simulation benchmarks and underpowered real evaluations have been misleading the field, and that standardized real-robot protocols with proper statistics are needed for the claims to mean anything. This is less glamorous than a new architecture but is arguably the bottleneck on progress: without honest evaluation the field cannot tell which of the competing action representations and data strategies actually works. DROID and the evaluation practices around it are early steps toward shared, statistically meaningful benchmarks.

Open source to read

• real-stanford/diffusion_policy — the reference diffusion-policy implementation from Chi et al. Open diffusion_policy/policy/diffusion_unet_lowdim_policy.py to see the conditional denoiser and the receding-horizon action sampling in one place.
• tonyzhaozh/act — the Action Chunking Transformer and the ALOHA teleop setup. Start at policy.py for the CVAE-style chunking head and imitate_episodes.py for the temporal-ensembling inference loop.
• openvla/openvla — the open 7B vision-language-action model. Read prismatic/models for the VLM-plus-action-head construction and the action de/tokenization utilities.
• octo-models/octo — an open cross-embodiment transformer policy with a diffusion head, built for fine-tuning onto new robots. The finetuning script and the observation/action tokenizers are the useful entry points.
• huggingface/lerobot — a broad library that packages ACT, diffusion policy, and VLA training with standardized datasets; the fastest way to run any of these end to end. Start from the policy configs and the dataset loaders.
• google-deepmind/open_x_embodiment — loaders and documentation for the pooled cross-embodiment dataset. The dataset-transform notebooks show the unified action-space conventions every cross-embodiment policy must respect.
• Farama-Foundation/Gymnasium-Robotics — standardized manipulation and fetch/relocate environments for controlled evaluation and for generating imitation data in simulation.
• google-deepmind/mujoco — the physics engine underneath most of this simulation work. Understanding its contact model is essential for interpreting where sim-to-real gaps for manipulation come from.

Common misconceptions

"Behavior cloning fails because the network underfits; a bigger model or more demonstrations fixes it." The failure is distributional, not a capacity problem. Even a model that perfectly fits the demonstrations, driving \( \epsilon \) toward zero on the expert's distribution, still suffers \( O(\epsilon T^2) \) compounding at test time because it is evaluated on the distribution it induces, not the one it trained on. More demonstrations of the same states help only if they cover the states the deployed policy will visit, which is DAgger's insight, not a scaling one.

"DAgger is strictly better than behavior cloning, so everyone should use it." DAgger needs an interactive expert that can be queried at arbitrary, possibly dangerous, states, which is cheap in a simulator but often impossible or unsafe on real hardware near people. Much of the modern stack is about getting DAgger-like state coverage without a live expert, through broad data collection, teacher-student distillation in simulation, or generated rollouts. The theory says what to aim for; the engineering is about approximating it.

"A diffusion policy is slow because diffusion is slow, so it cannot do real-time control." The naive sampler with dozens of steps is slow, but few-step samplers, distillation, and receding-horizon execution, where one expensive sample yields several executed actions, bring diffusion policies to tens of Hertz. The multimodality benefit is worth the cost, and the cost is an engineering variable, not a fixed property.

"A vision-language-action model understands physics because it was pretrained on the internet." It understands semantics, what objects are and what instructions mean, which is exactly the part that transfers. It does not learn contact dynamics or force control from web text and images; those come only from the robot data or from simulation. Conflating semantic grounding with physical competence is the most common overestimate of what VLAs can do.

"Domain randomization is a hack; system identification is the principled approach." Randomization has a clean second-order justification, it is a regularizer on the sensitivity of return to dynamics, and it buys a bounded worst-case loss over a range of real conditions. System identification achieves a higher peak but fails when its measurement is wrong or the dynamics drift. Neither dominates; the strong systems combine a randomized-and-distilled student with as much calibration as is cheaply available.

"A policy that succeeds 18 of 20 times is clearly better than one that succeeds 16 of 20." The 95% Wilson interval for 18/20 is roughly [70%, 97%] and for 16/20 roughly [58%, 92%]; they overlap almost entirely, and a two-proportion test does not distinguish them. At 20 trials, a two-out-of-twenty difference is noise. Distinguishing policies that truly differ by a dozen points of success rate takes on the order of a hundred to two hundred trials per arm, which is why bare success rates from small evaluations should be read with suspicion.

"Action chunking is just for efficiency, running the policy less often." Efficiency is a side effect. The primary benefits are statistical: chunking shortens the effective decision horizon so compounding error scales like \( T/k \) instead of \( T \), and it lets the model absorb non-Markovian demonstrator behavior into a single prediction. The reduced query rate is what makes latency-hiding possible, but it is not the reason chunking improves task success.

Self-check

References

1. Sutton, R. and Barto, A. (2018). Reinforcement Learning: An Introduction, 2nd edition. MIT Press. book site.
2. Kochenderfer, M., Wheeler, T., and Wray, K. (2022). Algorithms for Decision Making. MIT Press. algorithmsbook.com.
3. Pomerleau, D. (1988). ALVINN: An Autonomous Land Vehicle in a Neural Network. Advances in Neural Information Processing Systems (NeurIPS). proceedings.
4. Ross, S. and Bagnell, D. (2010). Efficient Reductions for Imitation Learning. AISTATS. PMLR v9.
5. Ross, S., Gordon, G., and Bagnell, D. (2011). A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (DAgger). AISTATS. arXiv:1011.0686.
6. Zhao, T., Kumar, V., Levine, S., and Finn, C. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT / ALOHA). Robotics: Science and Systems (RSS). arXiv:2304.13705.
7. Fu, Z., Zhao, T. Z., and Finn, C. (2024). Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation. CoRL. arXiv:2401.02117.
8. Chi, C., Xu, Z., Pan, C., Cousineau, E., Burchfiel, B., Feng, S., Tedrake, R., and Song, S. (2024). Universal Manipulation Interface: In-the-Wild Robot Teaching Without In-the-Wild Robots. RSS. arXiv:2402.10329.
9. Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS. arXiv:2303.04137.
10. Ho, J., Jain, A., and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS. arXiv:2006.11239.
11. Brohan, A., Brown, N., Carbajal, J., et al. (2022). RT-1: Robotics Transformer for Real-World Control at Scale. arXiv:2212.06817.
12. Brohan, A., Brown, N., Carbajal, J., et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv:2307.15818.
13. Open X-Embodiment Collaboration (2023). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. arXiv:2310.08864.
14. Kim, M., Pertsch, K., Karamcheti, S., et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model. arXiv:2406.09246.
15. Kim, M. J., Finn, C., and Liang, P. (2025). Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA-OFT). arXiv:2502.19645.
16. Octo Model Team (2024). Octo: An Open-Source Generalist Robot Policy. RSS. arXiv:2405.12213.
17. Black, K., Brown, N., Driess, D., et al. (2024). Pi-0: A Vision-Language-Action Flow Model for General Robot Control. Physical Intelligence. arXiv:2410.24164.
18. Black, K., Brown, N., et al. (2025). Pi-0.5: A Vision-Language-Action Model with Open-World Generalization. Physical Intelligence. arXiv:2504.16054.
19. Liu, S., Wu, L., Li, B., et al. (2024). RDT-1B: A Diffusion Foundation Model for Bimanual Manipulation. Tsinghua University. arXiv:2410.07864.
20. RDT Team, Tsinghua University (2025). RDT-2, an autoregressive vision-language-action model built on Qwen2.5-VL with a residual vector-quantized action tokenizer, trained on large-scale UMI data. github.com/thu-ml/RDT2.
21. Gemini Robotics Team, Google DeepMind (2025). Gemini Robotics: Bringing AI into the Physical World. arXiv:2503.20020.
22. NVIDIA (2025). GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv:2503.14734.
23. Luo, J., Xu, C., Wu, J., and Levine, S. (2024). Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning (HIL-SERL). github.com/rail-berkeley/hil-serl.
24. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017). Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS. arXiv:1703.06907.
25. Peng, X., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018). Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. ICRA. arXiv:1710.06537.
26. Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., and Hutter, M. (2020). Learning Quadrupedal Locomotion over Challenging Terrain. Science Robotics 5(47). arXiv:2010.11251.
27. Miki, T., Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., and Hutter, M. (2022). Learning Robust Perceptive Locomotion for Quadrupedal Robots in the Wild. Science Robotics 7(62). arXiv:2201.08117.
28. Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2020). Dream to Control: Learning Behaviors by Latent Imagination (Dreamer). ICLR. arXiv:1912.01603.
29. Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. (2023). Mastering Diverse Domains through World Models (DreamerV3). arXiv:2301.04104.
30. Walke, H., Black, K., Lee, A., et al. (2023). BridgeData V2: A Dataset for Robot Learning at Scale. CoRL. arXiv:2308.12952.
31. Khazatsky, A., Pertsch, K., Nair, S., et al. (2024). DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset. RSS. arXiv:2403.12945.
32. Wilson, E. B. (1927). Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 22(158), 209-212. DOI.
33. Clopper, C. and Pearson, E. (1934). The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial. Biometrika 26(4), 404-413. DOI.
34. Todorov, E., Erez, T., and Tassa, Y. (2012). MuJoCo: A Physics Engine for Model-Based Control. IROS. IEEE.