Interactive robotics: human models, shared autonomy, and learning from people

Why this subject matters now

The robots that left the cage in the last five years, warehouse mobile bases weaving between pickers, driver-assist stacks negotiating merges, assistive arms feeding a person with quadriplegia, surgical and rehabilitation devices that share control with a clinician, all share a property the classical motion-planning stack never had to confront: the environment contains an agent with goals of its own that reacts to the robot. A person is not a wall. A wall does not speed up because the robot slowed down, does not change lanes because it read the robot's blinker, and does not have a reward function the robot could infer and then help optimize. Treating people as obstacles produces two characteristic failures that the field spent a decade naming: the frozen robot problem, where a mobile base in a crowd concludes that every path is blocked and simply stops, and overcautious yielding, where an autonomous car at a four-way stop waits forever because it never models that a small nudge forward will make the human driver yield.

The intellectual center of the field is a single inversion. Forward reinforcement learning takes a reward and produces behavior; interactive robotics runs the arrow backward, taking observed human behavior and inferring the reward, the plan, or the next motion that produced it, then feeding that inference into the robot's own plan. Inverse reinforcement learning, trajectory prediction, and preference learning are three faces of that one inversion, and the maximum-entropy formulation ties them together: they are all maximum-likelihood estimation under a Boltzmann model that says a person picks behavior with probability exponential in its return. That same Boltzmann-over-returns model is, with the reward replaced by a learned scalar and the comparison made pairwise, the Bradley-Terry likelihood that trains reward models for large language models. The reader who understands maximum-entropy IRL on a gridworld already understands the objective behind reinforcement learning from human feedback; the difference is scale and function class, not principle.

Fluency here is now expected of anyone building embodied or aligned systems. An interviewer will assume one can explain why the IRL problem is ill-posed and what maximum entropy adds to make it well-posed, why the maximum-entropy gradient is exactly the difference between the expert's and the model's feature expectations, why a strong pedestrian predictor is often just a constant-velocity extrapolation with the right coordinate frame, why legibility and predictability of robot motion are not the same objective and can conflict, how shared autonomy blends a human's joystick with an autonomous policy without ever committing to a single guessed goal, and which query to a human most reduces uncertainty about their reward. These are derivations, not vocabulary, and the rest of the page is those derivations.

Core theory

Human motion prediction: the constant-velocity baseline

Prediction is the substrate everything else sits on: a robot cannot plan around a person without a distribution over where that person will be. Start with the simplest possible model and take it seriously, because it is a stronger baseline than intuition suggests. Let a person's planar position be \( x_t \in \R^2 \) observed at discrete times. The constant-velocity (CV) model estimates velocity from the last two observations and extrapolates linearly:

A slightly better estimator averages velocity over a window of \( m \) steps to damp observation noise, \( \hat{v}_t = \frac{1}{m}\sum_{i=1}^{m}(x_{t-i+1}-x_{t-i})/\Delta t = (x_t - x_{t-m})/(m\,\Delta t) \), which telescopes to a finite difference over the window. The reason to respect this baseline is empirical: Schöller, Aravantinos, Lay, and Knoll (2020) showed that on the standard pedestrian benchmarks a plain constant-velocity model, evaluated in the pedestrian's own heading frame, matches or beats a large fraction of the published deep sequence models of its era. The lesson is not that learning is useless; it is that most of a person's short-horizon motion is inertia, and a learned model earns its keep only on the residual: intent changes, interactions, and multimodality. Any predictor that cannot beat CV on a benchmark is not learning motion, it is learning the benchmark's average speed.

The CV model is a point predictor; a planner needs uncertainty. Promote it to a Gaussian by propagating a constant-velocity Kalman model with state \( s_t = (x_t, v_t) \), linear dynamics \( s_{t+1} = A s_t + w \) with \( A = \begin{pmatrix} I & \Delta t\, I \\ 0 & I \end{pmatrix} \) and process noise \( w \sim \N(0, Q) \). The \( k \)-step predictive covariance grows as \( \Sigma_{t+k} = A^k \Sigma_t (A^k)^\top + \sum_{j=0}^{k-1} A^j Q (A^j)^\top \), and the positional variance grows super-linearly in \( k \) because velocity uncertainty integrates into position. That growth is the honest statement of how far ahead a purely inertial model can be trusted, and it is why every serious planner replans far more often than its horizon would suggest.

Learned trajectory prediction and social interaction

The residual that CV leaves on the table is dominated by two effects: people have goals that bend their path, and people avoid each other. The social force model of Helbing and Molnár (1995) is the classical hand-built account of the second effect. It writes each pedestrian's acceleration as a sum of a term pulling toward a desired velocity and repulsive terms from other pedestrians and obstacles,

with \( U_{ij} \) a monotincreasing repulsive potential, typically exponential in distance. The model is a differential equation with interpretable parameters, and it reproduces emergent lane formation and bottleneck oscillation. Its weakness is that the potentials are hand-tuned and unimodal: it cannot represent that a person might go left or right around an obstacle with a bimodal distribution.

Learned predictors replace the hand-built potential with a function class fit to data. Alahi, Goel, Ramanathan, Robicquet, Fei-Fei, and Savarese (2016) introduced the Social LSTM, which runs a recurrent network per pedestrian and couples them through a social pooling layer that aggregates the hidden states of neighbors inside a spatial grid, so each person's next-step distribution is conditioned on the latent state of those nearby. The output is a bivariate Gaussian per step, trained by maximum likelihood. Trajectron++ (Salzmann, Ivanovic, Chen, and Pavone, 2020) is the modern representative of this line: a conditional variational autoencoder over a spatiotemporal graph, with a discrete latent that gives an explicit multimodal distribution over futures, and a decoder that can enforce dynamic feasibility by predicting controls rather than positions. The through-line from social forces to Trajectron++ is a steady move from a fixed potential to a learned, multimodal, interaction-aware conditional density, at the cost of interpretability and of the data required to fit it.

Why prediction and planning are coupled

The subtle and load-bearing point is that prediction is not an upstream module the planner consumes; the two are entangled, because the robot's plan changes the human's behavior, which changes the prediction. Trautman and Krause (2010) named the failure that follows from ignoring this the frozen-robot problem: a mobile base that predicts pedestrians as if the robot were not there sees the free space shrink to nothing in a dense crowd and stops, even though a human pilot would simply move, because the crowd would part. The prediction was self-defeating: it assumed non-interaction, and non-interaction is exactly the regime where the crowd does not cooperate.

Formally, if \( \xi_R \) is the robot's trajectory and \( \xi_H \) the human's, a non-interactive predictor models \( p(\xi_H) \) and the planner picks \( \xi_R \) to avoid the high-probability region of that fixed density. A coupled model instead reasons about the joint \( p(\xi_H, \xi_R) \), or equivalently a conditional \( p(\xi_H \mid \xi_R) \) that lets the robot's choice reshape the human's predicted distribution. The moment the prediction depends on the plan, prediction and planning become one optimization, and the clean pipeline picture is wrong. The game-theoretic and IRL formulations below are two ways to write that coupled problem down; the entire value of a human model is that it makes \( p(\xi_H \mid \xi_R) \) something the robot can compute rather than fear.

Interaction as a dynamic game

The cleanest formalization of coupling treats the human and the robot as two players in a dynamic game. Let a shared state \( x \) evolve under both agents' controls, \( x_{t+1} = f(x_t, u_t^R, u_t^H) \), and let each agent have its own cumulative cost over a horizon, \( J^R(u^R, u^H) \) and \( J^H(u^R, u^H) \), each depending on both control sequences because the state they share is driven by both. The autonomous-driving merge is the canonical instance: \( x \) stacks the two cars' positions and speeds, \( J^R \) trades progress against collision and comfort, and \( J^H \) is the human driver's analogous cost, which the robot does not know exactly and must model.

A Nash equilibrium is a pair \( (u^{R\star}, u^{H\star}) \) at which neither agent can lower its own cost by unilaterally changing its controls:

Nash treats the players as simultaneous and symmetric: each best-responds to the other's equilibrium strategy. A Stackelberg equilibrium instead assigns roles, a leader who commits first and a follower who observes the commitment and best-responds. If the robot is the leader, it solves a bilevel problem: it optimizes over its own controls while substituting in the human's best response to whatever it does,

The distinction is not academic. Sadigh, Sastry, Seshia, and Dragan (2016) built the autonomous-driving formulation around exactly the Stackelberg model with the robot as leader, and the payoff is precisely the ability to use the robot's influence on the human: because \( \mathrm{BR}_H(u^R) \) is inside the robot's objective, the planner discovers actions like inching forward at an intersection or beginning a lane change, actions whose value is not their direct progress but the human reaction they provoke. A robot that yields passively is running a degenerate game in which it assumes its own actions have no effect on the human; a robot that negotiates is solving the bilevel problem. The nesting of equilibria in a multi-agent stochastic game, correlated equilibria, and the non-stationarity that breaks independent learners are developed in advanced reinforcement learning; here the point is the leader-follower coupling and the influence it buys.

Inverse reinforcement learning: the ill-posed problem

The game formulation needs \( J^H \), the human's cost, and in general the robot does not have it. Inverse reinforcement learning is the problem of recovering it from behavior. Given an MDP without a reward, and demonstrations assumed to be (near) optimal under some unknown reward, infer a reward that explains them. Ng and Russell (2000) posed the problem and immediately exposed its central difficulty: it is badly ill-posed. Any policy is optimal for infinitely many rewards, including the degenerate reward that is constant everywhere, under which every policy ties. Their characterization: a policy \( \pi \) is optimal for reward \( r \) if and only if, writing the reward as a vector over states,

where \( P_\pi \) is the transition matrix under \( \pi \) and \( P_a \) the matrix for deviating to \( a \) then following \( \pi \). This is a set of linear inequalities in \( r \); it has a whole polytope of solutions, and \( r = 0 \) sits inside it. Ng and Russell patched the degeneracy with heuristics that prefer rewards making the observed policy optimal by the largest margin and with the smallest reward magnitude. Abbeel and Ng (2004) shifted the goal from recovering the reward to matching behavior: if the reward is linear in features, \( r(s) = w^\top \phi(s) \), then two policies with equal expected discounted feature counts \( \mu(\pi) = \E[\sum_t \gamma^t \phi(s_t)] \) achieve equal value for every such reward, so it suffices to find a policy whose feature expectations match the expert's. This is apprenticeship learning, and it makes the reward's non-identifiability irrelevant to the goal of imitation. But it still leaves the choice among behavior-matching policies underdetermined. Maximum entropy resolves that.

Maximum-entropy IRL: deriving the partition function and the gradient

Ziebart, Maas, Bagnell, and Dey (2008) removed the remaining ambiguity with a single principled choice: among all distributions over trajectories that reproduce the expert's feature expectations, pick the one of maximum entropy, which commits to nothing beyond the constraints. Let a trajectory be \( \tau = (s_0, a_0, s_1, \dots) \) with feature count \( \phi(\tau) = \sum_t \phi(s_t) \). We seek a distribution \( p(\tau) \) solving

where \( \tilde{\phi} \) is the empirical feature expectation of the demonstrations. Form the Lagrangian with multipliers \( w \) (a vector, one per feature) for the moment constraint and \( \lambda \) for normalization:

Differentiate with respect to \( p(\tau) \) and set to zero: \( -\log p(\tau) - 1 + w^\top \phi(\tau) + \lambda = 0 \), so \( p(\tau) = \exp\!\big(w^\top\phi(\tau) + \lambda - 1\big) \). Absorbing the constants into normalization gives the maximum-entropy trajectory distribution

This is the crux: demonstrations are modeled as drawn with probability exponential in their return \( w^\top\phi(\tau) \), so better trajectories are exponentially more likely but suboptimal ones are not forbidden. The normalizer \( Z(w) \), the partition function, sums over all trajectories and is where every computational difficulty lives. Now fit \( w \) by maximum likelihood on the demonstrations \( \{\tau_i\}_{i=1}^N \):

Differentiate. The first term gives \( \tilde\phi \). For the log-partition, \( \nabla_w \log Z = \frac{1}{Z}\sum_\tau \nabla_w e^{w^\top\phi(\tau)} = \sum_\tau \frac{e^{w^\top\phi(\tau)}}{Z}\,\phi(\tau) = \sum_\tau p(\tau\mid w)\,\phi(\tau) = \E_{p(\tau\mid w)}[\phi(\tau)] \). Therefore

The gradient is the difference between the expert's feature expectations and the current model's. At the maximum it vanishes, so the fitted reward is exactly the one under which the induced maximum-entropy policy matches the demonstrated feature counts. This is the same feature-matching condition Abbeel and Ng identified, now derived as the stationarity condition of a well-posed maximum-likelihood problem rather than imposed as a heuristic. The model's feature expectation \( \E_{p(\tau\mid w)}[\phi] \) is computed by a forward pass: a soft (log-sum-exp) value iteration produces the maximum-entropy policy, and a forward propagation of state-visitation probabilities under that policy yields the expected feature counts. That is precisely the algorithm implemented and run below, recovering the ground-truth terrain weights of a gridworld from demonstrations up to the one degree of freedom, an additive constant, that the reward genuinely cannot be identified beyond.

Bayesian IRL

Maximum-entropy IRL returns a point estimate of \( w \). When the robot must act under uncertainty about the reward, and every interactive setting requires this, it wants a posterior, not a point. Ramachandran and Amir (2007) gave the Bayesian formulation. Put a prior \( p(w) \) on the reward weights and a likelihood that scores how well each demonstrated state-action pair is explained by the optimal Q-values under \( w \):

a Boltzmann policy with rationality coefficient \( \beta \). The posterior over rewards is \( p(w \mid \mathcal{D}) \propto p(w)\prod_{(s,a)\in\mathcal{D}} p((s,a)\mid w) \). It has no closed form because \( Q^\star_w \) depends on \( w \) through a planning problem, so Ramachandran and Amir sample it with a Markov chain over reward vectors, re-solving the MDP at each proposed \( w \). The payoff of carrying the whole posterior rather than a point is that the robot can be risk-aware and can compute the value of asking a clarifying question, which is exactly what the active-query section below needs. Notice the same Boltzmann-rationality likelihood recurs: maximum-entropy IRL is the trajectory-level version, Bayesian IRL the state-action version, and preference learning the pairwise version. All three are one model of a noisily rational human.

Preference-based reward learning and the Bradley-Terry model

Demonstrations are expensive and often infeasible: a person cannot demonstrate a helicopter aerobatic maneuver or a good summary, but they can compare two of the robot's attempts and say which is better. Comparisons are cheap, low-variance, and require no expertise in executing the task. The statistical model for turning comparisons into a scalar reward predates robotics by seventy years. Bradley and Terry (1952) modeled paired comparisons by assigning each item a positive strength and setting the probability that item \( i \) beats item \( j \) to \( s_i / (s_i + s_j) \). Write the strength as the exponential of a score, \( s_i = e^{r_i} \), and the model becomes a logistic in the score difference:

Now let the items be trajectory segments \( \tau \) and the score be their return under a reward model, \( r(\tau) = \sum_{t} \hat{r}_\theta(s_t, a_t) \). A human comparison \( \tau_A \succ \tau_B \) has likelihood \( \sigma\!\big(r(\tau_A) - r(\tau_B)\big) \), and the reward parameters \( \theta \) are fit by minimizing the negative log-likelihood over a dataset of comparisons:

with \( y = 1 \) when \( A \) is preferred. This is the objective in Christiano, Leike, Brown, Martic, Legg, and Amodei (2017), deep reinforcement learning from human preferences, which trained MuJoCo and Atari agents, including a backflip no one could write a reward for, from roughly a thousand human comparisons, and it is the same objective in Sadigh, Dragan, Sastry, and Seshia (2017) for learning a driver's reward from pairwise preferences over trajectories. It is identical to the reward-model objective that trains large language models: a preference dataset, a scalar head, a Bradley-Terry logistic loss. The derivative with respect to the reward of the preferred segment is the clean logistic-regression residual \( \partial\mathcal{L}/\partial r(\tau_A) = -(y - \hat p) \) where \( \hat p = \sigma(r(\tau_A)-r(\tau_B)) \), which is why the same code trains a gridworld reward and a language-model reward model. The reinforcement-learning half, using the fitted reward to improve a policy with PPO or a direct preference objective, is developed in deep reinforcement learning; the reward-inference half is here.

Learning from demonstration versus learning from correction

There are two channels by which a person can shape a robot's behavior online, and they have different statistics. Learning from demonstration gives the robot whole trajectories. The simplest use is behavioral cloning: treat state-to-action as supervised learning and fit \( \pi_\theta(a\mid s) \) by maximum likelihood on the demonstrations. Its fatal weakness is covariate shift. The learner trains on the expert's state distribution but acts on its own; a small error moves it to a state the expert never visited, where its next prediction is worse, and the errors compound. If the per-step probability of a mistake is \( \epsilon \), the expected number of mistakes over a horizon \( T \) grows like \( \epsilon T^2 \) rather than \( \epsilon T \), the quadratic coming from the fact that one mistake changes the entire downstream distribution. IRL avoids this by recovering the reward and re-planning, so the robot knows what to do in states the expert never showed, at the cost of an inner planning loop.

Learning from correction gives the robot not a trajectory but a nudge: while the robot executes, the person pushes the arm, taps a joystick, or overrides one dimension of the motion. A physical push is a gradient, not a target. Bajcsy, Losey, O'Malley, and Dragan (2017) modeled such a physical interaction as evidence about the reward: the human applies force \( u_H \) because the current trajectory is not the one they want, so the observed force is informative about \( \nabla_\theta \) of their objective, and the robot updates its reward estimate in the direction the person pushed rather than merely complying and forgetting. The distinction matters in deployment: a correction is a single, cheap, in-context signal, and a robot that treats it as reward evidence generalizes the fix, whereas one that treats it as a one-off override repeats the mistake on the next trial.

Shared autonomy and arbitration under goal uncertainty

Assistive teleoperation is the setting where a person and an autonomous policy control the same system at the same time, and the robot does not know the person's goal. An assistive arm sees a joystick command and must decide how much of its own autonomous motion to blend in. The naive scheme is a fixed linear blend, \( u = (1-\alpha)\,u_H + \alpha\,u_R \), with \( \alpha \) a trust knob. It has a well-known failure: if the robot commits to a single guessed goal and the guess is wrong, the assistance actively fights the user. Javdani, Srinivasa, and Bagnell (2015, and the extended 2018 journal version) reformulated arbitration to never commit to a single goal. They model the goal \( g \) as hidden, maintain a belief \( b(g) \) updated from the user's inputs under a Boltzmann model of the user as a noisily optimal controller toward their true goal, and choose assistance that is optimal in expectation over the belief. This is a POMDP: the state includes the unknown goal, the observations are the user's inputs, and the objective is the user's own cost marginalized over the goal belief.

The exact POMDP is intractable, so they use hindsight optimization, an upper bound on the value that assumes the goal will be revealed after one step. Writing \( Q_g \) for the cost-to-go assuming goal \( g \) is known, the assistance action minimizes the belief-weighted cost

which produces the graceful behavior that when the belief is spread over several plausible goals the robot provides only the assistance those goals agree on, for instance moving toward the centroid of the candidate objects, and sharpens its help toward a specific goal only as the user's inputs concentrate the belief. The belief-space machinery, the POMDP value function, its piecewise-linear-and-convex structure, and QMDP and other approximations, is derived in decision making under uncertainty; shared autonomy is one of its most successful applications because the reward is the user's own and the observations are exactly the user's commands.

Legibility versus predictability

When a robot moves near a person, the motion communicates. Dragan, Lee, and Srinivasa (2013) drew a distinction that is easy to conflate and important to keep separate. Predictability asks: given that the observer knows the robot's goal, does the motion match what they expect, that is, is it the efficient, unsurprising path to that goal? Legibility asks the inverse: given that the observer is watching the motion and does not yet know the goal, does the motion let them infer the goal quickly and confidently? These are different inferences running in opposite directions. Predictability is \( P(\text{trajectory}\mid \text{goal}) \); legibility is \( P(\text{goal}\mid\text{trajectory}) \).

Formalize both with a Boltzmann model of a rational observer. Let \( C(\xi) \) be the cost of a trajectory and \( C(S \to G) \) the cost of the optimal path from start \( S \) to goal \( G \). An observer who believes the mover is approximately optimal, on seeing a partial trajectory from \( S \) to the current point \( x_t \), infers the goal by

the numerator scoring how cheap it still is to reach \( G \) through \( x_t \) against the cost of going straight to \( G \). The legibility of a trajectory to the true goal \( G_{\text{true}} \) integrates this posterior over the motion, weighted to reward confidence early:

Predictability instead rewards efficiency: a predictable trajectory to a known goal is one whose cost is close to optimal, scored for instance by \( \exp\!\big(-(C(\xi) - C(S\to G))\big) \), which is one exactly when the path is optimal and decays as the path wastes motion. The two objectives conflict. The optimal, most predictable path to a goal is often ambiguous early, because heading straight toward a point between two candidate goals looks equally consistent with either. A legible path deliberately exaggerates, curving away from the distractor goal early to disambiguate, which makes it slightly longer and therefore less predictable. The worked problem below computes both scores on a two-goal reaching example and shows the exaggerated path scoring higher on legibility (0.799 versus 0.737) precisely by paying a predictability cost (0.892 versus 1.0). A robot that only optimizes efficiency is predictable but can be illegible, and a person watching it will hesitate; deliberately trading a little efficiency for legibility is how a robot's motion becomes trustworthy to a bystander.

Active reward learning: choosing the query that maximizes information gain

If the robot maintains a belief over the human's reward, it should not ask random questions; it should ask the one whose answer it expects to learn the most from. Frame a query as a pair of trajectories \( (\xi_A, \xi_B) \) shown to the human, who answers with a Bradley-Terry preference. Before asking, the robot has a belief \( b(w) \) over reward weights with entropy \( H[b] \). A query with answer \( Y \in \{A, B\} \) has expected information gain equal to the mutual information between the answer and the reward,

where \( P(Y=A) = \sum_w b(w)\,\sigma\!\big(w^\top(\phi_A - \phi_B)\big) \) marginalizes the Bradley-Terry answer over the belief, and the posterior after an answer is \( b(w\mid A) \propto b(w)\,\sigma(w^\top(\phi_A-\phi_B)) \). Selecting the query that maximizes this is the acquisition strategy of Sadigh, Dragan, Sastry, and Seshia (2017) and Bıyık and Sadigh (2018). The intuition the worked problem confirms numerically: the best query is the one whose answer the robot is most uncertain about and that splits the hypotheses, a query that all reward hypotheses answer the same way carries almost no information (0.0045 bits in the example), while a query the hypotheses disagree on and whose answer is near a coin flip carries the most (0.149 bits out of a 1.585-bit prior). Choosing questions this way is why a robot can learn a usable reward from a handful of comparisons rather than thousands.

Safety under uncertainty about human intent

Prediction and reward inference are never certain, so a deployed interactive robot needs a safety layer that does not depend on the human model being right. Two high-level tools dominate. Hamilton-Jacobi reachability computes the backward-reachable set of states from which a disturbance, here the worst-case human action within a modeled bound, can force a collision, and it certifies safety by keeping the robot out of that set; Fisac, Bajcsy, Herbert, Fridovich-Keil, Wang, Tomlin, and Dragan (2018) combined a learned human predictor with a reachability backstop, so the robot plans efficiently against the predictor most of the time and falls back to the worst-case guarantee exactly when the human behaves outside the model's confidence. Control barrier functions (Ames, Grizzle, and Tabuada, 2014, and Ames, Coogan, Egerstedt, Notomista, Sreenath, and Tabuada, 2019) give a cheaper, per-step certificate: a function \( h(x) \) whose non-negative super-level set is the safe set, enforced by the constraint

with \( \alpha \) a class-\(\mathcal{K}\) function, which as a linear inequality in \( u \) can be added to a quadratic program that minimally modifies a nominal control to stay safe. Under intent uncertainty the barrier is written against a set of possible human trajectories, so the guarantee holds for any human behavior inside the modeled set. The derivations of both, the forward-invariance proof for control barrier functions and the level-set dynamics for reachability, live in robot kinematics and control; the interactive point is that a safety certificate lets the human model be optimistic and useful in the common case while a worst-case bound catches the tail, which is the only architecture that is simultaneously efficient and safe.

Worked problems

Implementation

The first block is the complete maximum-entropy IRL loop on the gridworld, self-contained in NumPy: it builds the deterministic grid, runs soft (log-sum-exp) value iteration to get the maximum-entropy policy, propagates state visitations forward to get the model's expected feature counts, and ascends the feature-matching gradient. It is the exact program used to produce the recovered weights quoted above. The soft backup uses \( V(s) = \log\sum_a \exp Q(s,a) \) in place of the hard max, which is the log-partition of the action values and the reason the induced policy is the Boltzmann policy the maximum-entropy model assumes.

import numpy as np

# 4x4 gridworld. States 0..15, actions up/down/left/right/stay.
H = W = 4; S = H * W; A = 5; GOAL = 15; gamma = 0.9
terrain = np.array([0,0,1,1, 0,2,2,1, 0,2,1,1, 2,2,2,0])  # 0 grass,1 mud,2 road
F = np.zeros((S, 3))                      # (S, n_features) one-hot terrain
for s in range(S): F[s, terrain[s]] = 1.0
w_true = np.array([-1.0, -3.0, -0.5])     # ground-truth per-step costs

def step(s, a):                           # deterministic transition, walls reflect
    r, c = divmod(s, W)
    dr, dc = [(-1,0),(1,0),(0,-1),(0,1),(0,0)][a]
    r2, c2 = r+dr, c+dc
    return r2*W + c2 if 0 <= r2 < H and 0 <= c2 < W else s
P = np.array([[step(s,a) for a in range(A)] for s in range(S)])  # (S, A) -> s'

def soft_value_iteration(r, iters=500):   # r: (S,) per-state reward
    V = np.zeros(S)
    for _ in range(iters):
        Q = r[P] + gamma * V[P]           # (S, A): reward of entered state + value
        m = Q.max(1)
        Vn = np.log(np.exp(Q - m[:,None]).sum(1)) + m   # soft (log-sum-exp) backup
        Vn[GOAL] = 0.0                    # terminal goal
        if np.abs(Vn - V).max() < 1e-12: V = Vn; break
        V = Vn
    Q = r[P] + gamma * V[P]
    pol = np.exp(Q - V[:,None]); pol /= pol.sum(1, keepdims=True)  # Boltzmann policy
    return pol

def visitation(pol, T=60):                # expected state-visit counts, goal absorbs
    mu = np.zeros(S); front = np.zeros(S); front[0] = 1.0
    for _ in range(T):
        mu += front
        nxt = np.zeros(S)
        for s in range(S):
            if s == GOAL: continue        # terminated, no outflow
            for a in range(A): nxt[P[s,a]] += front[s] * pol[s,a]
        front = nxt
    return mu

f_expert = F.T @ visitation(soft_value_iteration(F @ w_true))   # expert feature counts
w = np.zeros(3); lr = 0.03
for it in range(6000):
    D = visitation(soft_value_iteration(F @ w))
    grad = f_expert - F.T @ D             # feature matching: expert - model
    grad -= grad.mean()                   # project out additive-constant gauge
    w += lr * grad / D.sum()
print("recovered (centered on road):", np.round(w - w[2], 3))  # ~ [-0.5, -2.17, 0]
print("true      (centered on road):", w_true - w_true[2])     #   [-0.5, -2.5,  0]

The second block is the Bradley-Terry preference model as plain NumPy: the likelihood of a comparison, the dataset negative log-likelihood, and the logistic residual gradient, matching Problem 3. It is deliberately identical in form to the reward-model loss below, to make the point that the preference objective is function-class-agnostic.

import numpy as np
sigmoid = lambda z: 1.0 / (1.0 + np.exp(-z))

def bt_prob(rA, rB):                 # P(A preferred over B)
    return sigmoid(rA - rB)

# dataset: (return_first, return_second, label) with label 1 == first preferred
pairs = [(3.2,1.5,1), (0.8,2.1,0), (4.0,3.9,1), (1.0,1.0,1), (2.5,0.3,1)]
nll = 0.0
for a, b, y in pairs:
    p = bt_prob(a, b)
    nll += -(y*np.log(p) + (1-y)*np.log(1-p))
print("P(A>B) for (3.2,1.5):", round(bt_prob(3.2,1.5), 4))   # 0.8455
print("total preference NLL:", round(nll, 4))                # 1.8514

# gradient of the loss w.r.t. the preferred segment's reward is the logistic residual
a, b, y = pairs[0]; p = bt_prob(a, b)
print("dNLL/d r(A) = -(y - p):", round(-(y - p), 4))         # -0.1545

The third block trains an actual reward model on preferences, PyTorch and JAX side by side. Each trajectory segment is summarized by a feature vector; the reward is a small network mapping features to a scalar; the loss is the Bradley-Terry logistic on segment-return differences. This is the identical objective used to train reward models for language systems and for Christiano et al.'s preference-based control agents; only the encoder changes. The two implementations are numerically the same computation: forward both segments through the reward network, take the difference, and apply a binary cross-entropy on the preference label.

import torch
import torch.nn as nn
import torch.nn.functional as Fn

# Reward model: feature vector -> scalar reward. Segment return = sum over steps;
# here each segment is one feature vector phi in R^d (a summary), for clarity.
class RewardModel(nn.Module):
    def __init__(self, d, hidden=32):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(d, hidden), nn.ReLU(), nn.Linear(hidden, 1))
    def forward(self, phi):               # phi: (batch, d)
        return self.net(phi).squeeze(-1)  # (batch,)

def bt_loss(model, phi_A, phi_B, label):  # label: 1.0 if A preferred, else 0.0
    rA, rB = model(phi_A), model(phi_B)   # (batch,), (batch,)
    logits = rA - rB                      # Bradley-Terry: P(A>B) = sigmoid(rA - rB)
    return Fn.binary_cross_entropy_with_logits(logits, label)

torch.manual_seed(0)
d = 4
model = RewardModel(d)
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
# toy data: A is preferred exactly when its first feature is larger
phi_A = torch.randn(256, d); phi_B = torch.randn(256, d)
label = (phi_A[:, 0] > phi_B[:, 0]).float()
for step in range(400):
    opt.zero_grad()
    loss = bt_loss(model, phi_A, phi_B, label)
    loss.backward()
    opt.step()
with torch.no_grad():
    acc = ((model(phi_A) > model(phi_B)).float() == label).float().mean()
print("final BT loss:", round(loss.item(), 4), " train acc:", round(acc.item(), 4))

import jax, jax.numpy as jnp
import optax
from jax import random

def init_params(key, d, hidden=32):
    k1, k2 = random.split(key)
    return {
        "W1": random.normal(k1, (d, hidden)) * 0.1, "b1": jnp.zeros(hidden),
        "W2": random.normal(k2, (hidden, 1)) * 0.1, "b2": jnp.zeros(1),
    }

def reward(params, phi):                  # phi: (batch, d) -> (batch,)
    h = jax.nn.relu(phi @ params["W1"] + params["b1"])
    return (h @ params["W2"] + params["b2"]).squeeze(-1)

def bt_loss(params, phi_A, phi_B, label): # Bradley-Terry logistic loss
    logits = reward(params, phi_A) - reward(params, phi_B)
    # binary cross-entropy with logits, numerically stable
    return jnp.mean(optax.sigmoid_binary_cross_entropy(logits, label))

key = random.PRNGKey(0)
d = 4
k_a, k_b, k_p = random.split(key, 3)
phi_A = random.normal(k_a, (256, d)); phi_B = random.normal(k_b, (256, d))
label = (phi_A[:, 0] > phi_B[:, 0]).astype(jnp.float32)
params = init_params(k_p, d)
opt = optax.adam(1e-2); opt_state = opt.init(params)

@jax.jit
def update(params, opt_state):
    loss, grads = jax.value_and_grad(bt_loss)(params, phi_A, phi_B, label)
    updates, opt_state = opt.update(grads, opt_state)
    return optax.apply_updates(params, updates), opt_state, loss

for step in range(400):
    params, opt_state, loss = update(params, opt_state)
acc = jnp.mean(((reward(params, phi_A) > reward(params, phi_B)) == label))
print("final BT loss:", round(float(loss), 4), " train acc:", round(float(acc), 4))

How it is done in practice

The gap between the gridworld and a deployed interactive system is mostly about where the intractable expectation goes. In maximum-entropy IRL the partition function \( Z(w) \) is a sum over all trajectories, tractable by dynamic programming only when the state space is small and the dynamics known. For high-dimensional continuous control neither holds, and the field's answer is sampling-based IRL. Finn, Levine, and Abbeel (2016) with guided cost learning, and Ho and Ermon (2016) with generative adversarial imitation learning, both replace the intractable partition function with samples from the current policy and show that maximum-entropy IRL is, in this form, adversarial: the reward is a discriminator separating expert from policy rollouts, and the policy is a generator trained to fool it. This is why modern imitation-learning libraries ship GAIL and adversarial IRL rather than tabular maximum-entropy IRL; the derivation above is the tabular special case whose gradient the sampled version estimates.

Preference-based reward learning at scale is the same Bradley-Terry loss with a transformer as the reward model and a very different data pipeline. The reward model's forward and backward passes are ordinary transformer matrix multiplies, bound by the same throughput any transformer sees: on the NVIDIA H100 80GB HBM3 in this repository, dense matrix multiplies measure 744.6 bf16 TFLOP/s at \( n = 4096 \) and about 453 bf16 TFLOP/s at \( n = 2048 \), so a reward model is not a special kernel, it is a classifier head on a standard backbone. The engineering difficulty is entirely in the human data: comparisons must be de-noised (annotators disagree, and the Bradley-Terry model's rationality coefficient must be fit or the loss miscalibrates), the reward model is prone to overoptimization (a policy trained against it exploits its errors, the reward-hacking problem), and the queries must be chosen to cover the state distribution the policy will actually reach. The active-query information-gain criterion derived above is the principled answer to the last problem, and libraries such as the APReL active preference-learning package implement exactly the mutual-information acquisition function on the reward posterior.

Trajectory prediction in a production autonomy stack runs at tens of hertz over dozens of agents, so the expensive multimodal models are pruned hard: a common pattern is a cheap constant-velocity or kinematic fallback for distant, non-interacting agents and a learned interaction-aware model reserved for the few agents whose predicted futures actually constrain the plan. The coupling between prediction and planning is handled by replanning fast rather than predicting far: a horizon of a few seconds, replanned every hundred milliseconds, with a reachability or barrier-function safety layer that does not trust the predictor in the tail. The recurring production lesson is the one the constant-velocity baseline teaches: spend model capacity only on the residual that inertia and geometry do not already explain, and put the safety guarantee somewhere that does not depend on the human model being right.

The current research frontier

Five threads are active. The first is scaling preference learning and questioning the Bradley-Terry assumption itself. Reinforcement learning from human feedback, whose reward-model objective is exactly the Bradley-Terry loss above, became the dominant alignment method for language models (Christiano et al. 2017 at OpenAI and DeepMind established the method; the InstructGPT line, Ouyang et al. 2022, scaled it), and the frontier now includes methods that skip the explicit reward model, direct preference optimization (Rafailov et al. 2023) folding the Bradley-Terry likelihood directly into a policy objective, and active debate over whether pairwise preferences are the right signal at all versus richer feedback. That policy side is developed in deep reinforcement learning; the reward-inference side is the subject here.

The second thread is reward learning that is robust to a wrong human model. Every method on this page assumes a model of the human, Boltzmann-rational, feature-linear, or a fixed network, and the human is none of these. Work from Berkeley (Hadfield-Menell, Russell, Abbeel, and Dragan's cooperative inverse reinforcement learning, 2016, and the off-switch analysis of Hadfield-Menell, Dragan, Abbeel, and Russell, 2017) reframes reward learning as a cooperative game in which the human is a second player who knows the reward and the robot is uncertain about it, which changes the robot's incentives, notably giving it a reason to preserve its own corrigibility. Parallel work studies what happens when the human's rationality model is misspecified and how to detect it online.

The third thread is prediction and planning as one model. The clean separation this page argued against is dissolving in practice: joint prediction-planning architectures, game-theoretic trajectory optimizers solved to local Nash or Stackelberg equilibria (the driving line from Sadigh's ILIAD and InterACT groups and from CMU and MIT), and large learned models that output the robot's plan and the humans' predicted responses together. Trajectron++ (Pavone's autonomous-systems lab) and its successors push the multimodal, dynamically feasible prediction side; the integration with a planner that reasons about its own influence is where the field is actively moving.

The fourth thread is safety with learned human models. Combining a data-driven predictor with a worst-case certificate, so the robot is efficient in the common case and provably safe in the tail, is the architecture of Fisac et al. (2018) and the control-barrier-function line from Ames and collaborators and from Sreenath's group; the open problems are quantifying the predictor's confidence honestly, so the switch to the worst-case bound fires at the right time, and keeping the guaranteed-safe set from being so conservative that it reintroduces the frozen-robot problem the whole enterprise set out to avoid.

The fifth thread is human feedback inside real-robot training loops. Berkeley's SERL and HIL-SERL systems (2024) put a person in the loop of a reinforcement learner running on physical hardware, with operator interventions serving both as safety stops and as on-policy corrections that enter the update, and they reach near-perfect success on contact-rich manipulation tasks within a few hours of interaction. The same period saw large language models enter the reward-specification loop, with NVIDIA's Eureka using a code-writing model to propose and refine reward functions against simulation feedback. That reframes reward design, the very problem inverse reinforcement learning set out to sidestep, as an automated search guided by execution results, with the human moved up a level to judging outcomes rather than writing the objective.

Open source to read

• HumanCompatibleAI/imitation is the reference implementation of imitation and inverse RL: behavioral cloning, DAgger, GAIL, adversarial IRL, and preference comparisons in one library. Open src/imitation/algorithms/preference_comparisons.py to see the Bradley-Terry reward-learning loop derived above, and algorithms/adversarial/ for the sampling-based maximum-entropy IRL as a discriminator.
• Stanford-ILIAD/APReL is an active preference-based reward-learning library: it implements the belief over reward weights, the Bradley-Terry query likelihood, and acquisition functions including the information-gain criterion from the active-query section. Start from its query-optimization module to see the mutual-information objective in code.
• StanfordASL/Trajectron-plus-plus is the multimodal, interaction- and dynamics-aware trajectory predictor. Read the model directory for the conditional-VAE encoder over the spatiotemporal graph and the dynamics-integrating decoder that predicts controls rather than positions.
• openai/lm-human-preferences is the original code accompanying preference-based fine-tuning of language models; the reward-model training file shows the identical Bradley-Terry objective at transformer scale, which makes the equivalence with the gridworld preference model concrete.
• Farama-Foundation/Gymnasium (the maintained successor to openai/gym) is the environment interface every method here plugs into; its wrapper API is how a learned reward is injected into a training loop, and the classic-control and MuJoCo environments are where preference-based agents are usually first validated.
• HybridRobotics/CBF collects control-barrier-function implementations from Sreenath's group; open the quadratic-program filter to see the barrier constraint \( \dot h \ge -\alpha(h) \) turned into a minimal modification of a nominal control, which is the safety layer the last theory section describes.

Common misconceptions

“Inverse reinforcement learning recovers the true reward.” It does not, and cannot in general. Ng and Russell showed the problem is ill-posed: many rewards, including the zero reward, make the same behavior optimal. Maximum entropy makes the maximum-likelihood problem well-posed, but the recovered reward is still identifiable only up to gauge freedoms (an additive constant, potential-based shaping, and any feature the demonstrations never exercise), as the gridworld's underestimated mud weight shows. IRL recovers a reward that reproduces the behavior, not the reward in the demonstrator's head.

“A more expensive learned predictor always beats constant velocity.” On short horizons in the pedestrian's own frame, a plain constant-velocity model is competitive with, and sometimes better than, published deep predictors (Schöller et al. 2020). Most short-horizon motion is inertia; a learned model earns its cost only on the residual of intent changes and interactions, and a model that cannot beat constant velocity has learned the dataset's average speed, not motion.

“Legible and predictable motion are the same thing.” They are opposite inferences. Predictability is \( P(\text{trajectory}\mid\text{goal}) \), the efficiency of the path given a known goal; legibility is \( P(\text{goal}\mid\text{trajectory}) \), how fast an observer can infer the goal from the path. They can conflict: the most efficient path is often ambiguous early, and a legible path deliberately exaggerates at an efficiency cost, as the worked problem's 0.799-versus-0.737 legibility gap against a predictability drop from 1.0 to 0.892 makes precise.

“Shared autonomy means picking the user's goal and helping with it.” Committing to a single guessed goal is exactly the failure Javdani et al. avoid. The correct formulation keeps a belief over goals and provides assistance optimal in expectation over that belief, so when the goal is ambiguous the robot helps only in the directions all plausible goals agree on and sharpens as the user's inputs concentrate the belief. Guessing and being wrong makes the assistance fight the user.

“The robot should just predict the human and then plan around the prediction.” A prediction that assumes the robot has no effect is self-defeating in exactly the interactive regime that matters: it produces the frozen-robot problem in a crowd and endless yielding at an intersection. Prediction and planning are coupled because the robot's plan changes the human's behavior; the game-theoretic and conditional-prediction formulations exist precisely to make \( p(\xi_H\mid\xi_R) \) computable.

“Preference learning is a robotics-specific trick.” The Bradley-Terry model is from 1952 and the reward-model objective that trains large language models is line-for-line the same logistic loss on score differences used to learn a driver's reward from trajectory comparisons. The residual \( -(y-\hat p) \) is the identical gradient in all three. What changes across applications is the encoder and the data pipeline, not the objective.

Self-check

References

1. Sutton, R. and Barto, A. (2018). Reinforcement Learning: An Introduction, 2nd ed. MIT Press. book site.
2. Thrun, S., Burgard, W., and Fox, D. (2005). Probabilistic Robotics. MIT Press.
3. Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., and Peters, J. (2018). An algorithmic perspective on imitation learning. Foundations and Trends in Robotics. arXiv:1811.06711.
4. Ng, A. and Russell, S. (2000). Algorithms for inverse reinforcement learning. ICML. pdf.
5. Abbeel, P. and Ng, A. (2004). Apprenticeship learning via inverse reinforcement learning. ICML. pdf.
6. Ziebart, B., Maas, A., Bagnell, J. A., and Dey, A. (2008). Maximum entropy inverse reinforcement learning. AAAI. pdf.
7. Ziebart, B., Ratliff, N., Gallagher, G., Mertz, C., Peterson, K., Bagnell, J. A., Hebert, M., Dey, A., and Srinivasa, S. (2009). Planning-based prediction for pedestrians. IROS. pdf.
8. Ramachandran, D. and Amir, E. (2007). Bayesian inverse reinforcement learning. IJCAI. pdf.
9. Bradley, R. and Terry, M. (1952). Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika 39(3/4). doi:10.2307/2334029.
10. Christiano, P., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. NeurIPS. arXiv:1706.03741.
11. Sadigh, D., Sastry, S., Seshia, S., and Dragan, A. (2016). Planning for autonomous cars that leverage effects on human actions. RSS. pdf.
12. Sadigh, D., Dragan, A., Sastry, S., and Seshia, S. (2017). Active preference-based learning of reward functions. RSS. pdf.
13. Bıyık, E. and Sadigh, D. (2018). Batch active preference-based learning of reward functions. CoRL. arXiv:1810.04303.
14. Dragan, A., Lee, K., and Srinivasa, S. (2013). Legibility and predictability of robot motion. HRI. pdf.
15. Javdani, S., Srinivasa, S., and Bagnell, J. A. (2015). Shared autonomy via hindsight optimization. RSS. arXiv:1503.07619.
16. Javdani, S., Admoni, H., Pellegrinelli, S., Srinivasa, S., and Bagnell, J. A. (2018). Shared autonomy via hindsight optimization for teleoperation and teaming. IJRR. arXiv:1706.00155.
17. Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., and Savarese, S. (2016). Social LSTM: human trajectory prediction in crowded spaces. CVPR. pdf.
18. Salzmann, T., Ivanovic, B., Chen, P., and Pavone, M. (2020). Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. ECCV. arXiv:2001.03093.
19. Schöller, C., Aravantinos, V., Lay, F., and Knoll, A. (2020). What the constant velocity model can teach us about pedestrian motion prediction. IEEE RA-L. arXiv:1903.07933.
20. Helbing, D. and Molnár, P. (1995). Social force model for pedestrian dynamics. Physical Review E 51(5). arXiv:cond-mat/9805244.
21. Trautman, P. and Krause, A. (2010). Unfreezing the robot: navigation in dense, interacting crowds. IROS. pdf.
22. Bajcsy, A., Losey, D., O'Malley, M., and Dragan, A. (2017). Learning robot objectives from physical human interaction. CoRL. proceedings.
23. Hadfield-Menell, D., Russell, S., Abbeel, P., and Dragan, A. (2016). Cooperative inverse reinforcement learning. NeurIPS. arXiv:1606.03137.
24. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2017). The off-switch game. IJCAI. arXiv:1611.08219.
25. Ho, J. and Ermon, S. (2016). Generative adversarial imitation learning. NeurIPS. arXiv:1606.03476.
26. Finn, C., Levine, S., and Abbeel, P. (2016). Guided cost learning: deep inverse optimal control via policy optimization. ICML. arXiv:1603.00448.
27. Ames, A., Coogan, S., Egerstedt, M., Notomista, G., Sreenath, K., and Tabuada, P. (2019). Control barrier functions: theory and applications. ECC. arXiv:1903.11199.
28. Fisac, J., Bajcsy, A., Herbert, S., Fridovich-Keil, D., Wang, S., Tomlin, C., and Dragan, A. (2018). Probabilistically safe robot planning with confidence-based human predictions. RSS. arXiv:1806.00109.
29. Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS. arXiv:2203.02155.
30. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C., and Finn, C. (2023). Direct preference optimization: your language model is secretly a reward model. NeurIPS. arXiv:2305.18290.
31. Ma, Y. J., Liang, W., et al. (2024). Eureka: human-level reward design via coding large language models. ICLR. arXiv:2310.12931.
32. Luo, J., Xu, C., Wu, J., and Levine, S. (2024). Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning (HIL-SERL). github.com/rail-berkeley/hil-serl.