Deep reinforcement learning: policy gradients, off-policy control, and RL for language models

Why this subject matters now

Deep reinforcement learning has had two lives. The first ran from roughly 2013 to 2019. DQN played Atari from pixels (Mnih et al., Nature 2015), AlphaGo and AlphaZero mastered Go, chess, and shogi from self-play, and OpenAI Five and AlphaStar reached professional level in Dota 2 and StarCraft II. These results proved that value functions and policy gradients could be carried by deep networks, but they lived in simulators with cheap samples, attempts to move the recipes into robotics and products mostly stalled, and Henderson et al. (2018) documented that published results often failed to replicate across seeds. For a few years the field's honest summary was that deep RL worked impressively where experience was free and poorly almost everywhere else.

The second life began when the experience generator became a language model. Reinforcement learning from human feedback (Christiano et al. 2017, Ouyang et al. 2022, Bai et al. 2022) turned preference data into reward and made PPO the alignment workhorse behind every major chat assistant. Then reinforcement learning with verifiable rewards, the recipe behind OpenAI's o-series and DeepSeek-R1 (2025), showed that a policy-gradient loop over math and code problems with checkable answers produces models that reason measurably better, and that the improvement scales with the compute spent generating and grading attempts. Deep RL is now on the critical path of frontier model training at Anthropic, OpenAI, Google DeepMind, Meta, DeepSeek, Qwen, and Moonshot, and the practitioner bar has moved accordingly. Five years ago a strong applied engineer needed to know what a replay buffer was. Today they are expected to derive the policy gradient theorem, explain what PPO's clip does and does not guarantee, reproduce the DPO derivation, say why GRPO can drop the value network, and reason about why rollout throughput rather than gradient math dominates the cost of an RLHF run. That is the material this page covers.

Core theory

Markov decision processes, returns, and value functions

The formal object is a Markov decision process \( \mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma, \mu_0) \), a state space, an action space, a transition kernel \( P(s' \mid s, a) \), a reward function \( r(s,a) \), a discount \( \gamma \in [0, 1) \), and an initial-state distribution \( \mu_0 \). The Markov property is the load-bearing assumption. The future depends on the past only through the current state, so a policy \( \pi(a \mid s) \) that reads the state alone can be optimal. A trajectory is \( \tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots) \), and the discounted return from time \( t \) is

The discount does two jobs at once. Mathematically it makes the infinite sum converge and every operator below a contraction. Behaviorally it encodes that near rewards matter more than far ones, with an effective horizon of about \( 1/(1-\gamma) \) steps (at \( \gamma = 0.99 \), roughly 100 steps). The two functions that organize everything else are the state value and action value of a policy,

related by \( V^\pi(s) = \E_{a \sim \pi(\cdot\mid s)}[Q^\pi(s,a)] \). The advantage \( A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s) \) measures how much better action \( a \) is than the policy's average behavior in \( s \). It is the quantity every policy-gradient method below is ultimately estimating. The objective of the whole enterprise is \( J(\pi) = \E_{s_0 \sim \mu_0}[V^\pi(s_0)] \).

The Bellman expectation and optimality equations

Split the return at its first step, \( G_t = r_t + \gamma G_{t+1} \), and take expectations under the policy and the dynamics. This one-step decomposition gives the Bellman expectation equations,

These are linear in \( V^\pi \). For finite state spaces, \( V^\pi = (I - \gamma P^\pi)^{-1} r^\pi \) in matrix form, and the inverse exists because the spectral radius of \( \gamma P^\pi \) is at most \( \gamma < 1 \). The optimal value function replaces the expectation over the policy's actions with a maximization, giving the Bellman optimality equations,

These are nonlinear because of the max, but the operator \( (\mathcal{T}Q)(s,a) = r(s,a) + \gamma \E_{s'}[\max_{a'} Q(s',a')] \) is a \( \gamma \)-contraction in the sup norm. The proof is two lines. For any \( Q_1, Q_2 \),

where the first inequality uses \( |\max_x f(x) - \max_x g(x)| \le \max_x |f(x) - g(x)| \) (the max is 1-Lipschitz under the sup norm). By the Banach fixed-point theorem \( \mathcal{T} \) has a unique fixed point \( Q^* \), and iterating \( Q \leftarrow \mathcal{T}Q \) from any start converges geometrically. This is value iteration, and it is why tabular dynamic programming and tabular Q-learning come with clean guarantees. Puterman's book and Bertsekas's two volumes develop this theory in full, including average-reward and continuous-state cases this page does not need.

Function approximation and the deadly triad

Everything above assumed tables, one number per state or per state-action pair. With a neural network \( V_\phi \) or \( Q_\phi \), the update is no longer "set the entry to the target" but "take a gradient step toward the target", and the target itself is built from the network being updated. The precise trouble is stated by Sutton and Barto as the deadly triad. Instability and divergence become possible exactly when three ingredients combine. (1) Function approximation, so updating the value at one state moves the values of other states through shared parameters. (2) Bootstrapping, so the regression target \( r + \gamma V_\phi(s') \) depends on the current estimate rather than on a complete observed return. (3) Off-policy training, so the states being updated are weighted by a distribution different from the one the target policy would visit. Any two of the three are safe in a meaningful sense. Tabular Q-learning (no function approximation) converges under conditions given later. On-policy TD(0) with linear function approximation converges to a fixed point whose error is within a \( 1/(1-\gamma) \)-type factor of the best representable approximation. This is the Tsitsiklis and Van Roy (1997) result, and it works because the on-policy state distribution makes the projected Bellman operator a contraction under the corresponding weighted norm. Monte Carlo targets with function approximation, on or off policy, are ordinary supervised regression on a fixed objective and inherit SGD's guarantees.

Combine all three and the guarantees genuinely fail in small explicit examples. Baird's counterexample is a seven-state MDP with linear features on which off-policy TD diverges to infinity from almost any initialization. The mechanism is that the semi-gradient TD update is not the gradient of any objective (the target is treated as constant even though it depends on \( \phi \)), and under an off-policy weighting the expected update can have positive component along the error direction, so each step increases the very error it is reducing. Deep RL lives inside the triad by choice, because replay buffers and TD targets are too useful to give up, and the DQN lineage is best read as stabilizers bolted onto an update with no convergence theorem. Target networks freeze the bootstrap, replay decorrelates the distribution, double estimators remove a bias that function approximation amplifies. When a deep value method diverges in practice, and they do, this is the frame for diagnosing why.

The policy gradient theorem, from first principles

Policy-gradient methods skip value iteration entirely. Parameterize the policy \( \pi_\theta(a \mid s) \) and do gradient ascent on \( J(\theta) = \E_{\tau \sim \pi_\theta}[R(\tau)] \), where \( R(\tau) = \sum_{t=0}^{T-1} \gamma^t r_t \). The obstacle is that \( \theta \) affects \( J \) through the distribution of trajectories, not through a differentiable path from parameters to reward. The likelihood-ratio trick, used for policy search by Williams (1992), converts the gradient of an expectation into an expectation of a gradient. Write the trajectory density explicitly as

Differentiating under the integral and using the identity \( \nabla_\theta p_\theta = p_\theta \nabla_\theta \log p_\theta \) (the chain rule applied to the logarithm) gives

Now expand \( \log p_\theta(\tau) \), a sum of \( \log \mu_0(s_0) \), the policy terms \( \log \pi_\theta(a_t \mid s_t) \), and the dynamics terms \( \log P(s_{t+1} \mid s_t, a_t) \). Only the policy terms contain \( \theta \). The initial distribution and the transition kernel are properties of the environment, so their gradients are exactly zero. This is the step where the state distribution's derivative drops out, and it deserves a pause because it looks too good to be true. Changing \( \theta \) certainly changes which states the agent visits, so how can the gradient ignore that? The resolution is that the probability of reaching any state is a sum over paths, each path's probability a product whose only \( \theta \)-dependent factors are the \( \pi_\theta(a_t \mid s_t) \) along it, and the logarithm turns that product into the surviving sum. Nothing is ignored. The visitation shift is exactly accounted for by the score terms. The result is the REINFORCE estimator,

The same fact has a second formulation, the policy gradient theorem of Sutton, McAllester, Singh, and Mansour (2000), phrased in terms of the discounted state-visitation distribution \( d^\pi(s) \propto \sum_{t} \gamma^t \P(s_t = s) \),

The remarkable content is again that no \( \nabla_\theta d^\pi(s) \) term appears. The value-function route makes the cancellation mechanical. Differentiate the Bellman expectation equation, \( \nabla V^\pi(s) = \sum_a [\nabla \pi(a \mid s)\, Q^\pi(s,a) + \pi(a \mid s)\, \nabla Q^\pi(s,a)] \), note that \( \nabla Q^\pi(s,a) = \gamma \E_{s'}[\nabla V^\pi(s')] \) because reward and dynamics are \( \theta \)-free, and unroll the recursion. Each level contributes one \( \sum_a \nabla \pi \, Q^\pi \) term weighted by the discounted probability of being at that state at that depth, which is precisely \( d^\pi \). The two derivations are one theorem seen from the trajectory side and the state side.

Variance reduction I: causality and reward-to-go

REINFORCE as written multiplies every score term by the whole trajectory's return, including rewards earned before action \( a_t \) was taken. Those past rewards cannot depend on \( a_t \), so they contribute only variance. The proof that they can be dropped is the zero-mean property of the score function. For \( t' < t \), condition on the history \( h_t = (s_0, a_0, \dots, s_t) \), which determines \( r_{t'} \),

where the inner expectation vanishes because \( \E_{a \sim \pi}[\nabla \log \pi(a \mid s)] = \sum_a \pi(a \mid s) \frac{\nabla \pi(a \mid s)}{\pi(a \mid s)} = \nabla \sum_a \pi(a \mid s) = \nabla 1 = 0 \). Dropping those terms leaves the reward-to-go form, each action credited only with what followed it,

Variance reduction II: baselines, and the proof they are unbiased

The second reduction subtracts a state-dependent baseline \( b(s_t) \) from the reward-to-go. Unbiasedness is the same calculation once more, with \( b(s_t) \) in place of \( r_{t'} \). Conditioning on \( s_t \),

The critical requirement is that \( b \) depends on the state but not on the action taken in it. An action-dependent baseline would not factor out of the inner expectation and would in general bias the gradient. Any state-dependent \( b \), including a learned value network, leaves the estimator's expectation exactly equal to the true gradient while changing its variance. The variance-minimizing baseline can be derived by setting the derivative of the variance to zero. It is a score-weighted average of returns, close to but not exactly \( V^\pi(s) \). In practice \( V^\pi \) is used because it is the natural learnable surrogate, and subtracting it turns the reward-to-go into an estimate of the advantage \( A^\pi(s_t, a_t) \), centering the learning signal. Actions better than the policy's average get positive weight, worse ones negative, rather than everything being pushed up in proportion to a raw return that may be determined mostly by which state the agent happened to occupy.

Actor-critic: the TD error as a one-sample advantage

Learning \( V^\pi \) with a second network \( V_\phi \) (the critic) and using it both as the baseline and as a bootstrap gives actor-critic methods. The key identity is that the one-step TD error

is an unbiased estimate of the advantage when the value function is exact, since \( \E[\delta_t \mid s_t, a_t] = r(s_t,a_t) + \gamma \E[V^\pi(s_{t+1})] - V^\pi(s_t) = Q^\pi(s_t,a_t) - V^\pi(s_t) = A^\pi(s_t,a_t) \). With an approximate critic the estimate becomes biased, and the bias buys variance, since \( \delta_t \) involves one sampled reward and one bootstrap, versus the full-return estimator's sum over the remaining horizon. A3C (Mnih et al. 2016) made this family work at scale with parallel actors in place of a replay buffer. Its synchronous variant, A2C, is the skeleton inside PPO.

Generalized advantage estimation: the full bias-variance dial

Between the one-step TD error (low variance, biased by critic error) and the Monte Carlo advantage (unbiased, high variance) sits a family of \( k \)-step estimators. With \( \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t) \) built from the learned \( V \), the \( k \)-step advantage estimate is

where the second equality is a telescoping sum. The intermediate \( V \) terms cancel in pairs, leaving \( k \) real rewards, one bootstrap at depth \( k \), and the subtracted baseline. Small \( k \) leans on the critic (bias if the critic is wrong, little variance). Large \( k \) leans on sampled rewards (no critic bias, variance growing with the number of stochastic terms). Generalized advantage estimation (Schulman, Moritz, Levine, Jordan, Abbeel, 2015) takes the exponentially-weighted average with parameter \( \lambda \in [0,1] \),

Substituting the \( \delta \) form and swapping the order of summation, the coefficient collected by \( \delta_{t+l} \) is \( (1-\lambda)\, \gamma^l \sum_{k=l+1}^{\infty} \lambda^{k-1} = (1-\lambda)\, \gamma^l\, \frac{\lambda^{l}}{1-\lambda} = (\gamma\lambda)^l \), so the average collapses to a single geometric sum of TD errors,

The endpoints recover the extremes. \( \lambda = 0 \) gives \( \hat{A}_t = \delta_t \), the one-step actor-critic estimate, maximally biased by critic error and minimally noisy. \( \lambda = 1 \) gives \( \sum_l \gamma^l \delta_{t+l} = G_t - V(s_t) \) (telescoping again), the Monte Carlo return minus a baseline, unbiased regardless of the critic and maximally noisy. For intermediate \( \lambda \) the argument is quantitative. Critic error at horizon \( l \) enters with weight \( (\gamma\lambda)^l \), so bias decays geometrically with the depth at which the critic is consulted, the variance contribution of the \( l \)-th sampled reward is damped by the same factor, and the estimator effectively truncates the horizon at about \( 1/(1-\gamma\lambda) \) steps. The empirical sweet spot, \( \lambda \in [0.9, 0.97] \), reflects critics good enough to trust at short horizons and returns too noisy at long ones. Computationally the geometric form gives the backward recursion \( \hat{A}_t = \delta_t + \gamma\lambda \hat{A}_{t+1} \), one pass from the end of the rollout, cut at episode boundaries. That recursion is exactly what the implementations below compute.

Natural policy gradients and the Fisher information matrix

Vanilla gradient ascent moves \( \theta \) a fixed Euclidean distance, but Euclidean distance in parameter space is a poor proxy for how much the policy changed. A softmax layer's logits can absorb a large parameter step with almost no change in action probabilities, or a tiny step near saturation can flip a probability from 0.01 to 0.5. The principled fix, due to Kakade (2002) building on Amari's natural gradient, is to measure step size in distribution space using the KL divergence. Expand the KL between the current policy and a perturbed one to second order in the perturbation \( \delta \),

The zeroth-order term vanishes because the KL of a distribution with itself is zero, and the first-order term vanishes because the score has zero mean, so the Fisher information matrix \( F \), the covariance of the score, is the local metric on policy space. The natural gradient is the direction that maximizes improvement per unit of KL. Solve \( \max_\delta g\T \delta \) subject to \( \tfrac{1}{2} \delta\T F \delta \le \varepsilon \), where \( g = \nabla_\theta J \). The Lagrangian \( g\T\delta - \tfrac{\mu}{2}\delta\T F \delta \) gives \( \delta \propto F^{-1} g \), with the constraint fixing the scale,

The payoff is invariance. The update in distribution space does not depend on how the policy happens to be parameterized, so reparameterizing the network does not change the trajectory of policies visited. The cost is the linear solve against \( F \), handled in practice with conjugate gradient and Fisher-vector products rather than an explicit matrix.

Trust regions: the surrogate objective and TRPO's bound

Why insist on small policy changes at all? Because the policy-gradient estimate was computed under the current policy's state distribution, and it silently stops being valid as the policy moves. The tool for making this precise is the performance difference lemma (Kakade and Langford, 2002). For any two policies,

The proof is a telescoping argument. Write \( A^\pi(s_t,a_t) = \E_{s_{t+1}}[\, r_t + \gamma V^\pi(s_{t+1}) - V^\pi(s_t) \,] \) and sum under \( \tau \sim \pi' \). The reward terms sum to \( J(\pi') \), while the value terms form the telescoping series \( \sum_t \big( \gamma^{t+1} V^\pi(s_{t+1}) - \gamma^t V^\pi(s_t) \big) \), which collapses to \( -V^\pi(s_0) \), whose expectation is \( -J(\pi) \). So improving on \( \pi \) means finding \( \pi' \) with positive expected advantage under \( \pi' \)'s own state distribution, which is the thing we cannot sample from without deploying \( \pi' \). The practical surrogate replaces \( d^{\pi'} \) with \( d^{\pi} \) and corrects the action distribution with an importance ratio,

\( L_\pi \) matches \( J \) to first order at \( \pi' = \pi \) (same value, same gradient), and TRPO's theorem (Schulman et al., 2015) bounds how fast they diverge,

The shape of the bound is the important part. The surrogate is trustworthy up to a penalty quadratic in the policy change (KL is locally quadratic) with a constant that blows up as \( (1-\gamma)^{-2} \), the square of the horizon, because a state distribution shift compounds over time. Maximizing the right-hand side at each step yields a monotonic improvement guarantee. Since the right side equals \( J(\pi) \) at \( \pi' = \pi \), any \( \pi' \) that increases it satisfies \( J(\pi') \ge J(\pi) \). The penalty constant \( C \) is far too conservative to use literally, so TRPO in practice swaps the penalty for a hard constraint on the average KL, \( \bar{\KL}(\pi \| \pi') \le \delta \) with \( \delta \) around 0.01, and solves the constrained problem with conjugate gradient plus a line search. This works, but it is a second-order method, expensive per step, awkward with shared parameters, and hostile to minibatching.

PPO: the clipped surrogate as a cheap trust region

PPO (Schulman et al., 2017) keeps the surrogate and replaces the KL machinery with a clipping heuristic. With \( \rho_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)} \) and advantage estimates \( \hat{A}_t \) from GAE, the objective is

The case analysis explains the design. For \( \hat{A}_t > 0 \) the objective grows with \( \rho_t \) but the clip caps the reward at \( (1+\epsilon)\hat{A}_t \). Once the new policy already assigns the action \( 1+\epsilon \) times its old probability, there is no incentive to push further, because the min selects the clipped, constant branch whose gradient is zero. For \( \hat{A}_t < 0 \) the mirror happens at \( 1-\epsilon \). The min with the unclipped term matters for the pessimistic direction. If an earlier epoch overshot (say \( \rho_t = 1.6 \) with a negative advantage), the unclipped branch is the smaller one and its gradient is live, so the objective still pulls the ratio back. The clip removes the incentive to move further away. It does not remove the ability.

An honest account of what the clip guarantees is that it guarantees nothing, formally. It is not a trust region. Ratios routinely end up outside \( [1-\epsilon, 1+\epsilon] \), because a minibatch update moves all ratios simultaneously and only per-sample gradients are zeroed once outside the band. The KL between successive policies is not bounded by any function of \( \epsilon \), and no monotonic improvement theorem survives the approximation. What keeps PPO stable is the conjunction of the clip, few epochs per batch (so the policy never strays far before fresh data arrives), minibatch shuffling, and learning-rate discipline. Engstrom, Ilyas et al. (ICLR 2020) showed that much of PPO's measured advantage over TRPO came from code-level choices rather than the clipped objective, and Andrychowicz et al. (2020) reached the same conclusion from a different codebase across dozens of design axes. In deep RL, implementation details are not below the algorithmic waterline, they are the waterline.

The details that matter, in rough order of impact. Advantage normalization standardizes \( \hat{A} \) per minibatch to zero mean and unit variance, which makes the gradient scale independent of the reward scale and is close to mandatory. The value loss fits \( V_\phi \) to the GAE returns \( \hat{A}_t + V_{\mathrm{old}}(s_t) \). The popular clipped value loss (clip the value prediction's movement to \( \pm\epsilon \) of its old value, take the max of the two errors) is inherited from the original implementation and the ablation evidence for it is mixed at best, but it is what the reference code does. An entropy bonus adds \( c_2 \E[\mathcal{H}(\pi(\cdot|s))] \) (typically \( c_2 = 0.01 \)) to delay premature determinism. For epochs and minibatching, use 3 or 4 epochs of 4 to 32 minibatches per rollout batch. More epochs squeeze more from the data but increase the off-policy drift the clip must absorb. Orthogonal initialization with a small policy-head gain (0.01), learning-rate annealing to zero, gradient-norm clipping at 0.5, and reward normalization on continuous-control suites round out the list. Huang et al.'s "37 implementation details" post documents each with ablations, and the implementation section below includes all of them.

rollout (N envs x T steps)          optimize (K epochs)
┌──────────────────────────┐      ┌─────────────────────────────┐
│ pi_old collects          │      │ shuffle T*N samples         │
│ (s, a, logp_old, r, done)│ ───► │ for each minibatch:         │
│ V_old(s) recorded        │      │   rho = exp(logp - logp_old)│
│ GAE backward pass        │      │   L = min(rho*A, clip*A)    │
│  A_t = d_t + gl*A_{t+1}  │      │   + value loss + entropy    │
└──────────────────────────┘      └─────────────────────────────┘
        ▲                                      │
        └────────── weights sync ◄─────────────┘

Off-policy value methods: Q-learning to DQN

Q-learning (Watkins, 1989) is the sampled version of value iteration. After observing \( (s, a, r, s') \), update

It is off-policy, since the max over next actions evaluates the greedy policy regardless of how the data was collected, so any sufficiently exploratory behavior policy can feed it. In the tabular case the convergence theorem (Watkins and Dayan, 1992, and Jaakkola, Jordan, and Singh, 1994, via stochastic approximation) requires that every state-action pair is visited infinitely often, that step sizes satisfy the Robbins-Monro conditions \( \sum_t \alpha_t = \infty \) and \( \sum_t \alpha_t^2 < \infty \), and that rewards are bounded. Under these, \( Q \to Q^* \) with probability 1. Note what the theorem does not require (on-policy data) and what it silently assumes (a table, so no generalization between entries). The deadly triad section already explained which of these breaks first under function approximation.

DQN (Mnih et al., 2015) made the update work with a convolutional network on Atari by attacking the two most acute instabilities. Experience replay stores transitions in a large FIFO buffer and samples minibatches uniformly, breaking the temporal correlation of consecutive frames and reusing each transition many times. Target networks compute the bootstrap \( r + \gamma \max_{a'} Q_{\theta^-}(s', a') \) with a frozen copy \( \theta^- \), updated every \( C \) steps or by Polyak averaging, so the regression target does not chase the regressor within an update cycle. Without it, raising \( Q(s,a) \) immediately raises the targets of every state that can reach \( s \), a positive feedback loop through shared parameters. The loss is the Huber-smoothed TD error. The behavior policy is \( \epsilon \)-greedy, annealed to a small floor.

Double DQN: why the max of estimates is biased upward

The max operator in the Q-learning target introduces a systematic overestimation. The general fact is Jensen's inequality applied to the (convex) max function. For any random estimates \( \hat{Q}(a) \) with \( \E[\hat{Q}(a)] = Q(a) \),

with equality only when the argmax is almost surely constant. The estimator maximizes over noise as well as signal. Whichever action happens to be overestimated in this sample tends to win the max. A concrete computation shows the size of the effect. Suppose three actions all have true value 0, and each estimate is independently off by \( \pm 0.1 \) with probability \( 1/2 \) each. The max of the three estimates is \( -0.1 \) only when all three are low, probability \( 1/8 \), and otherwise it is \( +0.1 \). So \( \E[\max_a \hat{Q}(a)] = \tfrac{7}{8}(0.1) + \tfrac{1}{8}(-0.1) = 0.075 \), a bias of three quarters of the noise scale, from unbiased inputs. Worse, the bias grows with the number of actions and with the noise, and in TD learning it compounds. An inflated \( Q(s') \) becomes an inflated target for \( Q(s) \), which becomes an inflated target for its predecessors.

The double estimator (van Hasselt, 2010, applied to DQN by van Hasselt, Guez, and Silver, 2016) breaks the correlation between selecting the argmax and evaluating it. Use the online network to choose and the target network to score,

In the toy example, selecting the argmax with one estimate and evaluating with an independent second estimate gives an expected value of exactly 0, because the evaluator's error is independent of which action was selected. In Atari, where DQN's learned values demonstrably exceed the returns its greedy policy actually achieves, Double DQN both reduces the value inflation and improves play. The same decoupling idea reappears in TD3's clipped double critics for continuous control.

Dueling networks, prioritized replay, and n-step returns

The dueling architecture (Wang et al., 2016) decomposes \( Q(s,a) = V(s) + A(s,a) \) into two heads sharing a torso. The raw decomposition is unidentifiable (adding a constant to \( A \) and subtracting it from \( V \) changes nothing), so the aggregation subtracts the mean advantage, \( Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s,a') \). The benefit is that in states where the action choice barely matters, one gradient step through \( V \) updates all action values at once, and value estimation stops being entangled with action ranking.

Prioritized experience replay (Schaul et al., 2016) samples transition \( i \) with probability \( P(i) \propto p_i^\alpha \), where \( p_i = |\delta_i| + \varepsilon_p \) is its last absolute TD error, so surprising transitions are replayed more. Non-uniform sampling biases the expected gradient, so each sample carries the importance-sampling correction \( w_i = \big( \tfrac{1}{N} \cdot \tfrac{1}{P(i)} \big)^\beta \), normalized by \( \max_i w_i \), with \( \beta \) annealed from about 0.4 to 1 over training (full correction matters most near convergence). New transitions enter with maximal priority so they are seen at least once.

n-step returns replace the one-step target with \( r_t + \gamma r_{t+1} + \cdots + \gamma^{n-1} r_{t+n-1} + \gamma^n \max_{a'} Q_{\theta^-}(s_{t+n}, a') \), propagating reward information \( n \) steps per update instead of one. This is the same bias-variance dial as GAE, with a wrinkle. Taken from a replay buffer, the intermediate actions were chosen by an old policy, so the uncorrected n-step target is subtly off-policy. Rainbow uses \( n = 3 \) without correction and it helps anyway, an instructive case of theory and practice diverging.

Distributional RL: C51 and quantile regression

Distributional methods model the full distribution of the return \( Z(s,a) \), not just its mean \( Q(s,a) = \E[Z(s,a)] \). The distributional Bellman equation \( Z(s,a) \overset{D}{=} r + \gamma Z(s', a') \) is a contraction in the Wasserstein metric. C51 (Bellemare, Dabney, and Munos, 2017) represents \( Z \) as a categorical distribution over 51 fixed atoms \( z_i \) spanning \( [v_{\min}, v_{\max}] \). The Bellman update shifts and shrinks the atoms to \( r + \gamma z_i \), which land between grid points, so the probability mass is projected back onto the grid by linear interpolation, and the network is trained by cross-entropy against the projected target. QR-DQN (Dabney et al., 2018) transposes the parameterization. Fix \( N \) probability levels \( \tau_i = \frac{2i-1}{2N} \) and learn the quantile locations \( \theta_i(s,a) \), trained with the quantile regression loss \( \E\big[ \rho_{\tau}(u) \big] \) where \( \rho_\tau(u) = u \cdot (\tau - \mathbf{1}\{u < 0\}) \) (in practice its Huber-smoothed version), whose minimizer is exactly the \( \tau \)-quantile. This removes C51's fixed support and projection step. Why modeling the distribution helps even when only the mean is used to act remains debated. The leading explanations are richer training signal (an auxiliary-task effect) and reduced harm from state aliasing, while the empirical gains are consistent.

Rainbow: which pieces mattered

Rainbow (Hessel et al., 2018) stacked six extensions on DQN, namely double Q, dueling, prioritized replay, n-step returns, distributional C51, and noisy linear layers for exploration. The paper's ablation, removing one component at a time from the full agent, is the part worth remembering. Removing prioritized replay or multi-step returns hurt median Atari performance most, removing distributional learning hurt late-training performance next, while removing double Q or dueling made comparatively little difference once the rest were present (distributional learning's clipped support already tempers overestimation). The meta-lesson recurs throughout deep RL. Improvements are not additive, and an ablation against the full stack is worth more than six papers' individual comparisons against vanilla DQN.

Continuous control I: the deterministic policy gradient

With continuous actions the max in Q-learning becomes an optimization problem per step, and a stochastic policy's likelihood-ratio gradient carries variance that grows with action dimension. The deterministic policy gradient (Silver et al., 2014) sidesteps both. For a deterministic policy \( a = \mu_\theta(s) \), the objective \( J(\theta) = \E_{s \sim d^{\mu}}[ Q^\mu(s, \mu_\theta(s)) ] \) can be differentiated through the critic by the chain rule,

The theorem's content parallels the stochastic case. The state distribution's gradient again drops out (the proof unrolls \( \nabla_\theta V^\mu \) through the Bellman equation exactly as before, with the sum over actions replaced by evaluation at \( \mu_\theta(s) \)), and Silver et al. showed the deterministic gradient is the limit of the stochastic policy gradient as the policy's variance shrinks to zero. The estimator trades the score-function's high variance for reliance on \( \nabla_a Q \). The critic must now be accurate not just in value but in slope, which is a stronger requirement and the root of the DDPG family's fragility. Exploration must be added externally (the policy itself is deterministic), typically with Gaussian or Ornstein-Uhlenbeck noise on the actions.

DDPG (Lillicrap et al., 2016) is DPG plus the DQN toolkit, meaning a replay buffer, target networks for both actor and critic with Polyak averaging \( \theta^- \leftarrow \tau\theta + (1-\tau)\theta^- \), and action noise for exploration. It works, and it is famously brittle, sensitive to hyperparameters, prone to critic overestimation followed by actor exploitation of the inflated regions, and hard to reproduce, a combination documented across many independent studies.

Continuous control II: TD3's three fixes

TD3 (Fujimoto, van Hoof, and Meger, 2018) diagnosed DDPG's failure mode as overestimation feeding actor exploitation, and applied three targeted repairs. First, clipped double Q-learning. Train two critics and build the target from the minimum, \( y = r + \gamma \min_{j=1,2} Q_{\theta_j^-}(s', \tilde{a}') \). The reasoning is that a deterministic actor climbing \( \nabla_a Q \) actively seeks out the critic's overestimated regions, so the upward bias analyzed in the Double DQN section is not a passive nuisance but an amplified one. The min of two roughly independent estimates errs low, and a pessimistic critic merely slows learning while an optimistic one poisons the actor. Second, delayed policy updates. Update the actor and targets once per two critic updates, so the actor climbs a settled value landscape rather than chasing a critic in mid-regression. Third, target policy smoothing. Compute the target action as \( \tilde{a}' = \mu_{\theta^-}(s') + \mathrm{clip}(\eta, -c, c) \), \( \eta \sim \mathcal{N}(0, \sigma^2) \). A deterministic target evaluates \( Q \) at a single point, so a narrow spurious spike in the critic surface becomes a target other states bootstrap from. Averaging over a smoothed action distribution erases spikes narrower than the noise scale. Each fix addresses a specific term in the error analysis, which is why TD3 ablates cleanly and why its ideas migrated into later algorithms.

Continuous control III: SAC from the maximum-entropy objective

SAC (Haarnoja et al., 2018) changes the objective itself, augmenting reward with the policy's entropy at every step,

The temperature \( \alpha \) prices entropy in reward units. There are three practical consequences. Exploration is part of the objective rather than bolted-on noise. The policy is pushed to stay stochastic wherever action choice is not consequential, which improves robustness. And the softened landscape connects to a family of results linking entropy-regularized RL with probabilistic inference. The soft Q-function absorbs the entropy of future steps, \( Q^\pi_{\mathrm{soft}}(s,a) = r(s,a) + \gamma\, \E_{s'}\big[ V^\pi_{\mathrm{soft}}(s') \big] \) with \( V^\pi_{\mathrm{soft}}(s) = \E_{a \sim \pi}[ Q^\pi_{\mathrm{soft}}(s,a) - \alpha \log \pi(a|s) ] \), and the policy improvement step solves, per state,

the projection of the Boltzmann distribution over Q-values onto the representable policy class. Haarnoja et al. prove this improves the soft value at every state (soft policy iteration). For a Gaussian policy the KL objective is optimized by the reparameterization trick rather than the score function. Sample \( a = f_\theta(s, \epsilon) = \tanh\big( m_\theta(s) + \sigma_\theta(s) \odot \epsilon \big) \), \( \epsilon \sim \mathcal{N}(0, I) \), and differentiate the loss \( \E_\epsilon[ \alpha \log \pi_\theta(f_\theta(s,\epsilon) \mid s) - Q(s, f_\theta(s,\epsilon)) ] \) directly through \( f_\theta \) and the critic, a pathwise gradient with far lower variance than REINFORCE-style estimates.

The tanh squash is where a classic implementation bug lives. The Gaussian sample \( u \) is squashed to \( a = \tanh(u) \) to respect action bounds, and the density must follow the change of variables. For an invertible map, \( \pi(a \mid s) = \mathcal{N}(u \mid m, \sigma^2)\, \Big| \det \frac{\partial a}{\partial u} \Big|^{-1} \). The Jacobian of an elementwise tanh is diagonal with entries \( 1 - \tanh^2(u_i) \), so

Omit the correction and every entropy and KL term in SAC is computed against the wrong density. The algorithm often still runs, which is exactly why the bug survives code review. There is also a numerical trap inside the correct formula. The quantity \( 1 - \tanh^2(u) \) underflows for \( |u| \gtrsim 10 \) in float32, so production implementations (Spinning Up popularized this form) use the identity \( \log(1 - \tanh^2 u) = 2\big( \log 2 - u - \mathrm{softplus}(-2u) \big) \), which follows from \( 1 - \tanh^2 u = \operatorname{sech}^2 u = 4 e^{-2u} / (1 + e^{-2u})^2 \), and is finite for all \( u \). The implementation section verifies the two forms agree to float32 precision and that the resulting agent learns.

Finally, the temperature. A fixed \( \alpha \) must be tuned per environment because it trades off against the unknown reward scale. The automatic variant (Haarnoja et al., 2018, the "algorithms and applications" paper) recasts the problem as constrained optimization, maximizing return subject to \( \E[ -\log \pi(a|s) ] \ge \bar{\mathcal{H}} \), a minimum entropy target, by default \( \bar{\mathcal{H}} = -\dim(\mathcal{A}) \). Lagrangian duality turns this into a per-step update of \( \alpha \) minimizing

descended in \( \log \alpha \) for positivity. When the policy's entropy is below target the gradient raises \( \alpha \) (pricing entropy higher), and vice versa. In the verified run on this page, \( \alpha \) fell from 0.44 to 0.019 over 30k steps as the policy sharpened, which is the mechanism working as designed.

Model-based RL: Dyna, model bias, and planning with learned dynamics

Model-free methods pay for every gradient with environment interaction. Model-based methods learn the dynamics \( \hat{P}(s' \mid s, a) \) and generate cheap synthetic experience or plan through the model directly. The oldest clean formulation is Dyna (Sutton, 1991), which interleaves real steps, model updates, and Q-learning updates on model-sampled transitions, so each real transition is amortized across many imagined ones. The failure mode is model bias. A learned model is accurate near the data and wrong elsewhere, an optimizer is a machine for finding maxima, and the maxima of a wrong model are often exactly where it is wrong. Long imagined rollouts compound one-step error, the same \( O(T^2) \) arithmetic as behavior cloning below, so modern model-based RL is largely a set of answers to "how do we keep the optimizer away from the model's fantasies".

PETS (Chua et al., 2018) answers with uncertainty and short horizons. Learn an ensemble of probabilistic dynamics networks (each a Gaussian over \( s' \) for aleatoric noise, ensemble disagreement for epistemic uncertainty) and plan at decision time by model-predictive control. Sample action sequences, propagate particles through random ensemble members, refit the sampling distribution with the cross-entropy method (keep the elites, refit a Gaussian, iterate), execute the first action, replan. Nothing is trained to exploit the model over long horizons, so the bias has nowhere to compound. MBPO (Janner et al., 2019) transplants the caution into policy learning. Train SAC on real data plus short branched rollouts of length 1 to 5 started from real buffer states. The paper's bound makes the design explicit. True return is lower-bounded by model return minus terms growing with model error times rollout length, so short branches from on-distribution states keep the bound tight while multiplying the effective data.

The Dreamer line (Hafner et al., 2020-2023) commits fully to a latent world model. An RSSM (recurrent state-space model) encodes observations into a compact stochastic latent, trained by reconstruction and KL regularization. The actor and critic are then trained entirely inside the latent space, on imagined rollouts, with DreamerV1's gradients flowing through the differentiable dynamics. DreamerV2 switched to categorical latents and matched DQN-line agents on Atari. DreamerV3 added symlog transforms, return normalization, and robustness tricks that let one fixed hyperparameter setting work across 150+ tasks, including collecting diamonds in Minecraft from scratch. The actor-critic-in-latent-space idea is the durable contribution. Imagination is cheap, the model horizon stays modest (15 steps), and the critic bootstraps beyond the imagination horizon, exactly the GAE arithmetic from earlier applied inside a learned model.

MuZero (Schrittwieser et al., 2020) drops reconstruction entirely. It learns three functions, a representation \( h(o_{1:t}) \to z \), a dynamics \( g(z, a) \to (z', \hat{r}) \), and a prediction \( f(z) \to (\hat{p}, \hat{v}) \) giving policy prior and value. The latent is trained only to produce correct rewards, values, and policy targets along real trajectories, value-equivalent rather than generative, modeling only what planning needs. Acting runs AlphaZero-style MCTS in latent space. Search results (visit-count distributions and n-step returns) become the training targets, so search and learning bootstrap each other. MuZero matched AlphaZero at Go, chess, and shogi without being given the rules, the cleanest demonstration that a planning model need not be a simulator.

Imitation learning: behavior cloning and the compounding-error bound

When expert demonstrations exist, the simplest use of them is behavior cloning, fitting \( \pi_\theta(a \mid s) \) to expert pairs by supervised learning. The subtlety is that supervised learning controls the error on the expert's state distribution, while the cloned policy is evaluated on its own. Ross and Bagnell (2010) made the gap precise, and the derivation is short enough to give in full. Assume per-state costs in \( [0,1] \), horizon \( T \), and a learned policy whose probability of disagreeing with the expert is at most \( \epsilon \) on states drawn from the expert's distribution. Consider running \( \pi_\theta \) for \( T \) steps. As long as it has not yet deviated, it walks the expert's distribution, so at each such step it deviates with probability at most \( \epsilon \). Once it deviates, assume the worst. It is off distribution, its error rate is unbounded (the bound gives it cost 1 per step), and it never recovers. If the first deviation happens at step \( t \), the excess cost is at most \( T - t \). Summing over when the first mistake happens,

using \( \P(\text{first mistake at } t) \le \epsilon \) and \( \sum_{t=1}^T (T-t) = T(T-1)/2 \le T^2/2 \) (the constant is \( 1/2 \), and the order is what matters). The bound is tight. Ross and Bagnell exhibit MDPs where cloned policies really do incur \( \Theta(T^2 \epsilon) \) cost, because one mistake sends the agent somewhere the expert never goes, where it has learned nothing. This is covariate shift with a feedback loop, and it is the formal version of every practical report of a cloned driving policy drifting to the road edge and not knowing how to come back.

DAgger (Ross, Gordon, and Bagnell, 2011) repairs the bound by changing the data distribution rather than the learner. Roll out the current policy, have the expert label the states the policy visits, aggregate into the dataset, retrain, and repeat. Because the learner is trained on its own induced state distribution, the reduction to no-regret online learning gives \( J(\pi) \le J(\pi^*) + O(uT\epsilon_N) + o(1) \) with the quadratic dependence gone (\( u \) bounds the cost increase of a single deviation, \( \epsilon_N \) is the achievable supervised error on the aggregated distribution). The price is an interactive expert, which is exactly what is often unavailable. Much of imitation learning since is about approximating DAgger's distribution correction without DAgger's query access.

Inverse RL, maximum entropy, and GAIL

Inverse RL asks the complementary question, not "copy the actions" but "recover the reward the expert is optimizing", on the theory that reward transfers to new dynamics while actions do not. The problem is ill-posed (many rewards, including the zero reward, rationalize any behavior), and maximum-entropy IRL (Ziebart et al., 2008) resolves the ambiguity with a principle. Among all trajectory distributions matching the expert's feature expectations, choose the maximum-entropy one, the exponential family \( p(\tau) \propto \exp(\theta\T f(\tau)) \) under deterministic dynamics. Training maximizes demonstration likelihood. The gradient is \( \E_{\text{expert}}[f] - \E_{p_\theta}[f] \), so each step solves the soft planning problem induced by the current reward, the expensive inner loop that limits classic IRL to small problems. The formulation is also the bridge to SAC's soft-optimality view. The same \( \exp(Q/\alpha) \) Boltzmann policies appear in both.

GAIL (Ho and Ermon, 2016) removes the inner planning loop by reframing imitation as distribution matching. Every policy induces an occupancy measure \( \rho_\pi(s,a) \), the discounted visitation frequency of state-action pairs, and imitation becomes minimizing a divergence between \( \rho_\pi \) and \( \rho_{\text{expert}} \). GAIL does this adversarially. A discriminator \( D_\psi(s,a) \) learns to separate policy pairs from expert pairs, the policy takes RL steps (TRPO or PPO) on the reward \( -\log D_\psi(s,a) \), and at the saddle point the occupancy measures match, the GAN argument transported to RL. The structure is worth noticing. The RLHF pipeline below is the same architecture with the discriminator replaced by a preference-trained reward model and the adversarial loop replaced by a KL leash.

Offline RL: distributional shift and the conservative repairs

Offline (batch) RL learns a policy from a fixed dataset with no further interaction, the setting of medical, industrial, and recommendation logs. The naive approach, Q-learning on the buffer, fails for a sharper reason than the deadly triad. The Bellman target \( \max_{a'} Q(s', a') \) queries actions the dataset never contains, the function approximator freely extrapolates there, the max seeks out its overestimates (the Problem 4 mechanism with no environment to correct it), and the policy converges on actions precisely because no data contradicts their inflated values. Online RL self-corrects by executing the overrated action and observing disappointment. Offline RL cannot, so the error is structural. Levine et al.'s 2020 survey frames the whole field as managing this distributional shift.

Conservative Q-learning (Kumar et al., 2020) attacks the value side. Learn a Q-function whose implied values are a lower bound, by adding to the Bellman error a term that pushes down Q on actions the policy favors and up on actions the dataset contains,

where \( \mu \) is a distribution concentrated on high-value actions (in practice a log-sum-exp over actions, making the first term a soft maximum). The paper proves that with enough weight the resulting value estimate lower-bounds the true value of the learned policy, converting the overestimation failure into controlled pessimism. Unknown actions look bad, so the policy stays near the data unless deviation is genuinely supported.

Implicit Q-learning (Kostrikov, Nair, and Levine, 2022) removes the out-of-distribution query altogether. The trick is expectile regression. Fit a state-value network \( V_\psi \) to the \( \tau \)-expectile of \( Q(s,a) \) over dataset actions by minimizing the asymmetric loss \( \E_{(s,a) \sim \D}\big[ |\tau - \mathbf{1}\{u < 0\}| \, u^2 \big] \) with \( u = Q_{\hat\theta}(s,a) - V_\psi(s) \). At \( \tau = 0.5 \) this is regression to the mean. As \( \tau \to 1 \) it approaches the maximum of Q over the actions the dataset actually took at similar states, a max computed implicitly, without ever evaluating an action outside the data. The Q-network then regresses on \( r + \gamma V_\psi(s') \), and a policy is extracted at the end by advantage-weighted regression, \( \max_\pi \E[\exp(\beta A(s,a)) \log \pi(a|s)] \), a weighted behavior clone tilted toward good dataset actions. IQL's combination of simplicity and strong D4RL results made it the default offline baseline for years.

Decision Transformer (Chen et al., 2021) reframes offline RL as sequence modeling. Tokenize trajectories as (return-to-go, state, action) triples, train a causal transformer to predict actions, and at test time condition on a high desired return. No value function, no Bellman backup, no pessimism term. Its limitations follow from the framing. It cannot stitch. If the dataset has a good first half in some trajectories and a good second half in others, dynamic programming composes them, but return-conditioned modeling has never seen the composite return and cannot synthesize it. And in stochastic environments, conditioning on high return selects trajectories that got lucky as much as those that acted well, so the model imitates gamblers. Supervised sequence modeling buys stability and scale, and pays with exactly the compositional credit assignment that Bellman methods provide.

Exploration: from epsilon-greedy to RND, and why it is unsolved

Every method so far explores by injected noise, \( \epsilon \)-greedy random actions, Gaussian action noise, or policy entropy. Dithering works when reward is dense enough that random deviations stumble onto signal, and fails categorically on sparse long-horizon tasks. The canonical example, Atari's Montezuma's Revenge, requires a long exact sequence before any reward, and undirected noise reaches it with probability exponentially small in the sequence length. The principled tabular answer is optimism. Count visits \( N(s,a) \) and add a bonus \( \propto 1/\sqrt{N(s,a)} \), which is what UCB-style regret analyses justify. Deep RL's problem is that in large or continuous state spaces every state is visited once, so the count is meaningless. The modern methods are attempts to generalize counting.

Pseudo-counts (Bellemare et al., 2016) derive a count surrogate from a learned density model \( \rho \) over states. Comparing the model's probability of \( s \) before and after training on it (the prediction gain) yields a quantity \( \hat{N}(s) \) that behaves like a count, recovering true counts in the tabular case, and the bonus \( 1/\sqrt{\hat{N}(s)} \) produced the first meaningful progress on Montezuma. ICM curiosity (Pathak et al., 2017) uses learned forward-model error. Encode states with features trained by an inverse-dynamics objective (predict the action between two states, which discards environment noise the agent cannot control), and reward the agent where the forward model predicts those features poorly. RND (Burda et al., 2018) is the simpler, more robust distillation of the same idea. Fix a randomly initialized target network \( f \), train a predictor \( \hat{f} \) to match it on visited states, and use \( \| \hat{f}(s) - f(s) \|^2 \) as the novelty bonus. Because the target is a fixed deterministic function, irreducible environment stochasticity cannot keep the error high, which defuses the "noisy TV" failure that traps forward-model curiosity, where a source of random noise is eternally surprising to a dynamics model but quickly learnable as a function evaluation.

Exploration remains largely unsolved at scale for several reasons. The bonuses above measure novelty of states, but hard problems require novelty of behaviors over long horizons. A bonus that decays as states become familiar is a non-stationary reward that value functions track poorly. And intrinsic rewards distort the objective, with scale tuning that is problem-specific. Go-Explore (Ecoffet et al., 2021) solved Montezuma by abandoning the bonus framework entirely (archive states, return deterministically, explore from there), which is telling. The best result on the canonical hard-exploration task came from outside the theory. In LLM reinforcement learning the same gap reappears as entropy collapse and the difficulty of getting a model to try genuinely different reasoning strategies rather than reweighting existing ones. The current answers (temperature, entropy bonuses, data curricula) are the epsilon-greedy of this era.

RL for language models

The token-level MDP

Fine-tuning a language model with RL instantiates a specific, degenerate, and important MDP. The state is the prompt plus the tokens generated so far, \( s_t = (x, y_{1:t-1}) \). The action is the next token \( y_t \) from a vocabulary of tens of thousands. The transition is deterministic concatenation, \( s_{t+1} = s_t \circ y_t \). And the reward is sparse and terminal, zero until the sequence ends, then a single scalar \( r(x, y) \) from a reward model, a verifier, or a human. Several classical difficulties vanish. The dynamics are known and deterministic, so there is no model-learning problem and no transition noise. What remains is pure credit assignment over horizons of hundreds to tens of thousands of tokens against one terminal scalar, in an action space where the pretrained model is the only thing making search tractable. Every algorithm below is a small, carefully leashed perturbation of that initialization. Discounting is usually \( \gamma = 1 \) (sequences are finite), and the value function, where one exists, predicts expected final reward from a prefix.

Reward modeling from preferences: Bradley-Terry, derived

Human judgments arrive most reliably as comparisons. Shown two completions, annotators say which is better, with far higher agreement than absolute scoring. The Bradley-Terry model (1952) converts pairwise outcomes into latent scores. Derive it from a latent-utility view. Suppose comparing \( y_1 \) against \( y_2 \) is governed by noisy utilities \( u_i = r(x, y_i) + \varepsilon_i \) with \( \varepsilon_i \) i.i.d. Gumbel, and \( y_1 \) is preferred when \( u_1 > u_2 \). The difference of two independent Gumbels is logistic, so

the logistic function of the score difference. Training a reward model \( r_\phi \) is then maximum likelihood on comparisons. For a dataset of triples \( (x, y_w, y_l) \) with \( y_w \) preferred,

Only differences are identified (adding any function of \( x \) to \( r_\phi \) changes nothing), which is harmless for ranking and consequential for RL, since it means absolute reward values carry no meaning across prompts. The reward model is typically the pretrained LM with its unembedding replaced by a scalar head, trained on hundreds of thousands of comparisons (Ouyang et al. 2022, and Bai et al. 2022, whose HH-RLHF data is the standard open corpus).

The central failure mode is overoptimization, Goodhart's law applied to a learned proxy. Gao, Schulman, and Hilton (2023) measured it cleanly. Train a "gold" reward model, optimize against a smaller proxy RM, and plot gold score against KL from the initial policy. Proxy reward rises monotonically while gold reward rises, peaks, and then falls, with the peak following smooth scaling laws in \( d = \sqrt{\KL(\pi \| \pi_{\mathrm{init}})} \). Gold reward behaves like \( d(\alpha_{\mathrm{bon}} - \beta_{\mathrm{bon}} d) \) for best-of-n and \( d(\alpha_{\mathrm{RL}} - \beta_{\mathrm{RL}} \log d) \) for RL, with coefficients improving predictably in RM size and data. The practical readings follow. Distance from the reference is the right x-axis for optimization pressure. Larger reward models tolerate more optimization before Goodharting. And no amount of RM scale removes the eventual turnover, which is why every production pipeline bounds the KL rather than trusting the proxy.

RLHF with PPO: the KL leash and the instabilities

The RLHF objective as run in InstructGPT (Ouyang et al., 2022), descending from Christiano et al. (2017) and Ziegler et al. (2019), is KL-regularized reward maximization against a frozen reference policy \( \pi_{\mathrm{ref}} \) (usually the SFT model),

The KL term exists for three separable reasons. First, proxy validity. The reward model was trained on samples near \( \pi_{\mathrm{ref}} \)'s distribution, and the overoptimization curves say its judgments degrade with distance, so the leash keeps the policy where the proxy means something. Second, capability retention. Unconstrained maximization of a narrow reward cannibalizes behaviors the reward does not measure (fluency, calibration, knowledge). Third, entropy. The KL to a high-entropy reference opposes collapse onto a few high-reward modes. In implementations the sequence-level KL becomes a per-token penalty \( -\beta \log \frac{\pi_\theta(y_t \mid s_t)}{\pi_{\mathrm{ref}}(y_t \mid s_t)} \) added to the reward stream (dense shaping for an otherwise sparse problem), with the RM score arriving at the final token. PPO with GAE then runs exactly as derived earlier, one value head predicting expected final reward from each prefix. The value head is commonly initialized from the reward model, which starts it at roughly the right output scale. From-scratch initialization costs a period of useless high-variance advantages while the critic calibrates.

The practical instability modes are consistent enough across labs to list. Reward hacking, where the policy finds RM artifacts (verbosity, sycophancy, formats), visible as proxy reward rising while human evaluations stall. Length is the classic hack, strong enough that length-controlled evaluation became standard. Entropy collapse, where the policy sharpens onto few completions and learning stalls for lack of exploration. Critic divergence under distribution shift. KL blowups when \( \beta \) is too small, stagnation when too large. And the LLM-specific plumbing hazard, where log-probabilities computed by the inference engine during generation and by the training framework during optimization can disagree (different kernels, batching, precision), silently corrupting the importance ratios, a real production bug class that importance-correction schemes in modern frameworks exist to absorb.

DPO, derived in full

Direct preference optimization (Rafailov et al., 2023) starts from the observation that the KL-regularized objective above has a closed-form optimum, and that closed form can be run backwards. The derivation, in careful steps. Fix a prompt \( x \) and consider the inner optimization over the distribution \( \pi(\cdot \mid x) \),

Define the partition function \( Z(x) = \sum_y \pi_{\mathrm{ref}}(y \mid x)\, e^{r(x,y)/\beta} \) and the candidate distribution \( \pi^*(y \mid x) = \pi_{\mathrm{ref}}(y \mid x)\, e^{r(x,y)/\beta} / Z(x) \), which is normalized by construction. Rewrite the bracket by multiplying and dividing by \( Z(x) \),

\( \log Z(x) \) does not depend on \( \pi \), and the KL is minimized, uniquely, at zero. So the optimal policy is the Boltzmann tilt of the reference,

This is exact but useless directly, since \( Z(x) \) sums over all completions. The move that makes DPO work is inversion. Take logarithms and solve for the reward,

Every reward function is, up to a per-prompt constant, a scaled log-ratio between its own optimal policy and the reference. Now substitute into the Bradley-Terry likelihood. The preference probability depends only on the reward difference at the same prompt, so the intractable \( \beta \log Z(x) \) cancels,

Finally, parameterize the unknown optimal policy directly as \( \pi_\theta \) and do maximum likelihood on the preference dataset. The reward model has disappeared as a separate object. The policy itself is the implicit reward model,

The gradient is instructive. Writing \( \hat{r}_\theta(x,y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)} \) for the implicit reward,

The update pushes up the chosen completion and down the rejected one, with a weight that vanishes as the pair becomes correctly ranked with margin, a self-annealing property plain cloning lacks. What is given up relative to PPO-style RLHF is equally concrete. DPO is offline. It optimizes on the preference dataset's completions, not on samples from the evolving policy, so the state distribution mismatch that online RL continuously repairs is baked in. The Bradley-Terry loss is optimized by driving the margin to infinity when preferences are (nearly) deterministic, which in practice can push probability mass off both \( y_w \) and \( y_l \) onto unseen sequences. The measured symptom is that \( \log \pi_\theta(y_w) \) often falls during DPO training, with the margin maintained by \( \log \pi_\theta(y_l) \) falling faster. And the implicit reward is only trained where the data is, so DPO inherits offline RL's distributional-shift fragility without CQL-style pessimism.

The variants each patch one weakness. IPO (Azar et al., 2023, Google DeepMind) replaces the log-sigmoid with a squared loss pinning the margin to a finite target \( 1/(2\beta) \), removing the drive to infinite margins under deterministic preferences. KTO (Ethayarajh et al., 2024) learns from unpaired thumbs-up/down signals with a Kahneman-Tversky-inspired asymmetric value function. Paired comparisons are expensive, so this matters operationally. SimPO (Meng, Xia, and Chen, 2024) removes the reference model, using length-normalized average log-probability as the implicit reward plus a target margin. ORPO (Hong et al., 2024) folds preference optimization into SFT with an odds-ratio penalty. None changes the fundamental trade. Offline preference methods buy stability and cheapness by giving up on-policy sampling, and the strongest pipelines (Llama 3's post-training is a public example) interleave them with online methods rather than choosing once.

GRPO: group-relative advantages, no value network

PPO for LLMs carries a value network the size of the policy, trained to predict expected reward from every prefix, purely to serve as a baseline. GRPO (Shao et al., 2024, the DeepSeekMath paper) observes that in the LLM setting an empirical baseline is available for free. Sample a group of \( G \) completions \( \{y_1, \dots, y_G\} \) for the same prompt, score each with the reward function, and use the group statistics as the baseline. With outcome rewards, every token of completion \( i \) receives the same advantage

and the objective keeps PPO's clipped ratio per token plus an explicit KL penalty to the reference policy, estimated with the low-variance unbiased estimator \( \KL \approx \frac{\pi_{\mathrm{ref}}}{\pi_\theta} - \log \frac{\pi_{\mathrm{ref}}}{\pi_\theta} - 1 \) (nonnegative term by term, unlike the naive \( \log \pi_\theta - \log \pi_{\mathrm{ref}} \) sample). The value network is gone, a second full model's memory freed, no critic warmup, no value-loss tuning, and the baseline is exact for the prompt at hand rather than an amortized prediction. What is paid is baseline variance that shrinks only as \( 1/G \) with each group member a full generation, so the cost moved from training a critic to sampling rollouts, and the method needs multiple samples per prompt, natural for verifiable single-turn tasks and awkward for long multi-turn interaction. The design fits verifiable-reward settings precisely because rewards there are cheap, objective, and high-variance across samples, which is when an empirical per-prompt baseline shines.

The normalization arithmetic has real consequences. On a binary-reward task where the model solves a prompt with probability \( p \), a group of \( G \) samples has mean \( \approx p \) and standard deviation \( \approx \sqrt{p(1-p)} \), so correct answers get advantage \( \sqrt{(1-p)/p} \) and wrong ones \( -\sqrt{p/(1-p)} \). Rare successes on hard prompts receive very large advantages, and the division by a near-zero standard deviation on almost-always-solved or almost-never-solved prompts amplifies noise. Follow-up work made two specific critiques. The \( 1/\mathrm{std} \) normalization introduces a difficulty-dependent bias (easy and hard prompts are weighted up relative to medium ones), and GRPO's per-sequence length normalization biases against long correct answers and toward long incorrect ones. Dr. GRPO (Liu et al., 2025, Sea AI Lab) removes both terms, and DAPO (Yu et al., 2025, ByteDance) adds asymmetric clip ranges, dynamic filtering of all-correct and all-wrong groups (whose advantages are zero or pure noise), and token-level aggregation. RLOO (Ahmadian et al., 2024, Cohere) is the closely related leave-one-out variant. Baseline each sample with the mean of the other \( G-1 \) rewards, no std division, which is exactly unbiased. All of these are REINFORCE with a per-prompt Monte Carlo baseline. The family resemblance to the bandit arithmetic of Problem 2 is not a coincidence.

Verifiable rewards, rejection sampling, and test-time compute

RL with verifiable rewards (RLVR) replaces the learned reward model with a checker, exact-match or symbolic equivalence for math, unit tests for code, or format validators for structure. The reward is incorruptible in the Goodhart sense (though not hack-proof, since models game weak test suites and format leniency) and free to query, so the binding constraint becomes generation throughput, not annotation. DeepSeek-R1 (2025) is the load-bearing public result. R1-Zero applied GRPO with rule-based accuracy and format rewards directly to a base model, no SFT stage, and chain-of-thought length, reflection, and self-correction emerged over training. The full R1 pipeline interleaves cold-start SFT, reasoning RL, rejection-sampled SFT regeneration, and a final RL stage. Tülu 3 (Lambert et al., 2024, Allen AI) documented an open RLVR recipe at smaller scale. OpenAI's o-series, less documented, established the pattern first.

The simplest member of this family predates the more elaborate ones, rejection sampling / expert iteration. Sample \( k \) completions per prompt, keep the verified-correct ones, fine tune on the survivors, repeat. STaR (Zelikman et al., 2022), ReST (Gulcehre et al., 2023), and the rejection-sampling stages inside Llama and DeepSeek pipelines are all this loop, which is policy iteration with a hard, verifier-defined improvement operator. It is more stable than PPO-family RL, strictly on-policy in its data, and limited by the fact that it only reinforces what sampling already finds. Process versus outcome supervision is the credit-assignment fork. Lightman et al. (2023) trained a process reward model on 800k human step-level labels (PRM800K) and showed process supervision substantially outperforms outcome supervision for selecting MATH solutions. The operational difficulty is that step labels are expensive and automated step-scoring (Monte Carlo completion value from each step) reintroduces a learnable, hackable proxy, so production systems mix both signals.

The test-time-compute picture ties the section together. Repeated sampling with verification (Brown et al., 2024) shows coverage, the chance that at least one of \( k \) samples is correct, scaling smoothly over orders of magnitude of \( k \). Snell et al. (2024) showed adaptive test-time allocation can beat spending the same FLOPs on a larger model. And RLVR is what converts test-time search into trained-in ability, moving the pass@k distribution toward pass@1. The consensus recipe as of 2025 is to pretrain, SFT, preference-tune, then spend increasingly large RL budgets on verifiable domains with GRPO-family optimizers, the cost dominated by generation.

Infrastructure reality: the generation-training split

An LLM RL step is two different workloads glued together. Generation is autoregressive decoding, one token at a time, each token a matrix-vector pass over all weights, arithmetic intensity of order 1 FLOP per weight byte at small batch. Training is dense matmul at high arithmetic intensity. The H100 in this machine measures 2,992 GB/s of streaming HBM bandwidth and 729 bf16 TFLOPS on 8192-wide matmuls (classes/data/h100.json). The machine balance is roughly 240 bf16 FLOPs per byte, so an unbatched decode step, at about 1 FLOP per byte, is bound to a fraction of a percent of peak compute. LLM-RL systems design follows from that number. Rollouts dominate wall clock unless generation is batched aggressively, which is why every serious framework embeds an inference engine, vLLM or SGLang, with paged KV-cache and continuous batching. The remaining hard problem is weight synchronization. Training holds sharded FSDP or Megatron parameters, inference holds tensor-parallel copies, and after every update the new weights must be resharded and pushed without draining the pipeline. Stale weights make fresh data slightly off-policy. Frameworks accept the bias, correct it with truncated importance sampling, or enforce synchronous alternation.

The two open-source systems worth knowing map onto two scale regimes. TRL (Hugging Face) is the accessible one, with trainer classes (SFTTrainer, DPOTrainer, GRPOTrainer, PPOTrainer) built on transformers and accelerate, with optional vLLM-backed generation and PEFT/LoRA integration. verl (ByteDance, the HybridFlow paper, EuroSys 2025) is the production one, a single-controller programming model where the algorithm is written as dataflow over resource pools of actor, rollout, critic, and reward workers, letting the same GRPO script run colocated on 8 GPUs or disaggregated across thousands, with 3D-parallel training and vLLM/SGLang rollout workers exchanging resharded weights. The division of labor is the industry pattern in miniature. The algorithm is fifty lines, and the system that feeds it is the other hundred thousand.

Evaluation and reproducibility

Deep RL results are noisy in a way supervised learning does not prepare one for. Henderson et al. (2018) showed that two groups of five seeds of the same algorithm, same hyperparameters, same codebase can produce learning curves a naive read would call two different methods, and that reported baselines for the same algorithm differed across papers by more than most claimed improvements. Agarwal et al. (2021) quantified the field-wide consequence and proposed the now-standard repairs, interquartile mean over many runs, stratified bootstrap confidence intervals, and performance profiles (the rliable library). The standard suites each measure something different. The Arcade Learning Environment (Bellemare et al., 2013) and its Atari-100k variant for sample efficiency, Gymnasium MuJoCo and the DeepMind Control Suite for continuous control, Procgen for generalization, D4RL for offline RL. LLM-RL evaluation inherits all of this plus its own pathologies. The pass@1 metric is a high-variance Bernoulli mean entangled with sampling temperature, contamination is pervasive, and length and format effects masquerade as capability. The working rules are to use never fewer than three seeds (and know that three is thin), to report distributions rather than best runs, to fix and publish the evaluation harness, and to treat any improvement smaller than the seed spread as noise until proven otherwise.

Worked problems

Six problems are distributed through the theory above (the Bellman system, the bandit baseline, the GAE computation, the double-Q bias, the DPO arithmetic, and the GRPO group). Two more belong here, one derivation and one numerical exercise on the PPO objective itself.

Implementation

Three implementations, each verified on this machine's H100 (PyTorch 2.7, CUDA 12.8, JAX 0.6). The PyTorch PPO below was run as-is on CartPole-v1: over 400k environment steps (8 vectorized environments, 184 seconds wall clock) the 20-episode mean return went from 26 at the first update to 413 at the last, peaking near 437. The GAE routine was additionally checked against the hand-computed numbers of Problem 3. The JAX column carries the same algorithm as pure functions. Its GAE and loss outputs were checked to agree with the PyTorch versions to within float32 tolerance (max difference 4e-5 on random batches).

"""PPO with GAE on CartPole-v1. Verified: 26 -> 413 mean return,
400k steps, 184 s on one H100 (most of the time is env stepping)."""
import numpy as np
import torch
import torch.nn as nn
import gymnasium as gym

device = "cuda" if torch.cuda.is_available() else "cpu"

class ActorCritic(nn.Module):
    """Separate actor and critic MLPs for a discrete action space."""
    def __init__(self, obs_dim, n_actions, hidden=64):
        super().__init__()
        def mlp(out_dim, gain):
            layers, dims = [], [obs_dim, hidden, hidden]
            for i in range(len(dims) - 1):
                lin = nn.Linear(dims[i], dims[i + 1])
                nn.init.orthogonal_(lin.weight, np.sqrt(2))  # detail: ortho init
                nn.init.zeros_(lin.bias)
                layers += [lin, nn.Tanh()]
            head = nn.Linear(hidden, out_dim)
            nn.init.orthogonal_(head.weight, gain)   # 0.01 keeps the initial
            nn.init.zeros_(head.bias)                # policy near-uniform
            return nn.Sequential(*layers, head)
        self.pi = mlp(n_actions, gain=0.01)          # logits: (B, A)
        self.v = mlp(1, gain=1.0)                    # value:  (B, 1)

    def value(self, obs):                            # (B,)
        return self.v(obs).squeeze(-1)

    def dist(self, obs):
        return torch.distributions.Categorical(logits=self.pi(obs))

def compute_gae(rewards, values, dones, last_value, gamma=0.99, lam=0.95):
    """rewards, values, dones: (T, N); last_value: (N,).
    Backward recursion A_t = delta_t + gamma*lam*(1-done_t)*A_{t+1}."""
    T, N = rewards.shape
    adv = torch.zeros_like(rewards)
    gae = torch.zeros(N, device=rewards.device)
    for t in reversed(range(T)):
        next_v = last_value if t == T - 1 else values[t + 1]
        nonterminal = 1.0 - dones[t]                 # cut at episode ends
        delta = rewards[t] + gamma * next_v * nonterminal - values[t]
        gae = delta + gamma * lam * nonterminal * gae
        adv[t] = gae
    return adv, adv + values                         # (advantages, value targets)

def ppo_train(env_id="CartPole-v1", total_steps=400_000, n_envs=8,
              rollout_len=128, epochs=4, minibatches=4, clip=0.2,
              vf_coef=0.5, ent_coef=0.01, lr=2.5e-4, max_grad_norm=0.5):
    envs = gym.vector.SyncVectorEnv(
        [lambda: gym.make(env_id) for _ in range(n_envs)])
    obs_dim = envs.single_observation_space.shape[0]
    net = ActorCritic(obs_dim, envs.single_action_space.n).to(device)
    opt = torch.optim.Adam(net.parameters(), lr=lr, eps=1e-5)

    obs, _ = envs.reset(seed=0)
    obs = torch.as_tensor(obs, dtype=torch.float32, device=device)
    n_updates = total_steps // (n_envs * rollout_len)

    for update in range(n_updates):
        opt.param_groups[0]["lr"] = lr * (1 - update / n_updates)  # anneal
        # ---- rollout: (T, N) tensors under the current policy ----
        b_obs = torch.zeros(rollout_len, n_envs, obs_dim, device=device)
        b_act = torch.zeros(rollout_len, n_envs, dtype=torch.long, device=device)
        b_logp, b_rew, b_done, b_val = (
            torch.zeros(rollout_len, n_envs, device=device) for _ in range(4))
        for t in range(rollout_len):
            with torch.no_grad():
                dist = net.dist(obs)
                act = dist.sample()
                b_logp[t] = dist.log_prob(act)       # log pi_old(a|s)
                b_val[t] = net.value(obs)
            b_obs[t], b_act[t] = obs, act
            nobs, rew, term, trunc, _ = envs.step(act.cpu().numpy())
            done = np.logical_or(term, trunc)
            b_rew[t] = torch.as_tensor(rew, dtype=torch.float32, device=device)
            b_done[t] = torch.as_tensor(done, dtype=torch.float32, device=device)
            obs = torch.as_tensor(nobs, dtype=torch.float32, device=device)
        with torch.no_grad():
            last_value = net.value(obs)              # bootstrap for step T
        adv, ret = compute_gae(b_rew, b_val, b_done, last_value)

        # ---- flatten (T, N) -> (T*N,) and optimize for K epochs ----
        f_obs = b_obs.reshape(-1, obs_dim)
        f_act, f_logp = b_act.reshape(-1), b_logp.reshape(-1)
        f_adv, f_ret, f_val = adv.reshape(-1), ret.reshape(-1), b_val.reshape(-1)
        batch = rollout_len * n_envs
        mb_size = batch // minibatches
        for _ in range(epochs):
            perm = torch.randperm(batch, device=device)
            for start in range(0, batch, mb_size):
                idx = perm[start:start + mb_size]
                dist = net.dist(f_obs[idx])
                logp = dist.log_prob(f_act[idx])
                ratio = (logp - f_logp[idx]).exp()   # pi/pi_old per sample
                mb_adv = f_adv[idx]
                mb_adv = (mb_adv - mb_adv.mean()) / (mb_adv.std() + 1e-8)
                # clipped surrogate: pessimistic minimum (Problem 8)
                pg1 = -mb_adv * ratio
                pg2 = -mb_adv * ratio.clamp(1 - clip, 1 + clip)
                pg_loss = torch.max(pg1, pg2).mean()
                v = net.value(f_obs[idx])
                v_clip = f_val[idx] + (v - f_val[idx]).clamp(-clip, clip)
                v_loss = 0.5 * torch.max((v - f_ret[idx]) ** 2,
                                         (v_clip - f_ret[idx]) ** 2).mean()
                ent = dist.entropy().mean()
                loss = pg_loss + vf_coef * v_loss - ent_coef * ent
                opt.zero_grad()
                loss.backward()
                nn.utils.clip_grad_norm_(net.parameters(), max_grad_norm)
                opt.step()
    return net

"""PPO core in JAX: GAE as a backward lax.scan, the clipped loss,
and a jitted update. Verified to match the PyTorch version to 4e-5
on random batches. Env stepping stays in Python (or use gymnax to
jit the whole loop)."""
import jax
import jax.numpy as jnp
import optax

def compute_gae(rewards, values, dones, last_value,
                gamma=0.99, lam=0.95):
    """rewards, values, dones: (T, N); last_value: (N,)."""
    next_values = jnp.concatenate([values[1:], last_value[None]], axis=0)
    deltas = rewards + gamma * next_values * (1 - dones) - values

    def step(gae, x):                        # scan runs the backward
        delta, nonterminal = x               # recursion of Problem 3
        gae = delta + gamma * lam * nonterminal * gae
        return gae, gae

    _, adv_rev = jax.lax.scan(
        step, jnp.zeros_like(last_value),
        (deltas[::-1], (1 - dones)[::-1]))   # reverse time, scan, reverse
    adv = adv_rev[::-1]
    return adv, adv + values                 # (advantages, value targets)

def ppo_loss(params, apply_fn, batch, clip=0.2,
             vf_coef=0.5, ent_coef=0.01):
    """batch: dict of flattened (B,) arrays + obs (B, obs_dim)."""
    logits, value = apply_fn(params, batch["obs"])   # (B, A), (B,)
    logp_all = jax.nn.log_softmax(logits)
    logp = jnp.take_along_axis(
        logp_all, batch["act"][:, None], axis=1)[:, 0]
    ratio = jnp.exp(logp - batch["logp_old"])        # pi/pi_old
    adv = batch["adv"]
    adv = (adv - adv.mean()) / (adv.std() + 1e-8)    # per-minibatch norm
    pg_loss = jnp.mean(jnp.maximum(
        -adv * ratio,
        -adv * jnp.clip(ratio, 1 - clip, 1 + clip)))
    v_clip = batch["val_old"] + jnp.clip(
        value - batch["val_old"], -clip, clip)
    v_loss = 0.5 * jnp.mean(jnp.maximum(
        (value - batch["ret"]) ** 2,
        (v_clip - batch["ret"]) ** 2))
    entropy = -jnp.mean(jnp.sum(jnp.exp(logp_all) * logp_all, axis=-1))
    return pg_loss + vf_coef * v_loss - ent_coef * entropy

@jax.jit
def ppo_update(params, opt_state, batch, apply_fn, tx):
    loss, grads = jax.value_and_grad(ppo_loss)(params, apply_fn, batch)
    updates, opt_state = tx.update(grads, opt_state, params)
    params = optax.apply_updates(params, updates)
    return params, opt_state, loss

# wiring: tx = optax.chain(optax.clip_by_global_norm(0.5),
#                          optax.adam(2.5e-4, eps=1e-5))
# rollout in Python exactly as in the PyTorch tab, then for each
# epoch: permute indices, slice minibatches, call ppo_update.

Second, SAC on Pendulum-v1, with the squashed-Gaussian log-probability computed in the numerically stable form. In the verified run, 30k environment steps took 299 seconds, and the 10-episode mean return improved from about \( -424 \) at 5k steps (random-policy territory is \( -1200 \) and worse) to \( -121 \) at 30k, with the auto-tuned temperature falling from 0.44 to 0.019 as the policy sharpened. The two forms of the tanh correction were checked to agree to 5e-7 in float32 for moderate \( u \), while the naive \( \log(1 - \tanh^2 u) \) needs an epsilon to avoid \( -\infty \) at \( |u| \gtrsim 10 \).

"""SAC actor with the tanh log-prob correction, twin critics, and
the temperature update. Verified on Pendulum-v1: -424 -> -121 mean
return over 30k steps; alpha auto-tuned 0.44 -> 0.019."""
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

LOG_STD_MIN, LOG_STD_MAX = -5.0, 2.0

class SquashedGaussianActor(nn.Module):
    def __init__(self, obs_dim, act_dim, act_limit, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU())
        self.mu = nn.Linear(hidden, act_dim)
        self.log_std = nn.Linear(hidden, act_dim)
        self.act_limit = act_limit

    def forward(self, obs):                      # obs: (B, obs_dim)
        h = self.net(obs)
        mu = self.mu(h)                          # (B, act_dim)
        log_std = torch.clamp(self.log_std(h), LOG_STD_MIN, LOG_STD_MAX)
        std = log_std.exp()
        dist = torch.distributions.Normal(mu, std)
        u = dist.rsample()                       # reparameterized: mu + std*eps
        a = torch.tanh(u)                        # squash to (-1, 1)
        # Change of variables: log pi(a|s) = log N(u) - sum_i log(1 - tanh(u_i)^2)
        # Stable identity:     log(1 - tanh(u)^2) = 2*(log 2 - u - softplus(-2u))
        # Omitting this Jacobian term is the classic SAC bug: the entropy
        # and the alpha loss are then computed against the wrong density.
        logp = dist.log_prob(u).sum(-1)
        logp = logp - (2 * (np.log(2.0) - u - F.softplus(-2 * u))).sum(-1)
        return self.act_limit * a, logp          # (B, act_dim), (B,)

def sac_losses(actor, q1, q2, q1_targ, q2_targ, log_alpha,
               batch, gamma=0.99, target_entropy=-1.0):
    o, a, r, o2, d = (batch[k] for k in ("obs", "act", "rew", "obs2", "done"))
    alpha = log_alpha.exp().detach()

    with torch.no_grad():                        # critic target
        a2, logp2 = actor(o2)
        q_targ = torch.min(q1_targ(o2, a2), q2_targ(o2, a2))  # clipped double Q
        backup = r + gamma * (1 - d) * (q_targ - alpha * logp2)
    q_loss = F.mse_loss(q1(o, a), backup) + F.mse_loss(q2(o, a), backup)

    a_pi, logp_pi = actor(o)                     # pathwise policy gradient
    q_pi = torch.min(q1(o, a_pi), q2(o, a_pi))
    pi_loss = (alpha * logp_pi - q_pi).mean()    # KL projection objective

    # temperature: raise alpha when entropy is below target, lower when above
    alpha_loss = -(log_alpha.exp() *
                   (logp_pi.detach() + target_entropy)).mean()
    return q_loss, pi_loss, alpha_loss
# Training loop: replay buffer, one gradient step per env step for each
# of (q_loss, pi_loss, alpha_loss), Polyak-average the target critics
# with tau = 0.005. Full script mirrors spinningup's sac.py.

"""SAC's squashed-Gaussian log-prob and losses in JAX. The
correction term was checked to agree with the PyTorch version to
5e-7 in float32."""
import jax
import jax.numpy as jnp

LOG_STD_MIN, LOG_STD_MAX = -5.0, 2.0

def sample_action(params, apply_actor, obs, key):
    """Returns squashed action and its log-prob, reparameterized."""
    mu, log_std = apply_actor(params, obs)       # (B, A), (B, A)
    log_std = jnp.clip(log_std, LOG_STD_MIN, LOG_STD_MAX)
    std = jnp.exp(log_std)
    eps = jax.random.normal(key, mu.shape)
    u = mu + std * eps                           # pathwise sample
    a = jnp.tanh(u)
    normal_logp = (-0.5 * ((u - mu) / std) ** 2 - log_std
                   - 0.5 * jnp.log(2 * jnp.pi)).sum(-1)
    # log(1 - tanh(u)^2) = 2*(log 2 - u - softplus(-2u)); finite for all u,
    # unlike the direct form which underflows for |u| > ~10 in float32.
    correction = (2 * (jnp.log(2.0) - u
                       - jax.nn.softplus(-2.0 * u))).sum(-1)
    return a, normal_logp - correction           # (B, A), (B,)

def critic_loss(q_params, actor_params, targ_params, log_alpha,
                apply_q, apply_actor, batch, key, gamma=0.99):
    a2, logp2 = sample_action(actor_params, apply_actor,
                              batch["obs2"], key)
    q1_t = apply_q(targ_params[0], batch["obs2"], a2)
    q2_t = apply_q(targ_params[1], batch["obs2"], a2)
    q_targ = jnp.minimum(q1_t, q2_t)             # clipped double Q
    backup = batch["rew"] + gamma * (1 - batch["done"]) * (
        q_targ - jnp.exp(log_alpha) * logp2)
    backup = jax.lax.stop_gradient(backup)
    q1 = apply_q(q_params[0], batch["obs"], batch["act"])
    q2 = apply_q(q_params[1], batch["obs"], batch["act"])
    return ((q1 - backup) ** 2).mean() + ((q2 - backup) ** 2).mean()

def actor_and_alpha_loss(actor_params, q_params, log_alpha,
                         apply_q, apply_actor, batch, key,
                         target_entropy=-1.0):
    a_pi, logp_pi = sample_action(actor_params, apply_actor,
                                  batch["obs"], key)
    q_pi = jnp.minimum(apply_q(q_params[0], batch["obs"], a_pi),
                       apply_q(q_params[1], batch["obs"], a_pi))
    alpha = jnp.exp(log_alpha)
    pi_loss = (jax.lax.stop_gradient(alpha) * logp_pi - q_pi).mean()
    alpha_loss = -(alpha * jax.lax.stop_gradient(
        logp_pi + target_entropy)).mean()
    return pi_loss, alpha_loss
# Polyak update: targ = jax.tree.map(
#     lambda t, p: (1 - tau) * t + tau * p, targ, params)

Third, the DPO loss with exact reference-model handling. Implementations get two things wrong. The log-probabilities must be summed over completion tokens only (prompt tokens masked out), and the reference model's log-probs must be computed with gradients disabled, either from a frozen copy or precomputed once over the dataset (which halves memory and is what TRL's precompute_ref_log_probs does). On the numbers of Problem 5 this code returns loss 0.5544 and implicit rewards \( (+0.2, -0.1) \), matching the hand computation.

import torch
import torch.nn.functional as F

def completion_logps(logits, labels, completion_mask):
    """Sum of token log-probs over the completion only.
    logits: (B, T, V); labels: (B, T); completion_mask: (B, T) with 1
    on completion tokens, 0 on prompt and padding."""
    logps = torch.log_softmax(logits, dim=-1)
    tok = torch.gather(logps, 2, labels.unsqueeze(-1)).squeeze(-1)  # (B, T)
    return (tok * completion_mask).sum(-1)                          # (B,)

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps, ref_rejected_logps, beta=0.1):
    """All inputs: (B,) summed completion log-probs. The ref logps
    must come from a frozen model under torch.no_grad(), or be
    precomputed over the dataset."""
    pi_ratio = policy_chosen_logps - policy_rejected_logps
    ref_ratio = ref_chosen_logps - ref_rejected_logps
    logits = beta * (pi_ratio - ref_ratio)       # implicit reward margin
    loss = -F.logsigmoid(logits).mean()
    # diagnostics: implicit rewards, and accuracy of the implicit RM
    chosen_rw = beta * (policy_chosen_logps - ref_chosen_logps).detach()
    rejected_rw = beta * (policy_rejected_logps - ref_rejected_logps).detach()
    return loss, chosen_rw, rejected_rw

# Verified against Problem 5:
# dpo_loss(torch.tensor([-10.]), torch.tensor([-9.]),
#          torch.tensor([-12.]), torch.tensor([-8.]))
# -> loss 0.5544, chosen reward +0.2, rejected reward -0.1

import jax
import jax.numpy as jnp

def completion_logps(logits, labels, completion_mask):
    """logits: (B, T, V); labels, completion_mask: (B, T)."""
    logps = jax.nn.log_softmax(logits, axis=-1)
    tok = jnp.take_along_axis(
        logps, labels[..., None], axis=-1)[..., 0]   # (B, T)
    return (tok * completion_mask).sum(-1)           # (B,)

def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps, ref_rejected_logps, beta=0.1):
    """Ref logps are data here (precomputed or from a frozen apply
    wrapped in jax.lax.stop_gradient), so no gradient flows to them."""
    pi_ratio = policy_chosen_logps - policy_rejected_logps
    ref_ratio = jax.lax.stop_gradient(
        ref_chosen_logps - ref_rejected_logps)
    logits = beta * (pi_ratio - ref_ratio)
    loss = -jax.nn.log_sigmoid(logits).mean()
    chosen_rw = beta * (policy_chosen_logps - ref_chosen_logps)
    rejected_rw = beta * (policy_rejected_logps - ref_rejected_logps)
    return loss, (jax.lax.stop_gradient(chosen_rw),
                  jax.lax.stop_gradient(rejected_rw))

# Same check: inputs (-10, -9, -12, -8), beta 0.1
# -> loss 0.5544, margin 0.3, matching the hand computation.

How it is done in practice

The gap between the derivations and a production system is wide in classical deep RL and wider in LLM RL. For classical control and games the practice layer includes vectorized environments (the dominant cost in the verified CartPole run was Python environment stepping, not the network, which is why 400k steps took 184 seconds with the GPU mostly idle), observation and reward normalization with running statistics, saved with the checkpoint because a policy is meaningless without its normalizer, correct handling of time-limit truncation versus true termination (bootstrapping through a timeout as if it were death corrupts the value function, the reason Gymnasium separates terminated from truncated), and seeds, many of them. Frameworks split by philosophy. Stable-Baselines3 gives audited reference implementations behind a uniform API, CleanRL gives single-file implementations where every detail is visible at once, Tianshou and Acme give modular research scaffolding.

For LLM RL the practice layer is a distributed system. A GRPO training step at production scale runs as follows. Broadcast current weights to a fleet of vLLM or SGLang workers. Generate \( G \) completions per prompt with continuous batching (minutes of wall clock, where the arithmetic-intensity analysis bites). Run verifiers or reward models. Compute group advantages (microseconds, as in Problem 6). Recompute token log-probs under training precision. Take a few clipped-ratio gradient steps with the KL penalty. Then reshard and resync weights. Typical hyperparameters sit in narrow bands, KL coefficients \( 10^{-3} \) to \( 10^{-1} \), clip \( \epsilon \) 0.2 (sometimes asymmetric, as in DAPO), group sizes 8 to 64, one to a few epochs per batch, learning rates around \( 10^{-6} \) for full fine-tuning. The failure modes on call rotation are entropy collapse (watch mean token entropy), length drift (watch mean completion length), verifier exploits (read samples, always), and training-inference log-prob mismatch (assert on ratio statistics, since ratios far from 1 on the first epoch mean the two engines disagree about the same weights). The measured H100 numbers explain the budget. Generation at small batch runs orders of magnitude below peak, so the fleet exists to buy batching, and rollout throughput, not optimizer step time, sets the experiment cadence.

The current research frontier

The active edges, as of early 2026, with the groups pushing them. In RL for reasoning at scale, DeepSeek's R1 line and its replications (Qwen at Alibaba, Kimi at Moonshot, Mistral's Magistral) established GRPO-family training on verifiable rewards as the standard. The open questions are whether RLVR elicits new capabilities or sharpens sampling of existing ones (the pass@k debates, with Tsinghua-affiliated work arguing elicitation and others showing growth under sustained scale), extension to long-horizon agentic and tool-use settings, and entropy collapse over thousands of RL steps. Algorithmic refinements include Dr. GRPO (Sea AI Lab) and DAPO (ByteDance) on advantage normalization, VinePPO (Mila) on Monte Carlo credit assignment, RLOO (Cohere) on unbiased baselines, plus renewed theory on why minimal REINFORCE variants match PPO for LLMs. Process supervision and search covers process reward models past PRM800K (OpenAI, with Qwen and Shanghai groups on automated step labels), and the interaction of trained reasoning with inference-time search (Berkeley and Google DeepMind on compute-optimal test-time scaling). Preference-method theory takes in DPO's probability-mass squeezing, when offline matches online RL (Oxford, Cambridge, Berkeley), and iterated online-DPO schemes. Classical deep RL continues with DreamerV3 (Google DeepMind) as the sample-efficiency frontier, offline-to-online fine-tuning (Berkeley, CMU), and robot learning where RL fine-tunes behavior-cloned foundations (Physical Intelligence, NVIDIA, Toyota Research). And infrastructure is itself a research axis, spanning asynchronous off-policy-tolerant pipelines, importance corrections for stale rollouts, and open systems (verl, OpenRLHF, TRL, NeMo-RL) converging on disaggregated generation.

Open source to read

Nine codebases, ordered roughly from pedagogy to production, with the file to open first in each.

Common misconceptions

"The policy gradient ignores how the state distribution changes." It does not. It accounts for it exactly. The trajectory measure's only \( \theta \)-dependent factors are the per-step policy probabilities, so the score of the whole trajectory is the sum of per-step scores, and the visitation shift is carried entirely by those terms. What is true is that no separate \( \nabla d^\pi \) term needs to be estimated, which is the theorem's convenience, not an approximation.

"Baselines reduce variance because they make the rewards smaller." The mechanism is centering, not shrinking. A baseline that subtracted a huge constant would still be unbiased and would make variance worse. The variance reduction comes from making the multiplier of the score term close to zero on average per state (Problem 2 reaches exactly zero variance this way), and an action-dependent baseline, which looks like a further improvement, breaks the unbiasedness proof.

"PPO's clip enforces a trust region." It enforces nothing. Ratios leave the clip band routinely (once outside with the clipped branch active, the per-sample gradient is zero, so nothing pushes them back except the min in the adverse direction), and the KL between successive policies is unbounded by \( \epsilon \). PPO's stability comes from the clip plus few epochs, fresh data, minibatch shuffling, and learning-rate discipline together. The Engstrom/Ilyas and Andrychowicz studies both found that code-level choices explain much of the measured differences between algorithms.

"Q-learning converges, so DQN converges." The tabular theorem needs conditions (infinite visitation, Robbins-Monro step sizes) that function approximation does not merely weaken but voids. With bootstrapping and off-policy replay the update is not a gradient of any objective, and Baird's counterexample diverges. DQN's target networks and replay are stabilizers, not a proof, and deep value methods still diverge in practice.

"The max in the Q-target is fine because the estimates are unbiased." Unbiased inputs still give a biased output, since \( \E[\max] \ge \max[\E] \) by Jensen, with equality only when the argmax is deterministic. The worked example shows unbiased \( \pm 0.1 \) noise producing a +0.075 bias, and bootstrapping compounds it. Decoupling selection from evaluation (Double DQN, TD3's twin critics) removes the correlation that creates the bias.

"SAC's tanh squash is just an activation function." It is a change of variables, and the density must pay the Jacobian, so \( \log \pi(a|s) = \log \mathcal{N}(u) - \sum_i \log(1 - \tanh^2 u_i) \). Dropping the correction miscomputes every entropy and temperature term, and the algorithm often still trains, which is why the bug ships. The stable form via softplus is not optional either. The naive form underflows to \( -\infty \) in float32.

"DPO is RLHF without the RL, so it optimizes the same objective." DPO's derivation is exact only at the optimum of the KL-constrained objective and only on the preference data's distribution. It is an offline method, with no sampling from the evolving policy and no correction of distribution shift, and its Bradley-Terry loss can be reduced by pushing mass off both preferred and rejected completions. Empirically strong pipelines treat DPO-family and online-RL stages as complements, not substitutes.

"GRPO removed the baseline along with the value network." It replaced a learned baseline with an empirical one. The group mean is a per-prompt Monte Carlo estimate of \( V(x) \), and dividing by the group standard deviation is an additional, biasing choice (the Dr. GRPO critique), not a requirement of unbiasedness. REINFORCE's baseline theory from this page applies verbatim. RLOO's leave-one-out mean is the exactly-unbiased version.

Self-check

References

1. Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018. incompleteideas.net/book
2. Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994. Bertsekas, D. P. Dynamic Programming and Optimal Control, Vols. I-II, Athena Scientific, 4th ed., 2012/2017.
3. Agarwal, A., Jiang, N., Kakade, S., Sun, W. Reinforcement Learning: Theory and Algorithms, monograph, 2022. rltheorybook.github.io
4. Williams, R. J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Machine Learning 8, 1992. doi:10.1007/BF00992696
5. Sutton, R. S., McAllester, D., Singh, S., Mansour, Y. "Policy gradient methods for reinforcement learning with function approximation." NeurIPS 2000. Kakade, S. "A natural policy gradient." NeurIPS 2002. Kakade, S. and Langford, J. "Approximately optimal approximate reinforcement learning." ICML 2002.
6. Schulman, J. et al. "Trust region policy optimization." ICML 2015. arXiv:1502.05477. "High-dimensional continuous control using generalized advantage estimation," 2015. arXiv:1506.02438. "Proximal policy optimization algorithms," 2017. arXiv:1707.06347
7. Mnih, V. et al. "Human-level control through deep reinforcement learning." Nature 518, 2015. doi:10.1038/nature14236. "Asynchronous methods for deep reinforcement learning." ICML 2016.
8. van Hasselt, H., Guez, A., Silver, D. "Deep reinforcement learning with double Q-learning." AAAI 2016. arXiv:1509.06461. Wang, Z. et al. "Dueling network architectures." ICML 2016. Schaul, T. et al. "Prioritized experience replay." ICLR 2016.
9. Bellemare, M., Dabney, W., Munos, R. "A distributional perspective on reinforcement learning." ICML 2017. arXiv:1707.06887. Hessel, M. et al. "Rainbow: combining improvements in deep RL." AAAI 2018.
10. Silver, D. et al. "Deterministic policy gradient algorithms." ICML 2014. Lillicrap, T. et al. "Continuous control with deep reinforcement learning." ICLR 2016. Fujimoto, S., van Hoof, H., Meger, D. "Addressing function approximation error in actor-critic methods." ICML 2018. arXiv:1802.09477
11. Haarnoja, T. et al. "Soft actor-critic." ICML 2018, and "Soft actor-critic algorithms and applications," 2018. arXiv:1812.05905
12. Chua, K. et al. "Deep RL in a handful of trials (PETS)." NeurIPS 2018. Janner, M. et al. "When to trust your model (MBPO)." NeurIPS 2019. Hafner, D. et al. "Dream to control," ICLR 2020. "Mastering diverse domains through world models (DreamerV3)," 2023. arXiv:2301.04104. Schrittwieser, J. et al. "Mastering Atari, Go, chess and shogi by planning with a learned model (MuZero)." Nature 588, 2020.
13. Ross, S. and Bagnell, J. A. "Efficient reductions for imitation learning." AISTATS 2010. Ross, S., Gordon, G., Bagnell, J. A. "A reduction of imitation learning to no-regret online learning (DAgger)." AISTATS 2011. Ziebart, B. et al. "Maximum entropy inverse reinforcement learning." AAAI 2008. Ho, J. and Ermon, S. "Generative adversarial imitation learning." NeurIPS 2016.
14. Kumar, A. et al. "Conservative Q-learning for offline RL." NeurIPS 2020. arXiv:2006.04779. Kostrikov, I., Nair, A., Levine, S. "Offline RL with implicit Q-learning." ICLR 2022. Chen, L. et al. "Decision transformer." NeurIPS 2021. Levine, S. et al. "Offline RL: tutorial, review, and perspectives," 2020. arXiv:2005.01643
15. Bellemare, M. et al. "Unifying count-based exploration and intrinsic motivation." NeurIPS 2016. Pathak, D. et al. "Curiosity-driven exploration by self-supervised prediction." ICML 2017. Burda, Y. et al. "Exploration by random network distillation." ICLR 2019. Ecoffet, A. et al. "First return, then explore." Nature 590, 2021.
16. Christiano, P. et al. "Deep reinforcement learning from human preferences." NeurIPS 2017. Ouyang, L. et al. "Training language models to follow instructions with human feedback." NeurIPS 2022. arXiv:2203.02155
17. Bai, Y. et al. "Training a helpful and harmless assistant with RLHF," 2022. arXiv:2204.05862. "Constitutional AI: harmlessness from AI feedback," 2022. arXiv:2212.08073
18. Rafailov, R. et al. "Direct preference optimization: your language model is secretly a reward model." NeurIPS 2023. arXiv:2305.18290. Azar, M. G. et al. "A general theoretical paradigm (IPO)," 2023. Ethayarajh, K. et al. "KTO," 2024. Meng, Y., Xia, M., Chen, D. "SimPO," 2024. Hong, J. et al. "ORPO," 2024.
19. Shao, Z. et al. "DeepSeekMath: pushing the limits of mathematical reasoning in open language models." 2024. arXiv:2402.03300. DeepSeek-AI. "DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning." 2025. arXiv:2501.12948
20. Gao, L., Schulman, J., Hilton, J. "Scaling laws for reward model overoptimization." ICML 2023. arXiv:2210.10760. Lightman, H. et al. "Let's verify step by step." 2023. arXiv:2305.20050
21. Ahmadian, A. et al. "Back to basics: revisiting REINFORCE-style optimization for RLHF (RLOO)." 2024. Liu, Z. et al. "Understanding R1-Zero-like training (Dr. GRPO)." 2025. Yu, Q. et al. "DAPO: an open-source LLM RL system at scale." 2025. Lambert, N. et al. "Tulu 3: pushing frontiers in open language model post-training." 2024.
22. Engstrom, L., Ilyas, A. et al. "Implementation matters in deep policy gradients." ICLR 2020. arXiv:2005.12729. Andrychowicz, M. et al. "What matters in on-policy reinforcement learning," 2020. Huang, S. et al. "The 37 implementation details of proximal policy optimization." ICLR Blog Track, 2022.
23. Henderson, P. et al. "Deep reinforcement learning that matters." AAAI 2018. arXiv:1709.06560. Agarwal, R. et al. "Deep RL at the edge of the statistical precipice." NeurIPS 2021.
24. Tsitsiklis, J. and Van Roy, B. "An analysis of temporal-difference learning with function approximation." IEEE TAC, 1997. Watkins, C. and Dayan, P. "Q-learning." Machine Learning 8, 1992.
25. Kwon, W. et al. "Efficient memory management for LLM serving with PagedAttention (vLLM)." SOSP 2023. Sheng, G. et al. "HybridFlow: a flexible and efficient RLHF framework (verl)." EuroSys 2025. arXiv:2409.19256