Direct preference optimization

What it is and when you reach for it

Direct Preference Optimization (Rafailov et al., 2023) is an offline method for tuning a language model on human preference data. The data is the same as classic RLHF's: triples of a prompt x, a preferred response y_w, and a rejected response y_l, gathered by showing annotators (or a judge model) two candidate responses and asking which is better. Classic RLHF spends that data in two stages: fit a reward model to the preferences, then optimize the policy against it with PPO or a cousin like GRPO, generating fresh samples the whole way. DPO collapses the two stages into one supervised-looking step: a loss computed from four log-probabilities per pair (the policy and a frozen reference, each evaluated on the chosen and rejected responses), trained with ordinary gradient descent on a fixed dataset. Its title says the thesis out loud: your language model is secretly a reward model.

You reach for DPO when you have preference pairs and want the improvement without standing up an RL system: no reward-model training run, no generation engine in the training loop, no value network, no KL controller. The cost profile is that of fine-tuning with a second frozen model in memory, and the whole thing fits in a few dozen lines. What you give up is everything online RL buys: the model never generates during training, so it never explores beyond the dataset and never gets feedback on its own current behavior. That trade, cheap and stable but strictly offline, is why DPO settled into a specific slot in modern pipelines, after SFT and often before online RL, rather than replacing RL outright.

The math

The objective DPO inherits

Everything starts from the standard KL-constrained RLHF objective. Given a reward function r(x, y) and a reference policy π_ref (in practice the SFT model), find

max_π E_{x ~ D, y ~ π(·|x)}[ r(x, y) ] − β D_KL[ π(·|x) ‖ π_ref(·|x) ].

The KL term is not decoration. The reward model is only trustworthy near the distribution it was trained on, and unconstrained reward maximization walks straight out of that region into degenerate high-reward text; β prices how much probability mass the policy may move away from the reference. Every serious RLHF system optimizes some version of this penalized objective, which is exactly what makes the next step more than a textbook exercise: DPO solves the objective people actually use, not a simplified stand-in.

Step 1: the optimal policy in closed form

Fix one prompt x and expand the KL as an expectation under π:

max_π E_{y ~ π}[ r(x, y) − β log ( π(y|x) / π_ref(y|x) ) ].

Divide by −β (flipping max to min) and fold r inside the log:

min_π E_{y ~ π}[ log ( π(y|x) / ( π_ref(y|x) e^r(x,y)/β ) ) ].

The denominator is almost a probability distribution; it just is not normalized. Define the partition function Z(x) = Σ_y π_ref(y|x) e^r(x,y)/β and the distribution

π^*(y|x) = (1/Z(x)) π_ref(y|x) e^r(x,y)/β.

Multiplying and dividing by Z(x) inside the log splits the objective into min_π D_KL[ π(·|x) ‖ π^*(·|x) ] − log Z(x). The second term does not depend on π, and a KL divergence is minimized, at zero, exactly when the two distributions are equal. So the optimum of the RLHF objective is π^*: the optimal policy is the reference policy reweighted by exponentiated reward, softened by β. Small β sharpens toward pure reward maximization; large β stays glued to the reference. Nothing about neural networks was used; this holds for any r and any β > 0.

Step 2: invert it, so the policy defines a reward

Take logs of the closed form and solve for the reward:

r(x, y) = β log ( π^*(y|x) / π_ref(y|x) ) + β log Z(x).

This is the pivot of the whole paper. Every reward function corresponds to an optimal policy, and reading the equation backward, every policy can be interpreted as optimal for some reward, namely β times the log-ratio of that policy to the reference, up to a per-prompt constant β log Z(x). The constant looks like an obstruction, Z(x) is a sum over all possible responses and is hopeless to compute, but the next step makes it vanish.

Step 3: the Bradley-Terry likelihood eats the partition function

Preference data does not label absolute rewards; it labels comparisons. The standard model for turning latent scores into choice probabilities is Bradley-Terry:

p(y_w ≻ y_l | x) = σ( r(x, y_w) − r(x, y_l) ),

with σ the logistic function; this is exactly the model RLHF's reward-model stage fits by maximum likelihood. Substitute the inverted reward from step 2 for both responses. They share the same prompt, so both carry the identical β log Z(x) term, and the difference of rewards cancels the intractable partition function exactly:

p(y_w ≻ y_l | x) = σ( β log ( π^*(y_w|x) / π_ref(y_w|x) ) − β log ( π^*(y_l|x) / π_ref(y_l|x) ) ).

The preference probability now involves only the optimal policy and the reference, both of which assign computable probabilities to any given text (sum the per-token log-probs). Replace the unknown π^* with a parameterized π_θ and fit it by maximum likelihood on the preference dataset, and you get the DPO loss:

L_DPO(θ) = −E_{(x, yw, yl) ~ D} [ log σ( β log ( π_θ(y_w|x) / π_ref(y_w|x) ) − β log ( π_θ(y_l|x) / π_ref(y_l|x) ) ) ].

This is logistic regression where the logit is a difference of log-probability ratios. The quantity r̂_θ(x, y) = β log ( π_θ(y|x) / π_ref(y|x) ) is called the implicit reward: DPO is literally training the reward model of classic RLHF, except the reward model is parameterized through the policy itself, so that when the likelihood is maximized the policy is already the KL-constrained optimum for the reward it learned. The two RLHF stages did not get skipped; they got composed into one loss.

The gradient makes the mechanics concrete. Differentiating gives, per example,

∇_θL = −β · σ( r̂_θ(x, y_l) − r̂_θ(x, y_w) ) · [ ∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x) ].

Push up the chosen response's log-probability, push down the rejected one's, scaled by how wrong the implicit reward currently ranks the pair: examples the model already orders correctly contribute almost nothing, examples it orders backward dominate. That adaptive weighting is what a naive "maximize chosen minus rejected likelihood" loss lacks, and the paper shows it falls out of the RLHF objective rather than being a heuristic.

A worked pair

Let β = 0.1. Suppose the summed token log-probs are: policy on chosen −10, reference on chosen −12, policy on rejected −20, reference on rejected −18. Implicit rewards: r̂_w = 0.1 · (−10 − (−12)) = +0.2 and r̂_l = 0.1 · (−20 − (−18)) = −0.2. The margin is 0.2 − (−0.2) = 0.4, the modeled preference probability is σ(0.4) ≈ 0.599, and the loss is −log 0.599 ≈ 0.513. If instead the policy still equaled the reference everywhere, both implicit rewards would be 0, the margin 0, σ(0) = 0.5, and the loss log 2 ≈ 0.693: every DPO run starts at 0.693 by construction, and a first-batch loss that is not ≈ log 2 means the policy and reference disagree before training, usually a checkpoint or tokenization bug. Note also what the margin does not say: it compares ratios, not raw likelihoods, so it can grow while both responses' absolute probabilities fall, a real phenomenon discussed under traps.

What it is not, and the variant landscape

DPO is offline. The model never samples during training, so there is no exploration: it can only reshuffle probability among behaviors the dataset exhibits, not discover a better answer no annotator wrote down. And the guarantee it inherits is conditional: the derivation says the optimum of the DPO loss is the RLHF optimum for preferences distributed like the training pairs. As the policy moves, the responses it would actually generate drift away from the fixed dataset, and the loss keeps grinding on stale comparisons; online methods refresh their data every step precisely to avoid this. Empirically, papers and practitioner reports find well-tuned online RL (PPO/GRPO with a reward signal) tends to outperform pure DPO at the top end, while DPO wins decisively on cost and simplicity; both findings come with task and tuning caveats and the gap is an active research topic rather than a settled constant.

The loss also spawned a family. IPO (Azar et al.) replaces the log-sigmoid with a squared loss pulling the margin toward a fixed target, addressing DPO's tendency to grow margins without bound and overfit when preferences are near-deterministic. KTO (Ethayarajh et al.) drops the need for pairs entirely, learning from unpaired thumbs-up/thumbs-down examples with an asymmetric, prospect-theory-inspired weighting. SimPO (Meng et al.) removes the reference model, using the policy's length-normalized average log-probability as the implicit reward plus a target margin, cheaper and reportedly strong, at the price of losing the KL anchor. TRL implements these as one-flag variants of the same trainer, which tells you how much machinery they share.

Implementation, twice

The core loss needs exactly four vectors, the summed response log-probs for chosen and rejected under policy and reference, and β. A helper reduces per-token logits to those sums; the mask is what restricts the sum to response tokens only, excluding the prompt, which is a correctness requirement, not a convention.

import torch
import torch.nn.functional as F

def sequence_logprob(logits, tokens, mask):
    """logits: (B, T, V) next-token logits; tokens: (B, T) realized ids;
    mask: (B, T), 1.0 on response tokens, 0.0 on prompt and padding.
    Returns (B,) summed log-prob of each response given its prompt."""
    logp = F.log_softmax(logits, dim=-1)                       # (B, T, V)
    tok = logp.gather(-1, tokens.unsqueeze(-1)).squeeze(-1)    # (B, T)
    return (tok * mask).sum(-1)                                # (B,)

def dpo_loss(policy_chosen_logp, policy_rejected_logp,
             ref_chosen_logp, ref_rejected_logp, beta=0.1):
    """All inputs (B,): summed response log-probs. Returns
    (scalar loss, chosen implicit rewards, rejected implicit rewards)."""
    chosen_rw = beta * (policy_chosen_logp - ref_chosen_logp)      # (B,)
    rejected_rw = beta * (policy_rejected_logp - ref_rejected_logp)
    margin = chosen_rw - rejected_rw                               # (B,)
    # -log sigmoid(margin): logistic regression on the reward margin.
    loss = -F.logsigmoid(margin).mean()
    return loss, chosen_rw.detach(), rejected_rw.detach()

import jax
import jax.numpy as jnp

def sequence_logprob(logits, tokens, mask):
    """logits: (B, T, V) next-token logits; tokens: (B, T) realized ids;
    mask: (B, T), 1.0 on response tokens, 0.0 on prompt and padding.
    Returns (B,) summed log-prob of each response given its prompt."""
    logp = jax.nn.log_softmax(logits, axis=-1)                 # (B, T, V)
    tok = jnp.take_along_axis(
        logp, tokens[..., None], axis=-1)[..., 0]              # (B, T)
    return (tok * mask).sum(-1)                                # (B,)

def dpo_loss(policy_chosen_logp, policy_rejected_logp,
             ref_chosen_logp, ref_rejected_logp, beta=0.1):
    """All inputs (B,): summed response log-probs. Returns
    (scalar loss, chosen implicit rewards, rejected implicit rewards)."""
    chosen_rw = beta * (policy_chosen_logp - ref_chosen_logp)      # (B,)
    rejected_rw = beta * (policy_rejected_logp - ref_rejected_logp)
    margin = chosen_rw - rejected_rw                               # (B,)
    loss = -jax.nn.log_sigmoid(margin).mean()
    return loss, jax.lax.stop_gradient(chosen_rw), \
           jax.lax.stop_gradient(rejected_rw)

The training step wires two models, the trainable policy and a frozen copy as reference, to that loss. A tiny embedding-and-head language model stands in for the LLM so the whole thing runs on synthetic tensors, but the data flow, two forward passes per model per batch (chosen and rejected) with gradients only through the policy, is exactly production's. There are no rollouts anywhere: this is the entire loop.

import copy
import torch.nn as nn

class TinyLM(nn.Module):
    """Stand-in for a causal LM: embedding -> linear -> vocab logits."""
    def __init__(self, vocab=256, dim=64):
        super().__init__()
        self.emb = nn.Embedding(vocab, dim)
        self.head = nn.Linear(dim, vocab)

    def forward(self, tokens):            # (B, T) -> (B, T, V)
        return self.head(self.emb(tokens))

torch.manual_seed(0)
B, T, vocab = 8, 128, 256

policy = TinyLM(vocab)
ref = copy.deepcopy(policy).requires_grad_(False)   # frozen reference
opt = torch.optim.AdamW(policy.parameters(), lr=5e-5)

# Synthetic preference batch: ids and response masks for both sides.
chosen = torch.randint(vocab, (B, T))
rejected = torch.randint(vocab, (B, T))
c_mask = (torch.arange(T) >= 32).float().expand(B, T)  # prompt = 32 tokens
r_mask = c_mask.clone()

with torch.no_grad():                    # reference is inference-only
    ref_c = sequence_logprob(ref(chosen), chosen, c_mask)
    ref_r = sequence_logprob(ref(rejected), rejected, r_mask)

pol_c = sequence_logprob(policy(chosen), chosen, c_mask)
pol_r = sequence_logprob(policy(rejected), rejected, r_mask)

loss, rw_c, rw_r = dpo_loss(pol_c, pol_r, ref_c, ref_r, beta=0.1)
opt.zero_grad(); loss.backward(); opt.step()

print(float(loss))                       # 0.6931...: policy == ref
print(float((rw_c > rw_r).float().mean()))  # "reward accuracy" metric

def init_lm(key, vocab=256, dim=64):
    ke, kh = jax.random.split(key)
    return {'emb': jax.random.normal(ke, (vocab, dim)) * 0.02,
            'w': jax.random.normal(kh, (dim, vocab)) * 0.02,
            'b': jnp.zeros(vocab)}

def forward(params, tokens):              # (B, T) -> (B, T, V)
    return params['emb'][tokens] @ params['w'] + params['b']

key = jax.random.PRNGKey(0)
B, T, vocab = 8, 128, 256
k1, k2, k3 = jax.random.split(key, 3)

params = init_lm(k1, vocab)
ref_params = jax.tree_util.tree_map(jnp.copy, params)   # frozen reference

chosen = jax.random.randint(k2, (B, T), 0, vocab)
rejected = jax.random.randint(k3, (B, T), 0, vocab)
c_mask = jnp.broadcast_to(
    (jnp.arange(T) >= 32).astype(jnp.float32), (B, T))  # prompt = 32 tokens
r_mask = c_mask

ref_c = sequence_logprob(forward(ref_params, chosen), chosen, c_mask)
ref_r = sequence_logprob(forward(ref_params, rejected), rejected, r_mask)

@jax.jit
def train_step(params, lr=5e-5):
    def loss_fn(p):
        pol_c = sequence_logprob(forward(p, chosen), chosen, c_mask)
        pol_r = sequence_logprob(forward(p, rejected), rejected, r_mask)
        loss, _, _ = dpo_loss(pol_c, pol_r, ref_c, ref_r, beta=0.1)
        return loss
    loss, grads = jax.value_and_grad(loss_fn)(params)
    params = jax.tree_util.tree_map(lambda p, g: p - lr * g, params, grads)
    return params, loss

params, loss = train_step(params)
print(float(loss))                        # 0.6931...: policy == ref

Using it on a real shape of problem

A realistic DPO batch on a 7B model looks like B = 8 to 64 pairs of sequences of 512 to 4096 tokens over a ~32k to 128k vocabulary, which means the (B, T, V) logits, computed four times per step (policy and reference, chosen and rejected), dominate memory; this is why implementations concatenate chosen and rejected into one forward pass and why TRL offers precomputing all reference log-probs once up front so the reference model can be dropped from memory entirely. Run the synthetic loop above for a few hundred steps with made-up "preferences" and you will see the three numbers every real run is judged by: the loss falling from 0.693, the reward accuracy (fraction of pairs where the implicit chosen reward exceeds the rejected one) climbing from 0.5 toward the ceiling the data admits, and the reward margin growing. On real runs, expect epochs to matter a lot (1 to 3 is typical, more overfits visibly), expect β in the 0.05 to 0.5 range with 0.1 the common default, and expect the margins to keep growing long after actual output quality has plateaued, which is why held-out win-rate judged by humans or a strong model, not the training metrics, is the acceptance test. Exact curves are dataset- and hardware-dependent; the directions are not.

Applications

DPO's slot in production pipelines is the middle of post-training. The canonical public recipe is Zephyr-7B from Hugging Face's H4 team: SFT on strong instruction data, then DPO on judge-model-ranked preference pairs (UltraFeedback), which at the time beat much larger chat models and made "SFT then DPO" the default open-source alignment recipe. Meta's Llama 3 post-training ran iterative rounds of SFT plus DPO on preference data at frontier scale, an existence proof that the method holds up far beyond 7B. Allen AI's Tülu 3 is the clearest published version of the full modern staircase: SFT, then DPO, then online RL with verifiable rewards as a final stage. The logic of that ordering is cost-shaped: SFT teaches format and competence cheaply, DPO spends a fixed preference dataset to fix style, helpfulness, and refusal behavior without any generation infrastructure, and the expensive online RL budget (PPO or GRPO with rollouts) is saved for the last stretch where exploration and fresh feedback actually pay, reasoning accuracy and reward signals a static dataset cannot express. Beyond chat alignment, the same loss tunes models from AI-generated feedback (RLAIF-style pairs ranked by a judge model), and diffusion variants apply the identical math to image models, but text preference tuning remains the home turf.

Against the real libraries

TRL's DPOTrainer (my notes on the library at /rl/trl) is the standard implementation, and the distance between it and this page is instructive. The loss line is the same; around it TRL adds the things a real run needs: correct chat-template tokenization of prompt/chosen/rejected with the loss masked to response tokens, automatic creation of the frozen reference (or PEFT-based training where the reference is just the adapter turned off, halving memory), optional one-time precomputation of reference log-probs, distributed training, and logged reward-margin and reward-accuracy metrics. Its loss_type flag is a tour of the variant literature: "sigmoid" is the DPO loss above, "ipo" the squared-loss variant, plus hinge, robust, and a dozen research losses, with β reinterpreted per variant as documented. The second reference point is eric-mitchell/direct-preference-optimization, the original authors' repo: less production machinery, but the cleanest statement of the loss in the wild (its preference_loss function is essentially the PyTorch function above) and the code against which the paper's results were produced, which makes it the right cross-check for research forks.

The from-scratch version is genuinely enough when you control tokenization and just need the loss inside an existing training loop, which is why so many labs' internal trainers contain a twenty-line DPO rather than a TRL dependency. Verification is pleasantly deterministic because everything is offline: feed the same four log-prob tensors to your dpo_loss and to TRL's loss computation with loss_type="sigmoid" and the same β, and the scalars must agree to float tolerance (~1e-6); independently, assert the closed-form anchors, loss exactly log 2 and zero margins when policy equals reference, and loss decreasing when you add any positive constant to the chosen log-probs. If those pass and a real run still misbehaves, the bug is almost always in masking or tokenization, not the loss.

Traps and misconceptions

"DPO has no reward model." It has one; you just do not train it separately. The implicit reward β log(π_θ/π_ref) is a real reward model, fit by the same Bradley-Terry likelihood as RLHF's, and it inherits the same failure modes: it can overfit annotator quirks, it can be probed and gamed, and its margins are trustworthy only near the training distribution. The saved cost is infrastructural, not epistemic.

Summing versus masking log-probs. The loss needs the log-probability of the response given the prompt. Including prompt tokens in the sum, or forgetting to mask padding, silently changes every margin, and because chosen and rejected prompts are identical the bug partially cancels, making it look like a hyperparameter problem instead of a correctness one. The first-batch loss ≈ log 2 check catches many of these; length statistics of chosen versus rejected catch the rest.

Reading β backwards. β multiplies the log-ratio, but its role comes from the RLHF objective: it is the KL price. Small β means weak anchoring, large policy drift, and sharper fitting of the preferences; large β keeps the model close to the reference and mutes the update. People routinely assume the opposite because β "scales the reward". When outputs degrade into repetitive or unnatural text, the usual fix is a larger β or fewer epochs, not more data.

Margins up, likelihoods down. The loss only constrains the difference of ratios, so a well-documented training dynamic is both chosen and rejected log-probabilities falling while the margin grows, with probability mass leaking to sequences that appear in no pairs. Some of that is benign reallocation; enough of it degrades generation. This is a real limitation of the offline objective, it motivated variants like IPO and auxiliary SFT-loss terms, and it is why generation-based evaluation is mandatory even though training never generates.

Treating DPO as a drop-in replacement for online RL. The derivation's optimum coincides with RLHF's only under the training distribution of pairs; nothing corrects the policy on the distribution it actually induces after moving. For verifiable objectives (math, code) where rewards can be checked on fresh samples, online methods like GRPO exploit information DPO structurally cannot see. The mature reading of the evidence is placement, not ranking: DPO where a fixed preference set is the resource, online RL where rollouts and a checkable reward are.