TRL

Part I: The mental model

your data (prompts, preferences, completions)
      |
      v
trl/data_utils.py         apply_chat_template, is_conversational, pack_dataset
      |
      v
one TRL trainer           SFTTrainer | DPOTrainer | GRPOTrainer
      |                    RewardTrainer | KTOTrainer | RLOOTrainer
      |                    each = Trainer + {*Config, dataset prep, compute_loss}
      v
transformers.Trainer      the loop: accumulate, backward, clip, step, log, save
      |
      v
accelerate                DDP, FSDP, DeepSpeed, mixed precision, device placement
      |
      v
torch                     one or many GPUs

wrapped from above by: Axolotl, LLaMA-Factory, Unsloth, autotrain-advanced

The one-sentence identity, TRL is a collection of loss functions and dataset adapters bolted onto the Hugging Face Trainer, so that each RLHF and preference-tuning algorithm becomes a small, readable subclass rather than a new training framework. The Trainer already knows how to run a distributed loop, accumulate gradients, apply mixed precision, clip, step an optimizer and scheduler, log metrics, and write checkpoints. TRL contributes the parts that differ between algorithms and nothing else. A TRL trainer is the base Trainer with three replacements, a Config dataclass that extends TrainingArguments with algorithm knobs like beta or num_generations, a dataset preparation step that turns raw preference or prompt data into model-ready batches, and a compute_loss that encodes the algorithm's objective.

Two consequences follow. First, this is why TRL is the layer others build on. Axolotl, LLaMA-Factory, Unsloth, and autotrain-advanced are configuration front ends that call TRL trainers underneath, because TRL already owns the tricky loss math and delegates the boring, well-tested loop to the Trainer. Learn TRL and you have learned the engine inside all of them. Second, reading TRL is the shortest path to understanding what DPO, GRPO, PPO, and reward modeling actually compute, because each algorithm is isolated in one file whose center of gravity is a single loss method. Everything in this chapter is verified against the main branch in July 2026. TRL tracks the fast-moving post-training literature and refactors often, so where a detail is likely to shift I say so and stay at concept level. One such shift already happened, the current main promotes six trainers to first-class status in trl/trainer/, SFT, DPO, GRPO, KTO, RLOO, and Reward, while PPO and a long tail of research methods now live under trl/experimental/.

Part II: Using it

TRL is a normal Python package. It runs anywhere PyTorch and transformers run, including a laptop for the small models, and it uses whatever GPUs Accelerate finds. Install it with pip, optionally with the extras for parameter-efficient fine-tuning and fast generation:

pip install trl
# optional companions used below
pip install peft            # LoRA and QLoRA adapters
pip install vllm            # fast generation for the online methods (GRPO, PPO, RLOO)

The smallest useful program is supervised fine-tuning. Pass a model by name, a dataset, and let the defaults do the rest. This is the canonical quickstart from the project's own README:

from datasets import load_dataset
from trl import SFTTrainer

dataset = load_dataset("trl-lib/Capybara", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,
)
trainer.train()

That is the whole thing. SFTTrainer resolves the model string to an AutoModelForCausalLM, loads the matching tokenizer, detects that trl-lib/Capybara is a conversational dataset and applies the model's chat template, tokenizes, and then hands off to the inherited Trainer loop. To control anything, pass an SFTConfig, which is a TrainingArguments with a few extra fields. Note the argument is named processing_class, not tokenizer, because recent transformers renamed it to cover tokenizers and multimodal processors alike:

from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,
    args=SFTConfig(
        output_dir="Qwen2.5-0.5B-SFT",
        per_device_train_batch_size=8,
        gradient_accumulation_steps=2,
        learning_rate=2e-5,
        num_train_epochs=1,
        packing=True,              # concatenate short samples to fill the context
        max_length=2048,
    ),
)
trainer.train()

Preference tuning is the same shape. Direct Preference Optimization needs a dataset of triples, a prompt, a chosen answer, and a rejected answer, and it needs a reference model, which TRL will make for you by freezing a copy of the policy if you do not pass one:

from datasets import load_dataset
from trl import DPOConfig, DPOTrainer

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
    args=DPOConfig(output_dir="Qwen3-0.6B-DPO", beta=0.1),
)
trainer.train()

Online reinforcement learning with a reward comes through GRPOTrainer. Here you do not supply preferences, you supply a reward function, an ordinary Python callable that scores completions. TRL ships some ready made ones in trl.rewards, and writing your own is trivial:

from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

dataset = load_dataset("trl-lib/tldr", split="train")

# a reward is just: (completions, **kwargs) -> list[float], one score per completion
def reward_short_answers(completions, **kwargs):
    return [-abs(50 - len(c)) for c in completions]   # prefer ~50-char answers

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=reward_short_answers,
    train_dataset=dataset,
    args=GRPOConfig(output_dir="Qwen2.5-0.5B-GRPO", num_generations=8),
)
trainer.train()

Every trainer also has a command-line front end. The trl CLI reads the same config fields as flags, so a reproducible run needs no Python file at all:

trl sft  --model_name_or_path Qwen/Qwen2.5-0.5B \
         --dataset_name trl-lib/Capybara \
         --output_dir Qwen2.5-0.5B-SFT

trl dpo  --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
         --dataset_name argilla/Capybara-Preferences \
         --output_dir Qwen2.5-0.5B-DPO

trl env                     # dumps versions and hardware for bug reports
trl vllm-serve --model ...  # stand up a generation server for online methods

Now the mistakes beginners make. First, dataset format. TRL expects one of two shapes, standard (plain text columns like prompt, completion, chosen, rejected) or conversational (those same columns holding lists of {"role", "content"} messages). If a preference method sees the wrong columns it fails early, and if a conversational dataset is fed to a method expecting plain text the chat template is silently not applied. The data_utils.py helpers named later exist to make these conversions explicit. Second, for the online methods, num_generations (the group size) must divide the effective batch size, because a whole group of completions for one prompt has to stay together to be normalized. Third, memory, DPO holds two models on the device, the policy and the reference, so a model that just fit for SFT may not fit for DPO, which is exactly why LoRA is popular here, the reference is the same base weights with the adapter switched off, so there is no second copy. Fourth, and most fundamental, do not reach for an RL trainer when a preference trainer will do. If you already have preference pairs, DPO trains directly on them with no sampling, no reward model, and no rollouts, and it is almost always the cheaper and more stable first move.

Part III: When it is the right tool

TRL is the right tool when you are post-training a language model on Hugging Face infrastructure and want a small, forkable, honest implementation of a specific algorithm, SFT to teach a format, reward modeling to fit a preference signal, DPO or KTO to align on preferences directly, GRPO or PPO or RLOO to optimize against a reward with on-policy sampling. It is the reference these algorithms are read from, it lands new methods from the literature quickly, and it composes cleanly with the rest of the stack, PEFT for adapters, bitsandbytes for quantization, Accelerate for distribution, and vLLM for the generation-heavy online methods.

The honest cases for something else. If your job is plain supervised fine-tuning with no preference or RL component, the bare transformers Trainer already does it and TRL adds little beyond convenience formatting. If you want a configuration-first experience with a hundred recipes and no Python, Axolotl or LLaMA-Factory wrap TRL and give you that, Unsloth does the same with hand-optimized kernels for single-GPU speed, and autotrain-advanced turns it into a hosted job. If you are running large-scale RL across many nodes with heavy rollout throughput, the systems built for that, OpenRLHF and verl, separate generation and training into distinct actor pools with Ray and squeeze more out of a big cluster than TRL's single-controller design targets. TRL deliberately optimizes for legibility and breadth of methods over last-mile cluster throughput. And trlx, the older CarperAI RLHF library, is effectively superseded, its ideas live on here and in the systems above.

The domain-shaped warning is about the reward, not the code. Every online method here is a faithful optimizer, and a faithful optimizer of a bad reward produces a confidently bad model. The classic failure is reward hacking, the policy discovers that padding an answer, repeating a keyword, or emitting a fixed format scores high without being better, and GRPO or PPO will find that exploit reliably because finding exploits is precisely what they do. This is the same lesson the deep reinforcement learning material returns to again and again, specification is the hard part, and it is the reason DPO's appeal is partly that its reward is pinned to a reference policy rather than to a learned proxy that can be gamed.

Part IV: The full life of one DPO training step

The specimen, one step of Direct Preference Optimization, the method whose closed-form reward is the single most load-bearing idea in the library. Follow it once and the shape of every other preference trainer becomes obvious, because they differ only in the loss expression at the end. The trainer is trl/trainer/dpo_trainer.py and the config is dpo_config.py.

Stage 1: dataset preparation happens once, up front

DPOTrainer.__init__ does the algorithm-specific data work before training starts. For each row of {prompt, chosen, rejected} it applies the chat template when the data is conversational (maybe_apply_chat_template), tokenizes the prompt once and each of the two completions, and records which tokens are prompt (to be masked out of the loss) versus completion. The result is a dataset of tensors that a preference-aware collator can pad into batches. Doing this in __init__ rather than inside the loop means the expensive tokenization is paid once, and it is why a TRL trainer can accept a raw Hub dataset and still feed the inherited loop clean tensors.

Stage 2: the inherited loop calls compute_loss

trainer.train() is not overridden. It is the stock transformers loop, iterating the dataloader, handling gradient accumulation and mixed precision, and calling compute_loss(model, inputs) for each batch. TRL's entire contribution to the step is what happens inside that one method. DPOTrainer.compute_loss delegates to a helper (historically get_batch_loss_metrics) that produces both the scalar loss and a bundle of metrics to log.

Stage 3: one concatenated forward pass

The key efficiency trick lives in concatenated_forward. Rather than run the model twice, once on chosen and once on rejected, it stacks both into a single batch of 2N sequences and runs one forward pass. For each sequence it gathers the log-probability the model assigns to each true next token, masks out the prompt positions, and sums over the completion tokens to get a single sequence log-probability. Split back apart, this yields two numbers per example, policy_chosen_logps and policy_rejected_logps, the policy's total log-likelihood of the preferred and dispreferred answers.

Stage 4: the reference log-probs

DPO measures the policy against a frozen reference. There are three ways to get reference_chosen_logps and reference_rejected_logps, and TRL supports all three. With a separate reference model, a frozen copy made by create_reference_model, run the same concatenated forward through it under no_grad. With a LoRA adapter, skip the copy entirely, disable the adapter on the policy so the base weights act as the reference, forward, then re-enable, which halves the memory. Or precompute the reference log-probs for the whole dataset once before training (precompute_ref_log_probs) and cache them, trading disk and a preprocessing pass for never holding the reference in memory during the loop.

Stage 5: the closed-form reward and the loss

Now the heart of the method. DPO's insight is that you never need to train a reward model at all, because the reward implied by any policy relative to its reference has a closed form, the scaled log-ratio. Define the implicit reward of an answer as r(x, y) = beta * (log pi_theta(y|x) - log pi_ref(y|x)). The loss simply pushes the implicit reward of the chosen answer above that of the rejected one, through a logistic (Bradley-Terry) objective:

pi_logratios  = policy_chosen_logps    - policy_rejected_logps
ref_logratios = reference_chosen_logps - reference_rejected_logps
logits        = pi_logratios - ref_logratios        # difference of implicit rewards / beta
loss          = -logsigmoid(beta * logits)          # loss_type = "sigmoid" (original DPO)

# logged as metrics, the implicit rewards themselves:
chosen_reward   = beta * (policy_chosen_logps   - reference_chosen_logps)
rejected_reward = beta * (policy_rejected_logps - reference_rejected_logps)
reward_accuracy = mean(chosen_reward > rejected_reward)

The partition function that would normally make a reward model necessary cancels, because it depends only on the prompt and appears identically in the chosen and rejected terms, so subtracting them makes it vanish. What survives is a loss you can evaluate from four log-probabilities and one hyperparameter, beta, which sets how hard the policy is pulled away from the reference. That single algebraic cancellation is why DPO needs no reward model, no sampling, and no RL loop. The loss_type field selects variants that keep this structure but change the surrogate, "ipo" replaces the logistic with a squared objective that resists overfitting, "hinge" uses an SVM-style margin, and label_smoothing turns it into the conservative cDPO loss that tolerates noisy preferences. Every one of them is a few lines swapped in at this exact point.

Stage 6: backward, step, and logging, all inherited

compute_loss returns the scalar, and from here TRL does nothing special. The stock Trainer backpropagates, clips the gradient norm, steps AdamW and the scheduler, and, on logging steps, emits the metrics DPO stashed, rewards/chosen, rewards/rejected, rewards/accuracies, rewards/margins, and the raw log-probs. Checkpointing and resumption are the Trainer's too. That closes the life of one step, a preference triple in, two forward passes, one log-ratio subtraction, one logistic loss, and a gradient that raises the chosen answer's likelihood and lowers the rejected one's, exactly in proportion to how wrong the implicit reward ordering currently is.

Part V: Internals deep dives

Deep dive: how every trainer subclasses transformers.Trainer

The base Trainer is a large, battle-tested class that owns the training loop and, crucially, exposes the right override points. A TRL trainer is that class with three parts replaced, and recognizing the pattern lets you read any trainer in the repo in minutes:

Everything else is inherited untouched, the distributed loop, gradient accumulation, mixed precision, gradient clipping, the optimizer and LR scheduler, logging to every backend, and checkpoint save and resume. That is a remarkable amount of correctness to get for free, and it is the whole reason TRL can support a dozen algorithms without a dozen training loops.

There is a second tier of override for the online methods. SFT, DPO, KTO, and reward modeling are offline, their data is fixed, so replacing compute_loss is enough. GRPO, PPO, and RLOO must generate fresh completions from the current policy every step, so they additionally override the input preparation (a method like _prepare_inputs) or the training step to insert a rollout, sample completions, score them with a reward, compute advantages, and only then hand tensors to the loss. The unwrap_model_for_generation context manager from trl/models/ exists precisely for this, it temporarily unwraps the FSDP or DeepSpeed shell so generate can run on a whole model. So the mental taxonomy is, offline methods swap the loss, online methods swap the loss and the sampling, and both inherit the rest.

Deep dive: SFTTrainer, formatting is the feature

SFTTrainer looks trivial, its loss is plain causal language-modeling cross-entropy, which the base Trainer would compute anyway. Its real work is turning messy real datasets into clean token tensors, and that data path is worth knowing. It auto-detects conversational data and applies the tokenizer's chat template so the special tokens match what the model saw in pretraining. It supports packing, concatenating multiple short examples into one full-length sequence so no compute is wasted on padding, using a best-fit-decreasing bin-packing pass over the dataset (the pack_dataset utility) rather than naive concatenation. And it supports completion-only loss, masking the prompt tokens so the model is trained only on the response, plus an assistant-only variant for multi-turn chat that masks every turn except the assistant's. Each of these is a switch on SFTConfig. The lesson is that in practice the correctness of a fine-tune lives more in the masking and templating than in the loss, which is why SFTTrainer spends its code there. It also accepts a peft_config to wrap the model in a LoRA adapter before training, the same one-line path to parameter-efficient fine-tuning that the preference trainers use.

Deep dive: DPOTrainer and the closed-form reward, in full

Part IV traced the mechanics. Here is why the closed-form reward is true, because it is the idea the whole method rests on. Start from the standard RLHF objective, maximize expected reward while staying close to a reference policy in KL:

maximize over pi:   E_{x, y~pi}[ r(x,y) ]  -  beta * KL( pi(.|x) || pi_ref(.|x) )

This problem has a known optimal solution, the reference policy reweighted by the exponentiated reward:

pi*(y|x) = (1 / Z(x)) * pi_ref(y|x) * exp( r(x,y) / beta )

where Z(x) is a normalizing partition function that sums over all possible answers and is hopelessly expensive to compute. Now invert that equation to solve for the reward in terms of the optimal policy:

r(x,y) = beta * ( log pi*(y|x) - log pi_ref(y|x) ) + beta * log Z(x)

The reward is the scaled log-ratio plus a prompt-only term. Substitute this expression into the Bradley-Terry model of a human preference, which says the probability that y_w beats y_l is the logistic of their reward difference. The beta * log Z(x) term is identical for both answers to the same prompt, so it cancels in the difference and disappears. What remains is a probability written entirely in terms of the policy and the reference, and training the policy to maximize its likelihood on preference data is DPO. The intractable partition function never has to be computed because it cancels, and the reward model never has to be trained because the policy itself, measured against its reference, is the reward. That is the entire trick, and it collapses a three-stage RLHF pipeline (fit a reward model, then run PPO against it) into a single supervised-looking loss.

The practical corollaries are worth stating. beta is the only real dial and it controls conservatism, small beta lets the policy move far from the reference and risks reward hacking of the preference data itself, large beta keeps it close and safe but slow. The reference model is not optional, it is what makes the reward well-defined, and a DPO run whose reference has drifted from the SFT checkpoint it was trained on will behave oddly. And because there is no sampling, DPO is stable and cheap but it can only express preferences that are already in the dataset, it cannot discover new behaviors the way an online method exploring with a reward can. That trade, offline stability versus online exploration, is the axis the next two deep dives sit on.

Deep dive: GRPOTrainer, a critic-free policy gradient

Group Relative Policy Optimization, introduced with DeepSeekMath and made famous by DeepSeek-R1, is the method that trained a generation of reasoning models, and TRL's grpo_trainer.py is a widely read implementation. Its premise is a simplification of PPO, get rid of the value model. PPO needs a learned critic to estimate a baseline for the advantage. GRPO replaces that critic with a statistic computed on the fly, sample a group of num_generations completions for the same prompt, score all of them, and use the group's own mean as the baseline:

for each prompt x:
    sample G completions  o_1..o_G ~ pi_theta(.|x)
    score them            r_1..r_G  = reward_funcs(x, o_i)
    advantage of o_i:     A_i = (r_i - mean(r)) / (std(r) + eps)   # group-normalized

objective (per token t in completion i), a clipped policy gradient plus a KL leash:
    ratio  = pi_theta(o_it) / pi_theta_old(o_it)
    L_i    = min( ratio * A_i, clip(ratio, 1-eps, 1+eps) * A_i )  -  beta * KL(pi_theta || pi_ref)

The advantage is just how much better a completion did than its siblings, so a completion above the group average is reinforced and one below is suppressed, with no critic to train. The KL term (an unbiased, always-positive estimator) leashes the policy to a reference, and beta sets its strength, some recipes set it to zero and drop the reference model entirely. Two subtle knobs on GRPOConfig reward close reading. scale_rewards decides whether to divide by the group standard deviation, dividing normalizes scale but, as the Dr. GRPO analysis pointed out, biases toward easy prompts where variance is low, so leaving it off is sometimes better. importance_sampling_level chooses whether the ratio is computed per token or per sequence, the sequence-level choice is GSPO, which stabilizes long-completion training. And num_iterations controls how many gradient steps reuse one batch of rollouts, at one the ratio is one and the clip never bites so GRPO reduces to REINFORCE with a group baseline, above one it becomes genuinely off-policy PPO-style optimization.

The systems reality of GRPO is generation cost. Sampling a group of completions per prompt every step is far more expensive than the gradient update, so TRL integrates vLLM for the rollouts. use_vllm turns it on, and vllm_mode chooses between colocate, running vLLM in the same processes as training and sharing the GPUs, and server, talking to a separate trl vllm-serve process that owns dedicated generation GPUs. Weight synchronization between the training policy and the vLLM engine each step is the fiddly part that mode exists to manage. The reward side is deliberately open, reward_funcs takes any callable, a list of callables combined with reward_weights, or a model name that loads a reward model, so a verifiable reward like a math-answer checker and a learned preference reward can be mixed freely.

Deep dive: PPOTrainer, the value head, and reward modeling

PPO is the original RLHF algorithm and it is still here, though on current main it has moved into trl/experimental/ppo/ alongside its config and its value-head model, a signal that for language-model post-training GRPO and DPO have become the common defaults while PPO is kept for fidelity to the classic pipeline. PPO is the maximal version of the online loop, it needs four models, a policy being trained, a frozen reference for the KL penalty, a reward model that scores complete responses, and a value model (critic) that estimates expected future reward at every token so advantages can be computed with generalized advantage estimation. The value model is where the AutoModelForCausalLMWithValueHead wrapper (now in experimental/ppo/modeling_value_head.py) comes from, it takes a base transformer and adds a small linear head that reads the hidden states and outputs a scalar value per token. Historically the very first TRL PPOTrainer did not subclass the Trainer at all, it exposed a manual ppo_trainer.step(queries, responses, rewards) loop that you drove yourself. The current rewrite brings it back in line with the rest of the library, it subclasses the Trainer, takes explicit policy, reference, reward, and value models, and runs generation and the PPO update inside the standard loop. The honest summary is that PPO is the most powerful and the most finicky option here, four models to hold and a critic to train well, which is exactly the complexity GRPO set out to remove.

Reward modeling is the piece PPO consumes and DPO makes unnecessary, and RewardTrainer is the simplest trainer in the library to read. It fine-tunes an AutoModelForSequenceClassification with a single output (num_labels=1) into a scalar scorer, on the same chosen/rejected preference data DPO uses. Its loss is the Bradley-Terry objective directly, -logsigmoid(reward_chosen - reward_rejected), pushing the chosen response's score above the rejected one's. An optional per-example margin lets stronger preferences demand a larger gap, and an optional centering term keeps the raw scores from drifting. That is the whole method, one forward pass on each side of the pair and a logistic loss on the difference. Reading RewardTrainer and DPOTrainer back to back is the clearest way to see what DPO bought, the reward trainer fits a scorer you then optimize against with PPO, and DPO folds those two stages into one by making the policy its own reward.

Part VI: Reading the repository

The tree is organized by concept and is comfortable to read end to end. All paths verified on main, July 2026.

Stage 0, orientation. Read the README.md and the top-level trl/__init__.py, which lists the public API, the six first-class trainers and their configs, plus helpers like get_peft_config. Question, which names are exported at the top level, and which have been relegated to trl/experimental/?

Stage 1, the data contract. Read trl/data_utils.py. The functions there, is_conversational, apply_chat_template and maybe_apply_chat_template, extract_prompt, unpair_preference_dataset, pack_dataset, and maybe_convert_to_chatml, define the standard and conversational dataset formats every trainer assumes. Question, what exactly distinguishes a standard dataset from a conversational one, and where does the chat template get applied?

Stage 2, the simplest trainer. Read trl/trainer/reward_trainer.py with reward_config.py beside it. It is short and shows the subclass pattern in its purest form, a config extending TrainingArguments, a collator, and a compute_loss that is one logistic on a score difference. Question, what are the three things it replaces on the base Trainer, and what does it inherit?

Stage 3, the preference workhorse. Read trl/trainer/dpo_trainer.py, aiming for concatenated_forward, the reference-log-prob handling, and the loss block. Then skim kto_trainer.py to see the same skeleton with a different loss (KTO needs only a per-example thumbs-up or thumbs-down, not pairs). Question, how does the trainer get the reference log-probs in the PEFT case without a second model?

Stage 4, the online loop. Read trl/trainer/grpo_trainer.py and grpo_config.py. This is the longest and richest trainer, follow how a batch of prompts becomes a batch of scored, advantage-weighted completions before the loss, and where vLLM plugs in. Then read rloo_trainer.py for the leave-one-out baseline variant. Question, why must num_generations divide the batch, and what does num_iterations greater than one change?

Stage 5, the classic pipeline and the frontier. trl/experimental/ppo/ for the four-model PPO loop and modeling_value_head.py, then browse trl/experimental/ broadly, it holds ORPO, CPO, BCO, Online DPO, Nash-MD, XPO, GKD, PRM, and a stream of research variants. Also read trl/rewards/ for the ready-made reward functions and trl/models/utils.py for create_reference_model and unwrap_model_for_generation.

Where not to start, the experimental/ tree is by definition unstable and its methods move in and out, and the multimodal and vLLM server plumbing is best met after the dense single-modal, single-process story is solid.

Part VII: Hands-on labs

Labs 1 and 2 run on a single small GPU (or slowly on CPU). Labs 3 through 5 want a GPU with room for generation. Log formats shift with the fast pace of main.

Lab 1: SFT a 0.5B model and read the masks. Concept: the data path is the feature.

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

ds = load_dataset("trl-lib/Capybara", split="train[:2000]")
trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=ds,
    args=SFTConfig(output_dir="sft-lab", max_length=1024,
                   completion_only_loss=True, report_to="none"),
)
# inspect one prepared batch before training:
batch = next(iter(trainer.get_train_dataloader()))
print({k: v.shape for k, v in batch.items()})
print("masked (prompt) label positions:", (batch["labels"] == -100).sum().item())
trainer.train()

Observe that the prompt tokens carry a label of -100, the ignore index, so loss is computed only on the completion. Toggle completion_only_loss off and watch that count fall to the padding tokens alone.

Lab 2: DPO and the implicit reward. Concept: the closed-form reward as a logged metric.

from datasets import load_dataset
from trl import DPOConfig, DPOTrainer

ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train[:2000]")
trainer = DPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=ds,
    args=DPOConfig(output_dir="dpo-lab", beta=0.1,
                   logging_steps=5, report_to="none"),
)
trainer.train()

Watch rewards/chosen, rewards/rejected, and rewards/accuracies in the logs. Early on the accuracy hovers near 0.5, the policy has no preference, and it climbs as the chosen implicit reward pulls above the rejected one. Rerun with beta=0.5 and see the rewards move less, the policy is held closer to its reference.

Lab 3: GRPO with a reward you can read. Concept: group-normalized advantage.

from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

ds = load_dataset("trl-lib/tldr", split="train[:1000]")

def reward_unique_chars(completions, **kwargs):
    # a transparent, gameable reward: more distinct characters scores higher
    return [float(len(set(c))) for c in completions]

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=reward_unique_chars,
    train_dataset=ds,
    args=GRPOConfig(output_dir="grpo-lab", num_generations=8,
                    per_device_train_batch_size=8, logging_steps=1,
                    report_to="none"),
)
trainer.train()

Watch the mean reward rise, then read a few completions. Because the reward is gameable, the model will learn to stuff in unusual characters, a live demonstration of reward hacking. Swap in a real reward from trl.rewards and compare.

Lab 4: reward model, then see what DPO folded away. Concept: the two-stage pipeline DPO collapses.

from datasets import load_dataset
from trl import RewardConfig, RewardTrainer

ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train[:2000]")
trainer = RewardTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",   # loaded as a 1-logit sequence classifier
    train_dataset=ds,
    args=RewardConfig(output_dir="rm-lab", report_to="none"),
)
trainer.train()

This trains the scorer that PPO would optimize against. Note it uses the same dataset DPO used, then reflect, DPO's contribution is deleting this entire stage by turning the policy into its own reward.

Lab 5: the CLI, no Python. Concept: config fields are flags.

trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
        --dataset_name trl-lib/Capybara \
        --output_dir sft-cli --max_length 1024 --packing \
        --per_device_train_batch_size 8 --report_to none

trl env    # capture the exact versions for a reproducible record

Every flag maps to a field on SFTConfig. Change sft to dpo or grpo and the valid flags change with the config class, which is the CLI's whole design.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is TRL, in one sentence?

A library of post-training algorithms implemented as thin subclasses of the transformers Trainer, where each trainer replaces only the config, the dataset preparation, and the loss, and inherits the entire training loop.

2. What three things does a TRL trainer replace on the base Trainer, and what does it inherit?

It replaces the config (a dataclass extending TrainingArguments), the dataset preparation, and compute_loss. It inherits the distributed loop, gradient accumulation, mixed precision, clipping, the optimizer and scheduler, logging, and checkpointing.

3. Why does DPO need no reward model?

Because the reward implied by a policy relative to its reference has a closed form, the scaled log-ratio beta * (log pi_theta - log pi_ref). Substituted into the Bradley-Terry preference model, the intractable partition function cancels between the chosen and rejected terms, leaving a loss computable from four log-probabilities. The policy is its own reward.

4. What is the closed-form DPO loss?

-logsigmoid( beta * [ (log pi_theta(y_w) - log pi_ref(y_w)) - (log pi_theta(y_l) - log pi_ref(y_l)) ] ), the negative log-logistic of beta times the difference between the chosen and rejected implicit rewards.

5. What does the concatenated forward pass buy DPO?

One forward pass instead of two. Chosen and rejected sequences are stacked into a single batch, run together, then split apart to get the four log-probabilities. It halves the number of forward calls per step for the policy and again for the reference.

6. How does GRPO avoid PPO's value model?

It samples a group of completions per prompt and uses the group's mean reward as the baseline, so the advantage of a completion is how far its reward sits from the group average, optionally divided by the group standard deviation. No learned critic is needed.

7. Why must num_generations divide the batch size?

Because the advantage is computed within a group. All completions for one prompt must be present together to compute the group mean and standard deviation, so a group cannot be split across batches.

8. Which methods are offline and which are online, and why does it matter for the trainer?

SFT, reward modeling, DPO, and KTO are offline, their data is fixed, so they only override the loss. GRPO, PPO, and RLOO are online, they generate completions from the current policy each step, so they additionally override the input or step preparation to sample, score, and compute advantages before the loss runs.

9. What four models does PPO hold, and what is each for?

A policy being trained, a frozen reference for the KL penalty, a reward model that scores complete responses, and a value model (critic) that estimates expected future reward per token for the advantage estimate. GRPO removes the fourth.

10. What is the value head, and where does it live now?

A small linear layer added on top of a base transformer's hidden states that outputs a scalar value per token, wrapped by AutoModelForCausalLMWithValueHead. On current main it sits in trl/experimental/ppo/modeling_value_head.py, since PPO is its main consumer and PPO moved to experimental.

11. When would you pick GRPO over DPO, or DPO over GRPO?

DPO when you already have preference pairs and want a stable, cheap, sampling-free run that cannot easily be gamed. GRPO when you have a reward you can evaluate on fresh generations (a verifier, a scorer) and want the policy to explore and discover behaviors that are not present in any fixed dataset.

12. When would you reach past TRL to OpenRLHF or verl?

When large-scale online RL throughput is the bottleneck and you want generation and training split into separate actor pools across many nodes, typically orchestrated with Ray. TRL optimizes for legibility and breadth of methods on a single controller rather than last-mile cluster throughput.

13. What is reward hacking, and which methods are exposed to it?

The policy maximizing the letter of a reward while violating its intent, padding, repetition, format tricks. Every online method that optimizes a reward (GRPO, PPO, RLOO) is exposed, because finding such exploits is exactly what they do. DPO is less exposed because its reward is tied to a reference policy rather than a learned proxy, though it can still overfit its preference data.

14. Why is TRL described as the primitive layer others wrap?

Because the configuration-first front ends, Axolotl, LLaMA-Factory, Unsloth, autotrain-advanced, call TRL trainers underneath rather than reimplementing the algorithms. TRL owns the loss math and delegates the loop to the transformers Trainer, and the wrappers add recipes and UX on top.

Part IX: Design lessons

Subclass the loop you already trust. Rather than write a training loop per algorithm, TRL inherits one battle-tested loop and injects only the loss and the data. This is the template-method pattern at library scale, and it is why a dozen algorithms share one set of correctness guarantees for distribution, checkpointing, and mixed precision.

Reparameterize to delete a stage. DPO's whole value is algebraic, by writing the reward as a function of the policy it deletes the reward-model and RL stages entirely. The general move, whenever a pipeline has a stage whose output is determined by a later stage's parameters, ask whether the later stage can be expressed to make the earlier one vanish.

Replace a learned component with a statistic when you can. GRPO swaps PPO's trained critic for a group mean computed on the fly. A statistic needs no training, cannot itself be wrong, and removes a whole model from memory. Reach for a cheap empirical baseline before a learned one.

Make the data contract explicit and small. Two dataset formats, standard and conversational, with named columns and a handful of conversion utilities, mean every trainer agrees on what a dataset is. A small, enforced contract at the boundary is worth more than flexibility that each trainer interprets differently.

Keep the frontier quarantined. Six trainers are first-class and stable in trl/trainer/, and the stream of research methods lives behind trl/experimental/ where instability is expected. Separating the supported core from the moving frontier lets a fast-moving project stay both current and dependable.

Part X: Memorization framework

The one-sentence summary, TRL implements each post-training algorithm as a transformers Trainer subclass that swaps in a config, a dataset step, and a loss, so DPO becomes a log-ratio logistic, GRPO becomes a group-normalized policy gradient, reward modeling becomes a Bradley-Terry score, and the whole training loop is inherited.

raw data -> data_utils (template, pack) -> a TRL trainer
  trainer = transformers.Trainer + {*Config, dataset prep, compute_loss}
  offline (SFT, DPO, KTO, Reward): swap the loss
  online  (GRPO, PPO, RLOO): swap the loss AND the sampling (generate, score, advantage)
  loop, backward, clip, step, log, checkpoint: all inherited

The methods mapped to their one idea:

SFT     causal-LM cross-entropy; the feature is masking + templating + packing
Reward  fit a 1-logit scorer; loss = -logsigmoid(r_chosen - r_rejected)
DPO     policy is its own reward; loss = -logsigmoid(beta * (logratio_w - logratio_l))
KTO     like DPO but from unpaired thumbs-up / thumbs-down signals
GRPO    group mean as baseline; advantage = (r - mean)/std; clipped PG + KL
PPO     4 models (policy, ref, reward, value); GAE advantage; clipped PG + KL
RLOO    REINFORCE with a leave-one-out baseline over k samples

Memorize these blocks:

• The subclass recipe: config extends TrainingArguments, dataset prep builds the tensors, compute_loss encodes the algorithm, everything else is inherited.
• DPO reward: r(x,y) = beta * (log pi_theta(y|x) - log pi_ref(y|x)), the partition function cancels in the difference, so no reward model.
• GRPO advantage: A_i = (r_i - mean(r)) / (std(r) + eps) over a group of num_generations completions, no critic.
• Offline vs online: offline methods swap only the loss, online methods also generate every step and lean on vLLM.
• Current layout: six first-class trainers in trl/trainer/ (SFT, DPO, GRPO, KTO, RLOO, Reward), PPO and the research tail in trl/experimental/.

Part XI: Papers and further reading

The ideas in this walkthrough come from a small set of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Rafailov et al., Direct Preference Optimization, Your Language Model is Secretly a Reward Model, 2023. The closed-form reward Part IV traces through DPOTrainer. The DPO note on this site works the derivation.
2. Schulman et al., Proximal Policy Optimization Algorithms, 2017. The clipped objective behind PPOTrainer and the ratio inside GRPO. Derived step by step in the PPO note and the deep reinforcement learning class on this site.
3. Shao et al., DeepSeekMath, Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. Introduces GRPO, the group-baseline objective behind GRPOTrainer. The GRPO note on this site works the math.
4. Ouyang et al., Training language models to follow instructions with human feedback, 2022. The three-stage RLHF pipeline whose stages map onto SFTTrainer, RewardTrainer, and PPOTrainer.
5. DeepSeek-AI, DeepSeek-R1, Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. The reasoning recipe that made GRPO the common default for online training. The loops behind it are surveyed in the self-improving agents class.
6. Ethayarajh et al., KTO, Model Alignment as Prospect Theoretic Optimization, 2024. The unpaired thumbs-up and thumbs-down loss behind KTOTrainer.
7. Azar et al., A General Theoretical Paradigm to Understand Learning from Human Preferences, 2023. The analysis that yields the IPO loss selectable through loss_type.
8. Ahmadian et al., Back to Basics, Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, 2024. The leave-one-out baseline behind RLOOTrainer.
9. Liu et al., Understanding R1-Zero-Like Training, A Critical Perspective, 2025. The Dr. GRPO analysis behind the scale_rewards knob.
10. Zheng et al., Group Sequence Policy Optimization, 2025. The sequence-level importance ratio behind importance_sampling_level.
11. Hu et al., LoRA, Low-Rank Adaptation of Large Language Models, 2021. The adapter trick that lets DPO's reference be the base weights with the adapter switched off, covered in the PEFT walkthrough.
12. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The engine behind the online trainers' rollouts, covered in the vLLM walkthrough.

Part XII: Final takeaway

If the algorithms themselves are the gap, the RL foundations behind PPO and GRPO are built up in the deep reinforcement learning material, and the place these methods sit in a practical alignment pipeline is the subject of the applied generative AI material. The base loop TRL leans on is the ordinary transformers Trainer, and the generation backend behind its online methods is vLLM. Read dpo_trainer.py and reward_trainer.py back to back once, and DPO's whole argument, that a policy measured against its reference is already a reward, will read like the short piece of algebra it is.