TRL

Part I: The mental model

Hub dataset ("prompt"/"chosen"/"rejected", or "prompt" + reward fn)
      │
      ▼
Dataset prep: chat template, tokenize once     trl/trainer/*_trainer.py (_prepare_dataset)
      │
      ▼
Data collator: batch layout for the method     DataCollatorForPreference, ...
      │
      ▼
compute_loss(): the method IS this function    dpo_trainer.py, grpo_trainer.py, ...
      │                (GRPO adds a generation step: vLLM / transformers)
      ▼
transformers Trainer loop                      inherited, not reimplemented
      │
      ▼
accelerate (DDP / FSDP / DeepSpeed)  +  peft (LoRA / QLoRA)  +  optimizer

The one-sentence identity: TRL is a family of loss functions and data pipelines wearing the transformers Trainer. Each trainer class, SFTTrainer, DPOTrainer, GRPOTrainer, and their siblings, owns exactly three things: how to turn a Hub-style dataset into tensors, how to lay those tensors out in a batch, and what compute_loss does with one batch. Everything else, the training loop, gradient accumulation, mixed precision, checkpointing, logging, distributed execution, is inherited from transformers.Trainer and delegated to accelerate. That is both the superpower and the boundary: if a method fits the "one model, one loss, one collator" shape, TRL makes it a fifty-line script; if it needs its own distributed choreography, TRL stretches (GRPO with a vLLM server is the stretch) and eventually you reach for verl.

The second thing to internalize is the version history, because it decides which documentation you can trust. TRL lived at 0.x from 2020 through early 2026, accumulating a dozen trainers of wildly varying maturity. v1.0.0 (March 2026) drew a hard line: a small stable core in trl/trainer/ (SFT, DPO, GRPO, RLOO, KTO, Reward) and everything else, including PPOTrainer, OnlineDPO, ORPO, and CPO, moved to trl/experimental/ with an explicit no-stability-guarantee. Most tutorials on the internet predate this split and import trainers from locations that no longer exist. This chapter describes v1.9.0.

Why does a "reinforcement learning" library headline DPO, which is not RL at all? Because the library tracks what the field actually does: DPO turned the RLHF objective into a supervised classification loss (the identity is derived on the DPO page), and GRPO turned online RL into "sample a group, score it, weight the log-probs" (GRPO page). TRL's bet is that post-training methods keep collapsing into shapes a Trainer subclass can hold, and so far the field keeps proving it right.

Part II: Using it

Installing

TRL is a pure-Python package on top of torch and transformers, so installation is the same on Linux and macOS:

pip install trl        # or: uv pip install trl
pip install trl[vllm]  # adds vLLM for fast generation in online methods (Linux+GPU)
pip install peft       # LoRA/QLoRA support, used throughout this chapter

CPU and Apple-silicon machines can run the small offline examples (slowly); the online methods (GRPO, RLOO) want a CUDA GPU, and the vLLM extra is Linux-only. Nothing else to configure: multi-GPU comes later via accelerate launch.

First real session: a DPO run

The v1-era API is aggressively minimal; strings are accepted where objects used to be required, and defaults are chosen to work. This is the documented quickstart, verbatim in spirit:

# train_dpo.py
from datasets import load_dataset
from trl import DPOTrainer

trainer = DPOTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
)
trainer.train()

python train_dpo.py          # single GPU
accelerate launch train_dpo.py   # same script, multi-GPU

What happens: the model and tokenizer load from the Hub, the dataset's conversational prompt/chosen/rejected columns are rendered through the model's chat template and tokenized once up front (with a progress bar), a frozen copy of the model is created as the reference, and the standard Trainer loop starts logging loss, rewards/chosen, rewards/rejected, rewards/accuracies, and rewards/margins every few steps. On one modern 24 GB GPU the 0.6B model trains comfortably; the metrics to watch are accuracy (how often the implicit reward ranks chosen above rejected; drifting up from 0.5) and margin (growing from zero). Add a DPOConfig when you want control:

from trl import DPOConfig, DPOTrainer

args = DPOConfig(
    output_dir="qwen-dpo",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=5e-6,
    beta=0.1,              # strength of the preference signal (default 0.1)
    max_length=1024,       # truncation length for prompt+completion
    logging_steps=10,
    bf16=True,
)
trainer = DPOTrainer(model="Qwen/Qwen3-0.6B", args=args, train_dataset=...)

Dataset formats: the thing to get right first

Every TRL trainer autodetects two axes of the dataset: standard (plain strings) versus conversational (lists of role/content messages), and explicit versus implicit prompt. For DPO the recommended shape is explicit-prompt conversational:

{"prompt":   [{"role": "user", "content": "What color is the sky?"}],
 "chosen":   [{"role": "assistant", "content": "It is blue."}],
 "rejected": [{"role": "assistant", "content": "It is green."}]}

Conversational datasets get the chat template applied automatically; standard ones are used as-is. The classic beginner mistake is pre-applying the chat template to a conversational dataset, or feeding raw strings that contain hand-written role markers: the template then gets applied twice or not at all, and the model trains on malformed turn structure. Wrong versus right:

# Wrong: template applied by hand, then TRL applies it again
row = {"prompt": "<|im_start|>user\nHi<|im_end|>", ...}

# Right: give TRL messages; it owns the template
row = {"prompt": [{"role": "user", "content": "Hi"}], ...}

A GRPO run with a reward function

GRPO needs only prompts plus something that scores completions. TRL ships reward functions in trl.rewards (accuracy_reward parses math answers and checks them against the dataset's solution column, think_format_reward checks reasoning tags), and the documented quickstart is math on DeepMath-103K:

# train_grpo.py
from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward

dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=accuracy_reward,
    train_dataset=dataset,
)
trainer.train()

Run with accelerate launch train_grpo.py; the docs note the full dataset takes on the order of a day across 8 GPUs, so cap max_steps for a first pass. A custom reward function is any callable taking prompts, completions, and the dataset's extra columns as keyword arguments and returning one float per completion (or None to abstain on a sample):

def brevity_reward(completions, **kwargs):
    # conversational completions: list of message lists
    return [-len(c[0]["content"]) / 100 for c in completions]

trainer = GRPOTrainer(model=..., reward_funcs=[accuracy_reward, brevity_reward],
                      train_dataset=dataset)

Multiple reward functions are summed (weighted by reward_weights), and each gets its own rewards/<name>/mean metric, which is how you catch one reward silently dominating another. The remaining beginner mistakes worth naming: expecting a KL penalty by default (beta defaults to 0.0 in GRPOConfig, so there is no reference model and no KL term unless you set it); and scoring completions as strings when the dataset is conversational, in which case each completion is a list of messages, not a string.

Part III: When it is the right tool

TRL is the right default for post-training that fits one machine or a modest cluster: SFT on your data, DPO/KTO from preference pairs, reward-model training, and GRPO-style verifiable-reward RL on models up to the tens of billions with LoRA. Its unfair advantages are ecosystem gravity, everything on the Hub loads directly and every transformers feature (quantization, attention implementations, VLMs) arrives nearly for free, and the accessibility stack: with peft_config and 4-bit loading, a single consumer GPU can run DPO or GRPO on models that would otherwise need a node, which no other framework in this space matches as smoothly.

The honest alternatives: verl (and OpenRLHF) exist for the regime where the RL loop itself is the distributed system, multi-node rollouts on dedicated inference fleets, Megatron-sharded models too big for FSDP, disaggregated reward models, and placement control. TRL's online trainers drive vLLM, but the trainer remains the center of gravity; verl inverts that, orchestrating engines as peers. Axolotl and Llama Factory compete on config-file ergonomics over largely the same TRL-adjacent machinery rather than on architecture. A useful rule: if your bottleneck is reading the docs, use TRL; if your bottleneck is the cluster, use verl.

The architecture-shaped warning: GRPO colocate mode shares each GPU between vLLM and training, with vllm_gpu_memory_utilization defaulting to a deliberately small 0.3. Treating that number like a serving deployment's 0.9, or running server mode on the same GPUs as the trainer, is this system's "writable SQLite on NFS": it boots, then dies mid-run at the first full optimizer step, or deadlocks NCCL when trainer and server fight over devices.

Safe:                                  Dangerous:
colocate: vLLM 0.3 + training 0.7      colocate: vLLM 0.9 + training  → OOM at step 1
server:   GPUs 0-3 trainer,            server: trainer and vllm-serve on the
          GPUs 4-7 trl vllm-serve              same CUDA_VISIBLE_DEVICES → NCCL errors

Part IV: The full life of one DPOTrainer run

The canonical operation is one DPO training run, followed through every stage. Paths are v1.9.0; the whole story lives in trl/trainer/dpo_trainer.py (about 1,800 lines) and what it inherits.

Stage 1: Construction

DPOTrainer.__init__ resolves strings to objects: the model via the architecture class named in its config, the processing class via AutoProcessor (padding side must be left for generation-adjacent code paths; a pad token is required, falling back to EOS). Then the reference policy: if you pass no ref_model and no PEFT config, it deep copies the model and freezes it; if you train with LoRA, it skips the copy entirely, because disabling the adapters recovers the base model, so the reference policy is the same weights with the adapters turned off, which halves memory and is the trick that puts DPO on one GPU.

Stage 2: Dataset preparation, once

_prepare_dataset runs before training, not per step. For conversational data it applies the chat template to prompt, chosen, and rejected; it tokenizes each into prompt_ids, chosen_ids, rejected_ids, appends EOS to completions, and stores plain integer lists in the dataset. Doing this eagerly means tokenization cost is paid once and the collator stays trivial; it also means max_length truncation (default 1024, from the left via truncation_mode) is a data decision you can audit before burning GPU hours.

Stage 3: The collator builds the concatenated batch

DataCollatorForPreference is where DPO's signature layout appears. For B examples it emits 2B rows: prompt+chosen for all examples, then prompt+rejected for all examples, right-padded together, with a completion_mask marking which positions are completion tokens:

row 0..B-1 :  [prompt | chosen   | pad]   ← first half
row B..2B-1:  [prompt | rejected | pad]   ← second half
completion_mask: 0 on prompt/pad, 1 on completion tokens

The point of concatenation: chosen and rejected go through the model in one forward pass instead of two, halving kernel launches and letting the pair share the batch's padding envelope. The cost is that the effective batch the GPU sees is twice per_device_train_batch_size, which is the single most common source of DPO OOMs.

Stage 4: The policy forward pass

compute_loss delegates to _compute_loss. The model runs over the 2B rows with use_cache=False; logits are shifted against the input ids, and selective_log_softmax extracts the log-probability of each realized token, a memory-lean gather that avoids materializing full log-softmax tensors. Prompt and padding positions are zeroed via the completion mask, per-token log-probs are summed per sequence, and logps.chunk(2) splits the result into chosen_logps and rejected_logps, exploiting the collator's ordering.

Stage 5: The reference forward pass

The same computation runs under torch.no_grad() against the reference: the frozen copy, or the PEFT model with adapters disabled. Two escape hatches matter in practice: precompute_ref_log_probs=True runs all reference forwards once before training and caches ref_chosen_logps/ref_rejected_logps in the dataset, so training pays one forward per step instead of two (and the reference model can be discarded entirely); and with cached values the collator just passes them through.

Stage 6: The loss

With the four log-prob vectors, the log-ratios are chosen_logps - ref_chosen_logps and its rejected twin; their difference is the margin, and the default sigmoid loss is the DPO paper's objective:

chosen_logratios  = chosen_logps  - ref_chosen_logps
rejected_logratios = rejected_logps - ref_rejected_logps
delta = chosen_logratios - rejected_logratios     # the margin
loss = -F.logsigmoid(beta * delta)                # per pair, then mean

v1.9.0 generalizes this in two directions without changing the skeleton: f_divergence_type transforms the log-ratios before the margin (reverse KL is standard DPO; forward KL, JS, and alpha divergences are available), and loss_type is a list, so ["sigmoid", "sft"] with loss_weights mixes an NLL term on the chosen response into the preference loss (the RPO-style regularization), while ipo, hinge, robust, apo_zero, and a dozen published variants are one string away. The implicit rewards logged as rewards/chosen are just beta * chosen_logratios detached, which is why DPO needs no reward model: the policy-reference ratio is the reward.

Stage 7: The inherited loop

From here it is pure transformers Trainer: training_step calls compute_loss, accelerate scales and backpropagates, gradient accumulation and clipping apply, the optimizer steps, callbacks fire, checkpoints save (with a TRL override to also save processor and PEFT state correctly). Nothing DPO-specific exists in the loop, which is exactly the design.

Secondarily: one GRPOTrainer step

GRPO changes the front of the pipeline, not the back. Each optimization cycle, the trainer takes a generation batch of prompts (sized per_device_train_batch_size × num_processes × steps_per_generation), generates num_generations completions per prompt (default 8) through vLLM, transformers continuous batching, or plain generate(), then calls every reward function, sums weighted rewards, and computes advantages by group: reward minus the group mean, divided by the group std under the default scale_rewards="group" ("batch" and "none" implement the Lite-PPO and Dr.GRPO recommendations). The loss recomputes per-token log-probs on the trainer side, forms the ratio against the behavior policy, clips it in [1-epsilon, 1+epsilon] (0.2, with an optional DAPO-style asymmetric epsilon_high), and aggregates with loss_type="dapo" token-level normalization by default. Because vLLM's numerics differ from the trainer's, TRL applies truncated importance sampling to the vLLM log-probs by default (vllm_importance_sampling_correction), taming the training-inference mismatch that otherwise destabilizes long runs. With beta=0.0 (default) there is no reference model at all; setting it nonzero adds the KL term and instantiates one. The same "generate, score, weight log-probs" body serves RLOOTrainer with a leave-one-out baseline instead of the group mean.

Part V: Internals deep dives

The trainer family and the v1 stability line

The stable core in trl/trainer/ is small enough to memorize, and each member is "collator + loss" over the same chassis:

PPOTrainer lives in trl/experimental/ppo now, alongside OnlineDPO, Nash-MD, XPO, ORPO, CPO, BCO, PRM, GKD, and a rotating cast of research trainers; the experimental namespace is explicitly allowed to break between minor versions. That placement is a statement about the field: classic PPO-RLHF with a learned value head has been displaced by GRPO-style group baselines for verifiable rewards (see PPO versus GRPO) and by DPO offline, and TRL promoted what people actually run. The trap for readers of older material: imports like from trl import PPOTrainer and pre-1.0 PPO tutorials do not describe the current stable API; check the taxonomy page in the docs, which marks experimental trainers with a flask icon.

Riding the stack: Trainer, accelerate, peft

TRL's most consequential design decision is what it refuses to build. The training loop, LR schedules, eval, resumption, and reporting come from transformers.Trainer; distribution comes from accelerate, so the same script scales from one GPU to DDP, FSDP, or DeepSpeed ZeRO by changing the accelerate config, not the code, and TRL ships ready-made configs (trl/accelerate_configs/: zero1 through zero3, fsdp1/fsdp2, multi_gpu). Parameter- efficiency comes from peft: pass a peft_config and the trainer wraps the model in LoRA adapters; add a quantization_config (bitsandbytes) and it is QLoRA.

The accessibility story compounds across these layers. QLoRA DPO on a 7B model: 4-bit base weights (~4 GB), frozen; LoRA adapters and their optimizer state (megabytes); no reference copy, because adapter-disabling provides it. The 2023 blog title "fine-tuning 20B LLMs with RLHF on a 24 GB consumer GPU" was this exact stack, and it remains TRL's center of mass: the marginal cost of trying preference optimization dropped to one gaming GPU, which, more than any single algorithm, is why the open post-training ecosystem runs through this library. The trade is inherited constraints too: Trainer's assumptions (one model owns the step; the loop is sacrosanct) are exactly why the heavily orchestrated online methods feel bolted on, and why the escape hatch to verl exists.

GRPO's generation plumbing: three backends and a server

Generation dominates online-RL wall-clock, so GRPOTrainer treats it as a pluggable subsystem built around vLLM. Colocate mode (use_vllm=True, the default vllm_mode="colocate") instantiates a vLLM engine inside each training process, sharing the GPU under the 0.3 memory default, with optional sleep mode (vllm_enable_sleep_mode) to release engine memory during the backward pass, a miniature of verl's hybrid choreography. Server mode runs trl vllm-serve --model <name> on dedicated GPUs (trl/scripts/vllm_serve.py); trainers connect over HTTP (trl/generation/vllm_client.py), and updated weights stream to the server between steps so generation stays near-on-policy. The third backend, transformers continuous batching, is the dependency-free middle ground. Two famous traps, both fixable from Part III's diagram: colocate OOMs come from treating the 0.3 as too conservative, and server-mode NCCL errors come from letting the server and trainer see the same devices. And one correction worth stating plainly: adding vLLM does not change the math by magic; it changes the sampling distribution subtly, which is precisely why the importance-sampling correction defaults on and why the vllm_importance_sampling_mode knobs exist.

Part VI: Reading the repository

Verified against v1.9.0. The package is compact (about 640 files, one directory of interest) and unusually readable; docstrings carry real information.

Stage 0, orientation (one evening). Read the docs' quickstart, dataset_formats page, and the index taxonomy; then the DPO paper (arXiv 2305.18290) and the GRPO section of DeepSeekMath (arXiv 2402.03300). Questions: what are standard versus conversational, explicit versus implicit datasets? Which trainers are stable versus experimental, and why does that split exist?

Stage 1, the chassis. Read trl/trainer/base_trainer.py, then skim trl/trainer/sft_trainer.py as the simplest complete member, alongside trl/data_utils.py (format detection, template application). Questions: what does a TRL trainer add on top of transformers Trainer? Where does chat templating happen, and exactly once?

Stage 2, DPO end to end. Read trl/trainer/dpo_trainer.py in this order: DataCollatorForPreference.torch_call, _prepare_dataset, _compute_loss, then dpo_config.py for every knob you just saw used. Questions: why 2B rows per batch? Where would a length bias enter, and which loss_types counter it? How does the PEFT reference trick work?

Stage 3, GRPO and generation. Read trl/trainer/grpo_config.py first (the docstrings are the best GRPO systems documentation anywhere), then grpo_trainer.py focusing on _generate_and_score_completions and the loss, then trl/generation/vllm_client.py and trl/scripts/vllm_serve.py, and trl/rewards/. Questions: trace num_generations through batching, advantage, and loss. What exactly does the importance-sampling correction multiply, and when is it clipped versus masked?

Stage 4, the edges. Skim trl/experimental/ to see tomorrow's stable API (async GRPO, OpenEnv integration), trl/cli/ for the trl sft/dpo/grpo command-line entry points, and examples/scripts/ for maintained end-to-end scripts. Questions: what would promoting an experimental trainer require? How does the CLI map flags onto the config dataclasses?

Where not to start: trl/experimental/ as a learning surface (it changes without notice), VLM-specific code paths, and the Liger fused-kernel integrations. All assume you already hold the collator-loss-chassis picture, and none will teach you what TRL is.

Part VII: Hands-on labs

Labs 1, 2, and 5 run anywhere PyTorch does (small models); labs 3 and 4 want a CUDA GPU. Losses and rewards are stochastic; trends, not values, are the observable.

Lab 1: a minimal DPO run with real metrics. Run the Part II DPO script with max_steps=100, logging_steps=5 on Qwen/Qwen3-0.6B (or trl-internal-testing tiny models on CPU). Plot rewards/accuracies and rewards/margins. Concept taught: the implicit reward is the β-scaled policy/reference log-ratio, and its margin is what DPO optimizes.

Lab 2: see the concatenated batch. Instantiate DataCollatorForPreference(pad_token_id=0) on two toy examples (the class docstring contains a worked example) and print input_ids and completion_mask. Confirm the first half is chosen and the second rejected, then verify chunk(2) recovers them. Concept taught: Stage 3 and 4 of Part IV, byte for byte.

Lab 3: GRPO with your own reward. Run the GRPO quickstart on a 1,000-row slice with accuracy_reward plus a custom brevity reward, and watch the per-function rewards/*/mean metrics diverge. Then set reward_weights=[1.0, 0.1] and observe completion lengths respond. Concept taught: the reward function interface and multi-reward composition.

Lab 4: vLLM colocate versus plain generation. Repeat Lab 3 with use_vllm=True (after pip install trl[vllm]) and compare step_time. Then lower vllm_gpu_memory_utilization to 0.1 and watch generation slow as the KV cache starves; raise it until you OOM to find your machine's budget line. Concept taught: colocation is a memory-sharing contract.

Lab 5: LoRA halves the DPO footprint. Run Lab 1 twice, once full-parameter and once with peft_config=LoraConfig(r=16), and record peak memory (torch.cuda.max_memory_allocated()). Verify the LoRA run keeps no separate reference model. Concept taught: adapter-disabling as the reference policy.

Lab 6: beta is the thermostat. Sweep DPO beta over {0.05, 0.1, 0.5} for 200 steps each. Watch margins grow faster at high beta while logps/chosen falls more (the policy drifts further from the reference). Concept taught: beta trades preference-fitting strength against staying close to the reference, DPO's stand-in for a KL budget.

Part VIII: Understanding checks

What is TRL, structurally? A set of post-training methods packaged as transformers Trainer subclasses, each contributing dataset preparation, a collator, and a compute_loss, while inheriting the loop, distribution (accelerate), and parameter-efficiency (peft) from the surrounding Hugging Face stack.

Why can DPO skip the reward model and the rollout engine? DPO reparameterizes the RLHF objective so the optimal policy's log-ratio against the reference is itself the reward; preference pairs then train it with a classification loss offline. No sampling during training, no separate reward network, just two forward passes per batch.

Why does DPOTrainer concatenate chosen and rejected into one batch? One forward pass over 2B rows instead of two passes over B: fewer launches, shared padding, and the chunk(2) split keeps the pairing implicit in row order. The cost is doubled effective batch memory, the usual DPO OOM cause.

What does the completion_mask do? It zeroes per-token log-probs on prompt and padding positions so sequence log-probs sum only over completion tokens. Both policy and reference passes reuse the same mask, keeping the four log-prob terms comparable.

How does TRL avoid keeping a second copy of the model as the reference when using LoRA? With PEFT, the base weights are frozen and only adapters train, so disabling adapters reproduces the reference policy exactly. The trainer wraps the reference forward in an adapter-disable context instead of cloning the model.

What does precompute_ref_log_probs buy? All reference forwards run once before training and cache per-example log-probs in the dataset; training then costs one forward per step instead of two, and the reference model can be freed. The trade is a preprocessing pass and staleness-free but fixed reference values.

State the GRPO advantage computation in TRL. Each prompt's num_generations completions form a group; the summed weighted rewards get the group mean subtracted and, under scale_rewards="group", are divided by the group std. Every token of a completion carries its sequence's scalar advantage.

Why is beta=0.0 the GRPO default, and what does it imply? Recent large-scale results showed the KL term against a reference contributes little for verifiable-reward training, so TRL drops it by default, which also means no reference model is loaded at all: less memory, and policy drift is bounded only by clipping.

What is the training-inference mismatch in GRPO with vLLM? The sampling engine and the trainer compute slightly different token probabilities (kernels, precision, fused ops), so trajectories are drawn from a subtly different policy than the one being updated: silent off-policy bias. TRL corrects it by importance-weighting with truncation or masking of extreme ratios, on by default.

Colocate versus server mode: when do you pick which? Colocate for simplicity and single-node runs, sharing each GPU between engine and trainer under a small memory fraction; server for dedicated inference GPUs at larger scale, started with trl vllm-serve and fed weight updates over the wire. Never point server mode at the trainer's own GPUs.

Where did PPOTrainer go, and why? To trl/experimental/ppo at the v1.0 reorganization: classic value-head PPO-RLHF ceded the mainstream to GRPO/RLOO online and DPO offline, and TRL scoped its stability guarantee to what the field runs. Experimental modules may change between minor releases.

A DPO run shows accuracy near 1.0 within 50 steps. Celebrate? Suspect leakage or triviality first: chosen and rejected may differ by formatting artifacts the model pattern-matches (length, EOS placement, template errors). Check margins on held-out pairs and read samples; a real preference signal climbs gradually.

Your GRPO rewards are all identical within each group. What happens and what do you do? Group std is zero and advantages vanish, so those prompts teach nothing (watch frac_reward_zero_std). Fix the data difficulty (filter too-easy/too-hard prompts), raise sampling temperature, or increase group size.

When do you leave TRL for verl or OpenRLHF? When rollout generation needs its own fleet, the model needs Megatron-style sharding beyond FSDP, or placement of actor/reward/rollout across pools becomes the problem; that orchestration is verl's core competency and TRL's ceiling. For one node and standard methods, TRL gets there in a tenth of the code.

Part IX: Design lessons

Inherit the loop, own the loss. TRL's leverage comes from refusing to rebuild training infrastructure: a method is a collator plus a loss on a battle-tested chassis. The same shape appears in scikit-learn estimators and Keras layers: libraries thrive by defining the smallest surface a contributor must implement.

Meet users at their data. Autodetecting standard/conversational and explicit/implicit formats, and owning the chat template exactly once, removes the most common entire class of silent bugs. Contrast with systems that accept "whatever, pre-tokenized": they push correctness onto every user. Format normalization at the boundary is cheap insurance.

Draw the stability line publicly. The v1.0 stable/experimental split converts "is this API trustworthy?" from tribal knowledge into a namespace. Rust's std-versus-nightly and Kubernetes API versioning are the same social technology: explicit tiers let a project move fast and be trusted simultaneously.

Defaults are research positions. beta=0.0, loss_type="dapo", and importance-sampling correction on-by-default each encode a finding from the recent literature. Well-maintained libraries curate defaults as aggressively as code; reading a config dataclass's defaults, and its changelog, is reading the field's current consensus.

Optimize the marginal cost of trying. The string-accepting constructors, the one-GPU QLoRA path, the bundled reward functions: all minimize the distance from idea to first loss curve. Ecosystems are won by whoever makes the first hour effortless, because the first hour is when tools get chosen.

Part X: Memorization framework

One sentence: TRL turns post-training methods into transformers Trainer subclasses, so DPO is a collator that concatenates preference pairs plus a logsigmoid on the β-scaled ratio margin, GRPO is a generation step plus group-normalized advantages on a clipped token loss, and everything below the loss line is inherited from Trainer, accelerate, and peft.

DPO:  pairs → tokenize once → [chosen;rejected] batch → one fwd → chunk(2)
        → ref fwd (or adapters-off / precomputed) → β·Δlogratio → -logsigmoid → Trainer
GRPO: prompts → vLLM gen ×G → reward fns → (r-mean)/std per group → clipped token loss

The chain mapped to files (v1.9.0):

Formats/template  trl/data_utils.py, docs dataset_formats
Chassis           trl/trainer/base_trainer.py (on transformers Trainer)
DPO               trl/trainer/dpo_trainer.py (collator, _compute_loss), dpo_config.py
GRPO              trl/trainer/grpo_trainer.py, grpo_config.py, trl/rewards/
Generation        trl/generation/vllm_client.py, trl/scripts/vllm_serve.py
Scale/PEFT        trl/accelerate_configs/, peft_config= / quantization_config=
Frontier          trl/experimental/ (incl. PPOTrainer)

Memorize these:

The DPO fact: 2B-row concatenated batch, one policy forward, one reference forward (skippable via LoRA adapters-off or precompute), completion-masked summed log-probs, loss = -logsigmoid(β·margin); implicit reward = β·log-ratio.

The GRPO fact: num_generations=8 per prompt, reward functions receive (prompts, completions, columns) and return floats or None, advantage = group-mean-centered (std-scaled by default), loss_type=dapo, epsilon=0.2, beta=0 so no reference model by default.

The stack fact: loop from transformers, distribution from accelerate (DDP/FSDP/ZeRO by config), LoRA/QLoRA from peft: the same script scales without edits.

The version fact: v1.0 (March 2026) split stable (SFT, DPO, GRPO, RLOO, KTO, Reward) from experimental (PPO, OnlineDPO, ORPO, CPO, ...); vllm_mode defaults to colocate; most pre-1.0 tutorials are out of date.