verl

Part I: The mental model

Driver process (ONE Python process, the "single controller")
  RayPPOTrainer.fit()                       verl/trainer/ppo/ray_trainer.py
      │  for batch in dataloader:
      │
      ├─► generate_sequences(prompts)  ──►  rollout workers        verl/workers/rollout/
      │        (vLLM or SGLang engines generate n samples/prompt)
      ├─► compute reward               ──►  rule verifier or RM    verl/workers/reward_manager/
      ├─► compute_log_prob / ref_log_prob ► training engine fwd    verl/workers/engine_workers.py
      ├─► compute_advantage            ──►  on the driver itself   verl/trainer/ppo/core_algos.py
      ├─► update_actor(batch)          ──►  FSDP / Megatron step   verl/workers/engine/
      └─► update_weights()             ──►  trainer ─► rollout     resharding + transfer
                                            (same GPUs, taking turns)

The one-sentence identity: verl is a single Python process that remote-controls two heavyweight distributed systems, a training engine and an inference engine, which take turns wearing the same GPUs. RL post-training is awkward precisely because it needs both: generation wants an inference engine built for throughput (vLLM, SGLang), while the gradient update wants a training engine built for sharded optimization (FSDP, Megatron-LM). verl's contribution is not a new algorithm; it is the machinery that lets one readable training loop drive both systems and shuttle a model's weights between their incompatible sharding layouts every single step.

The HybridFlow paper frames this as a two-level dataflow problem. The control flow, "first roll out, then score, then compute advantages, then update", is the RL algorithm, and it is cheap: a few tensors per sequence. The computation flow inside each node of that graph, a forward pass over a 70B model across 64 GPUs, is enormous and multi-process. Older frameworks fused the two, so every algorithm change meant touching distributed code. verl separates them: the algorithm lives in one single-process loop that reads like pseudocode, and each heavy operator is a decorated method on a Ray worker group that knows how to split, dispatch, and re-collect its own data. That is the "single-controller / multi-worker" model, and everything else in this chapter is either an instance of it or the plumbing it needs.

One naming note before any code. The project began as volcengine/verl and migrated to the verl-project GitHub organization in January 2026; old links redirect. In the same cleanup the recipe/ directory of community-contributed algorithm reproductions (DAPO, ReTool, SPPO, and friends) moved into a separate verl-recipe repository, kept in the main tree as a git submodule. A lot of writing about verl also predates the v0.7/v0.8 "model engine" refactor that merged the old fsdp_workers.py and megatron_workers.py into a unified worker layer; this chapter describes the current layout and flags what moved.

Part II: Using it

Installing

verl is Linux-plus-NVIDIA-first (Python >= 3.10, CUDA >= 12.8 recommended; AMD ROCm and Ascend NPU paths exist). There is no macOS story worth pursuing: the stack underneath is vLLM or SGLang plus FSDP on real GPUs. The docs recommend starting from the prebuilt Docker images (published under verlai/verl on Docker Hub, built on the vLLM and SGLang release images), inside which verl itself installs without dependencies:

git clone https://github.com/verl-project/verl && cd verl
pip3 install --no-deps -e .

In a custom environment you install with an extra that pulls the inference engine you want:

pip3 install -e ".[vllm]"     # rollout with vLLM
pip3 install -e ".[sglang]"   # rollout with SGLang

Training additionally wants FSDP (which comes with PyTorch) or Megatron-LM (a separate, version-pinned install; v0.8.0 supports Megatron core v0.13.1). For everything in this chapter, FSDP plus vLLM on a single node is enough.

First real session: GSM8K

The canonical first run is math with a rule-based reward. Step one preprocesses GSM8K into parquet files that carry the fields the reward function needs:

python3 examples/data_preprocess/gsm8k.py --local_save_dir ~/data/gsm8k

The reward is a regular expression: GSM8K solutions end in #### <answer>, so verl/utils/reward_score/gsm8k.py extracts the final answer from the model output and compares it to the ground truth, scoring 1.0 for correct and 0 otherwise. No reward model, no human feedback, just a verifier. Step two launches training. The docs' quickstart runs PPO with a critic on a single 24 GB GPU with Qwen/Qwen2.5-0.5B-Instruct; the GRPO variant, which is what most people actually run now, drops the critic and samples a group per prompt. A minimal single-GPU GRPO launch:

PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
 algorithm.adv_estimator=grpo \
 data.train_files=$HOME/data/gsm8k/train.parquet \
 data.val_files=$HOME/data/gsm8k/test.parquet \
 data.train_batch_size=256 \
 data.max_prompt_length=512 \
 data.max_response_length=512 \
 actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
 actor_rollout_ref.actor.optim.lr=1e-6 \
 actor_rollout_ref.actor.ppo_mini_batch_size=64 \
 actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
 actor_rollout_ref.actor.use_kl_loss=True \
 actor_rollout_ref.actor.kl_loss_coef=0.001 \
 actor_rollout_ref.rollout.name=vllm \
 actor_rollout_ref.rollout.n=5 \
 actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
 actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
 actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
 actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
 trainer.logger=console \
 trainer.n_gpus_per_node=1 trainer.nnodes=1 \
 trainer.total_epochs=15 2>&1 | tee verl_demo.log

Note that GRPO does not get its own entrypoint. In verl, every algorithm in the PPO family is main_ppo.py plus config overrides: algorithm.adv_estimator=grpo selects the advantage function, rollout.n=5 makes the rollout engine sample a group of five per prompt, and use_kl_loss=True moves KL regularization from the reward into the actor loss, per the GRPO paper. What you watch in the logs is one line per step with a timing breakdown (timing/gen, timing/update_actor, ...), reward statistics, and, every trainer.test_freq steps, val/test_score/openai/gsm8k climbing from a few percent toward a ceiling that depends on the model. Exact numbers vary run to run; the shape of the curve does not. Checkpoints land in checkpoints/<project>/<experiment>/ in sharded engine format, and python3 -m verl.model_merger merge --backend fsdp ... reassembles a Hugging Face checkpoint from them.

The knobs that matter

The config system is Hydra over a tree of YAML defaults (verl/trainer/config/ppo_trainer.yaml composes per-role files from actor/, rollout/, critic/, ref/, engine/), and the batch-size hierarchy is the part everyone trips over:

• data.train_batch_size: prompts sampled per training step. Multiply by rollout.n for the number of trajectories.
• actor.ppo_mini_batch_size: trajectories per optimizer update; the rollout batch is split into these, so one step performs train_batch_size * n / ppo_mini_batch_size gradient updates. A global size, not per GPU.
• actor.ppo_micro_batch_size_per_gpu: forward or backward chunk per GPU, pure gradient accumulation. Micro-batch sizes are memory knobs and never change convergence; mini-batch size is an algorithm knob and does. The modern alternative is actor.use_dynamic_bsz=True with ppo_max_token_len_per_gpu, which packs by token count instead of sequence count.
• rollout.gpu_memory_utilization: the fraction of the GPU handed to the inference engine during generation (0.4 to 0.6 typical when colocated, because the trainer needs the rest).
• algorithm.use_kl_in_reward versus actor.use_kl_loss: KL as a reward penalty (classic RLHF PPO) versus KL as a loss term (GRPO). Enabling either is what makes the reference policy worker exist at all.

The classic beginner mistake pair, wrong versus right, is treating micro-batch as the tuning surface:

# Wrong intent: "make training converge differently"
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=16   # only changes memory/speed

# Right: change the actual update granularity
actor_rollout_ref.actor.ppo_mini_batch_size=128           # changes optimization

Part III: When it is the right tool

verl is the right default when the RL loop itself is the workload: GRPO/PPO-style reasoning training on verifiable rewards, RLHF with reward models, multi-turn agent and tool-use training, at any scale from one node to thousands of GPUs, with models from 0.5B to Megatron-scale MoE (the docs document DeepSeek-671B and Qwen3-235B runs). Its pluggability is the feature: FSDP or Megatron behind one worker API, vLLM or SGLang behind another, and an algorithm zoo that turns most published RL-for-LLM papers into a config diff or a recipe directory.

The honest alternatives: TRL is the right choice when a Hugging Face transformers Trainer subclass on one node covers your needs, especially for offline methods (DPO, SFT) where no rollout engine is needed at all, and its GRPOTrainer is genuinely capable on single-node setups; the step up to verl buys you Ray-based multi-node orchestration, Megatron sharding for models that FSDP cannot hold, and placement flexibility. OpenRLHF occupies a middle position, also Ray plus vLLM, with a smaller codebase and a smaller ecosystem. If your problem is "serve a model", none of these apply; that is the vLLM chapter.

The architecture-shaped warning is about colocation memory choreography. In the default hybrid mode, the trainer and the rollout engine share every GPU, and the design only works because they alternate: vLLM's sleep mode releases weight and KV memory during the training phase, and the trainer offloads or frees before generation resumes. If you break the choreography, by pinning rollout.free_cache_engine=False while also keeping full FSDP parameters resident, or by sizing gpu_memory_utilization as if the engine owned the device the way a serving deployment does, you get the OOM that only appears at step boundaries.

Safe (time-sliced):                    Dangerous (both resident):
t0 rollout awake (0.5 GPU) ── gen      vLLM engine claims 0.9
t1 rollout sleeps, frees KV+weights    FSDP params+optimizer resident too
t2 trainer wakes ── fwd/bwd/step         → OOM at the first update step,
t3 sync weights, rollout wakes             or thrash on every transition

Part IV: The full life of one GRPO step

The canonical operation is one iteration of RayPPOTrainer.fit() with adv_estimator=grpo. Everything happens on, or is ordered by, the driver process; the heavy lifting happens on worker groups it commands. Paths are v0.8.0.

Stage 1: Boot, once

python3 -m verl.trainer.main_ppo resolves the Hydra config, starts Ray, and launches a TaskRunner actor (deliberately not on the head node) whose run() builds the tokenizer, datasets, resource pools, and the role-to-worker mapping: Role.ActorRollout (or ActorRolloutRef when a reference policy is needed) maps to ActorRolloutRefWorker and Role.Critic to TrainingWorker, both from verl/workers/engine_workers.py. GRPO has no critic, so only the first group is created. trainer.init_workers() then spawns one Ray worker per GPU per group, and each worker's init_model() builds three things in place: the training engine for the actor (FSDP by default), the rollout engine (a vLLM instance per replica), and the checkpoint engine used later for weight sync. In v0.8.0 a note in the code marks main_ppo.py deprecated in favor of main_ppo_sync.py; both drive the same trainer.

Stage 2: The driver builds the batch

fit() pulls a batch of prompts from a stateful dataloader and wraps it in a DataProto (verl/protocol.py), verl's universal envelope: a TensorDict of batched tensors plus non-tensor arrays plus meta-info, which every worker method consumes and returns. It tags each prompt with a UID, then repeats the batch rollout.n times, interleaved, because GRPO needs a group of n rollouts per prompt and downstream code groups by that UID.

Stage 3: Rollout generation

The driver calls async_rollout_manager.generate_sequences(gen_batch). Under the hood the rollout side of ActorRolloutRefWorker runs vLLM in server mode (verl/workers/rollout/vllm_rollout/, with sglang_rollout/ and an experimental TensorRT-LLM backend as siblings); requests fan out across replicas, each replica being a tensor-parallel vLLM engine. Sampling parameters, chat templating, and multi-turn tool loops (the agent_loop machinery in verl/experimental/agent_loop/) live on this side. When generation returns, the driver puts the replicas to sleep, releasing their weight and KV memory so the training engine can have the GPUs. The output DataProto now carries responses, masks, and optionally the rollout engine's own log-probs (rollout_log_probs), which matter in Stage 5.

Stage 4: Reward

Rewards are computed by a reward manager (verl/workers/reward_manager/; naive is the default, dapo and prime implement paper-specific shaping). For rule-based tasks it calls the per-dataset scorer in verl/utils/reward_score/ keyed by the data_source field, or your own function via reward.custom_reward_function.path. If a neural reward model is enabled (reward.reward_model.enable), scoring runs as a server-mode model, optionally on its own resource pool. Either way the result is a token-level score tensor, which for outcome rewards is zero everywhere except the final response token.

Stage 5: Recomputing log-probs

The driver then asks the actor for compute_log_prob(batch): a forward-only pass of the training engine over the generated sequences, producing old_log_probs, the probabilities that anchor the PPO/GRPO ratio. This looks redundant, since the rollout engine already reported log-probs while sampling, but it is load-bearing: vLLM's numerics and the training engine's numerics differ, and the ratio must be computed against the policy as the trainer sees it. v0.8.0 makes the discrepancy a first-class citizen: the algorithm.rollout_correction config can compute importance-sampling weights between rollout and trainer policies (decoupled mode), or bypass the recompute entirely and trust rollout_log_probs (bypass mode) when you accept two-policy training. If KL regularization is on, the fused reference policy answers compute_ref_log_prob the same way.

Stage 6: Advantages, on the driver

Advantage computation is deliberately cheap enough to run in the driver process: compute_advantage() dispatches through a registry in verl/trainer/ppo/core_algos.py to compute_grpo_outcome_advantage, which groups rewards by prompt UID, subtracts the group mean, and (with norm_adv_by_std_in_grpo=True, the default) divides by the group standard deviation. Every sequence in a group gets its scalar advantage broadcast across its response tokens. With adv_estimator=gae this stage instead runs generalized advantage estimation over the critic's values, which is the PPO path (see the PPO page for the derivation).

Stage 7: The update

update_actor(batch) ships the assembled batch (responses, old and ref log-probs, advantages) to the actor workers, where TrainingWorker.train_mini_batch slices it into mini-batches and micro-batches and drives the engine (verl/workers/engine/fsdp/transformer_impl.py or the Megatron twin) through forward, loss, backward, step. The loss function is ppo_loss from verl/workers/utils/losses.py, assembling the clipped surrogate from core_algos.py pieces: ratio clipping with clip_ratio, optional entropy bonus, and the direct KL loss term when use_kl_loss=True. Sequence-length balancing (trainer.balance_batch, verl/utils/seqlen_balancing.py) has already reordered the batch so each data-parallel rank chews a similar token count.

Stage 8: Weight sync, closing the loop

Finally the driver calls checkpoint_manager.update_weights(global_steps), which fans out to ActorRolloutRefWorker.update_weights(). In colocated sync mode the sequence is: wake the rollout engine's weight memory, stream the actor engine's parameters per tensor into the vLLM replicas, offload or free trainer memory, then restore the KV cache. The rollout engine now holds the post-update policy, and the next iteration's generations are on-policy again. This stage is the hard engineering at the center of the system and gets its own deep dive below.

Part V: Internals deep dives

HybridFlow: the single-controller / multi-worker model

The paper's key observation is that RL training is a nested dataflow: an outer graph whose nodes are "rollout", "reward", "update" (the control flow), and inner graphs which are the distributed neural computations themselves (the computation flow). Prior systems made the outer graph multi-process too, SPMD everywhere, which is fastest but welds the algorithm to one execution strategy. verl chooses the other corner: the outer graph runs in one process, and pays a data movement tax at every node boundary to buy algorithm-level programmability. The bet is that for LLM-scale nodes the tax is small relative to the computation, and that bet has held.

The mechanism is in verl/single_controller/. A Worker exposes methods decorated with @register(dispatch_mode=...) (base/decorator.py); a RayWorkerGroup (ray/base.py) binds those methods onto itself so the driver can call worker_group.method(data) as if it were local. The dispatch mode is the contract: ONE_TO_ALL broadcasts the same arguments to every worker (init, checkpoint ops); DP_COMPUTE_PROTO and the newer mesh-aware make_nd_compute_dataproto_dispatch_fn split a DataProto across data-parallel ranks, invoke, and concatenate the results. The decorated signature is exactly the paper's separation made concrete:

driver:   out = actor_wg.compute_log_prob(batch)     one line, single process
            │ dispatch_fn: split batch into dp chunks
            ▼
workers:  each rank runs compute_log_prob(chunk)     SPMD inside
            │ collect_fn: gather + concat
            ▼
driver:   out is a plain DataProto again

Two consequences are worth internalizing. First, swapping FSDP for Megatron, or vLLM for SGLang, changes nothing in ray_trainer.py, because the trainer only speaks worker-group method names; that is why the same fit() serves PPO, GRPO, and a dozen recipe algorithms. Second, the driver is a real bottleneck surface: every batch crosses process boundaries at least twice per stage, which is why DataProto exists (a compact, serializable convention rather than loose kwargs) and why newer experimental paths (verl/experimental/transfer_queue, async training) work to take bulk data off the driver hop. The famous misconception to correct: "single controller" does not mean single point of computation; the driver never touches model weights, it only routes metadata and modest tensors, and advantage computation is the one numeric task it keeps because it is O(batch) rather than O(model).

The worker architecture and colocation

v0.8.0 has two load-bearing worker classes in verl/workers/engine_workers.py. TrainingWorker wraps one model engine, the unified abstraction over FSDP/FSDP2, Megatron, TorchTitan, and VeOmni backends in verl/workers/engine/, and exposes exactly three compute verbs: train_mini_batch, train_batch, infer_batch. The critic is just a TrainingWorker with a value-head model and a value loss. ActorRolloutRefWorker composes up to three roles in one process per GPU: an actor TrainingWorker, an optional ref TrainingWorker, and a rollout engine, and its role string ("actor_rollout", "actor_rollout_ref", ...) decides which get built.

One GPU, hybrid mode:
┌────────────────────────────────────────────────┐
│ ActorRolloutRefWorker (one Ray actor)          │
│   actor:  TrainingWorker ── FSDP shard         │
│   ref:    TrainingWorker ── frozen shard       │
│   rollout: vLLM engine (TP slice of a replica) │
│   checkpoint_engine: weight-sync transport     │
└────────────────────────────────────────────────┘
mapping: {ActorRolloutRef: "global_pool"}  ← all roles, all GPUs

Why fuse them? Weight sync becomes device-local (actor and rollout share a GPU, so parameters move by NCCL or even in-process pointer handoff instead of across a network), and the reference policy is nearly free when LoRA is used, because the base model already sitting under the actor's adapters is the reference. The cost is the time-slicing choreography from Part III. Placement is a config decision, not a code one: the ResourcePoolManager maps roles to named GPU pools, so putting a neural reward model on its own pool (reward.reward_model.enable_resource_pool=True), or disaggregating rollout from training entirely (the one-step-off-policy and fully-async recipes in verl/experimental/), is the same trainer with a different mapping. The misconception to correct here: rewards are not a worker by default. Rule-based scoring runs as plain functions orchestrated by the reward loop, and only an actual neural reward model occupies GPUs.

The resharding problem: FSDP shards to vLLM tensor parallel

Here is the hard problem stated plainly. After the optimizer step, the current policy exists as FSDP shards: each of N training ranks holds 1/N of every parameter, flattened and wrapped. The rollout engine needs the same weights as vLLM tensor-parallel shards: each of TP ranks holds specific row- or column-slices of each projection matrix, in vLLM's fused-layer naming scheme. These layouts share nothing, and the translation must run every step, on GPUs that are also holding optimizer state, without materializing a full copy of the model if the model is large.

FSDP rank0  [1/N of every param]──┐
FSDP rank1  [1/N of every param]──┤  all-gather per tensor
   ...                            ├─►  full tensor (transient, bucketed)
FSDP rankN  [1/N of every param]──┘        │ slice per vLLM TP rank
                                           ▼
                              vLLM TP0 [cols 0..k]   TP1 [cols k..2k] ...

The v0.8.0 mechanics, in ActorRolloutRefWorker.update_weights(): the engine yields parameters through get_per_tensor_param(), a generator that all-gathers one tensor (or one layer, with layered_summon) at a time; the rollout side consumes the stream and scatters slices into engine memory, with bucketed_weight_transfer.py batching tensors into buckets sized by rollout.checkpoint_engine.update_weights_bucket_megabytes (default 2048) so transfer overlaps and peak memory stays bounded. Before any of this, the rollout engine's weight memory is resumed from sleep; after it, trainer parameters offload and the KV cache resumes. LoRA has a fast path: with model.lora.merge=True adapters merge into base weights before sync so the engine sees a standard update, and without merging only adapter deltas move after a one-time base sync.

The transport itself is pluggable through verl/checkpoint_engine/: naive is the colocated direct path, while nccl, nixl, mooncake, and vendor engines serve disaggregated async training where trainer and rollout live on different GPUs and weights cross a fabric. Two traps to correct. First, "just save a checkpoint and reload it" is the strawman this machinery replaces; at 70B scale that is minutes per step versus seconds. Second, weight sync is also where silent correctness bugs live: if a new model architecture maps names wrong between the training engine and vLLM's fused parameters, training "works" while rollouts quietly come from a stale or scrambled policy, which is why rollout_corr_helper metrics comparing rollout and trainer log-probs double as a canary.

The algorithm zoo and the config system

The zoo has two layers. In-tree, core_algos.py registers advantage estimators: gae, grpo, reinforce_plus_plus, remax, rloo, opo, grpo_passk, gpg, gdpo, and vectorized variants; @register_adv_est lets a user module add one without forking. Combined with the policy-loss options (clipping, loss_agg_mode token-mean versus seq-mean variants, KL-in-loss versus KL-in-reward), most "new algorithms" are points in this config space: DrGRPO is GRPO with loss_agg_mode=seq-mean-token-sum-norm and norm_adv_by_std_in_grpo=False; RLOO is a different baseline in the same loop. The second layer is the verl-recipe submodule, where paper reproductions (DAPO, SPPO, ReTool, and others) subclass the trainer or TaskRunner when they need loop changes, which is exactly what the single-controller design makes cheap. The trap: the config tree is powerful and unchecked combinations are legal-looking; ppo_epochs>1 with a tiny group size, or KL in both reward and loss at once, will run and quietly train something you did not mean. Read verl/trainer/config/algorithm.py and the validate_config checks before inventing combinations.

Part VI: Reading the repository

Verified against v0.8.0. The tree is large (about 500 files under verl/), but the request path from Part IV covers everything foundational.

Stage 0, orientation (one evening). Read the HybridFlow paper (arXiv 2409.19256) sections 1 to 3, then docs/hybrid_flow.rst in the repo, which is the maintainers' own programming guide and matches the current code. Questions: why is RL a two-level dataflow? What does the single-controller choice trade away, and why is that acceptable for LLMs?

Stage 1, the programming model. Read verl/protocol.py (DataProto), then verl/single_controller/base/decorator.py, base/worker.py, base/worker_group.py, and verl/single_controller/ray/base.py. Questions: what exactly does @register attach to a method? Trace one DP_COMPUTE_PROTO call from driver to workers and back. Where does colocation of multiple roles into one Ray actor happen?

Stage 2, the algorithm loop. Read verl/trainer/main_ppo.py, then verl/trainer/ppo/ray_trainer.py (start at fit() and work outward), then verl/trainer/ppo/core_algos.py for compute_grpo_outcome_advantage and the loss helpers. Questions: which computations run on the driver? Where does rollout.n take effect? What is the difference between bypass and decoupled rollout correction?

Stage 3, the workers. Read verl/workers/engine_workers.py top to bottom, then skim verl/workers/engine/base.py and engine/fsdp/transformer_impl.py, and verl/workers/rollout/vllm_rollout/vllm_rollout.py. Finish with update_weights() and verl/checkpoint_engine/base.py. Questions: what three verbs does a TrainingWorker expose? Walk the weight-sync sequence including sleep/resume. What changes when rollout.name=sglang?

Stage 4, configuration and examples. Read verl/trainer/config/ppo_trainer.yaml with the config docs page open, then one real script, examples/grpo_trainer/run_qwen3_8b_fsdp.sh, and map every override to the tree. Then clone the verl-recipe submodule and read the DAPO recipe as a worked example of extending the trainer. Questions: which batch size is global and which is per GPU? What would you change, and only that, to run DrGRPO?

Where not to start: the Megatron engine backend, verl/experimental/ (async policy, transfer queue, agent loops), the NPU/ROCm paths, and the non-naive checkpoint engines. All are specialist territory that assumes the synchronous FSDP+vLLM path is already solid in your head, and none of them will teach you what verl is.

Part VII: Hands-on labs

Labs assume one Linux node with at least one 24 GB NVIDIA GPU, verl installed with the vLLM extra, and the GSM8K parquet files from Part II. Log wording shifts between releases; the quantities do not.

Lab 1: run the quickstart and read one step. Launch the Part II GRPO command with trainer.total_epochs=1 and trainer.logger=console. For one step line, account for every field: timing/gen, timing/old_log_prob, timing/adv, timing/update_actor, reward mean/max, response lengths. Concept taught: the Part IV pipeline is literally the log format.

Lab 2: PPO versus GRPO on the same data. Run the docs' PPO quickstart (critic enabled, GAE) and the GRPO variant, and compare: GPU memory (the critic is a whole second model), time per step, and the reward curve over the first 50 steps. Concept taught: what the group baseline buys and costs versus a learned value function.

Lab 3: group size and advantage collapse. Run GRPO with rollout.n=2 and n=8. Watch the fraction of prompts where every sample gets the same reward (all correct or all wrong): those groups have zero advantage and contribute nothing. Concept taught: why GRPO needs meaningful group sizes and why curricula filter prompts that are too easy or too hard.

Lab 4: write a custom reward. Create my_reward.py with def compute_score(data_source, solution_str, ground_truth, extra_info=None): returning 1.0 when the answer string appears in the output, and pass reward.custom_reward_function.path=my_reward.py. Verify with trainer.val_only=True that validation uses your scores. Concept taught: the reward manager boundary, and how verifiers plug in without touching workers.

Lab 5: break the memory choreography on purpose. On a single GPU, raise rollout.gpu_memory_utilization to 0.9 and watch where the run dies (usually at the first update or the first wake-up transition), then restore 0.4 and set actor.fsdp_config.param_offload=True and compare step time. Concept taught: colocation is time-slicing, and both residents must be configured as guests.

Lab 6: watch the weight sync matter. Set rollout.checkpoint_engine.update_weights_bucket_megabytes=128 versus 2048 and compare the update_weights timing in the logs. Then read the log lines around "Before resume weights / After update_weights" (GPU memory snapshots) for one step. Concept taught: resharding is a real, measurable stage with a bandwidth/memory trade-off, not bookkeeping.

Part VIII: Understanding checks

What problem does verl solve that a plain training script does not have? RL post-training needs a throughput-optimized inference engine for rollouts and a sharded training engine for updates, with weights synchronized between incompatible layouts every step, across many GPUs. verl packages that orchestration so the algorithm remains a readable single-process loop.

What is the single-controller model? The RL control flow runs in one driver process; each heavy computation is a method on a Ray worker group, decorated with a dispatch mode that defines how input data is split across workers and how outputs are collected. The driver composes these calls like local functions; the workers run SPMD internally.

What does @register(dispatch_mode=...) actually do? It attaches dispatch and collect functions to a worker method so the WorkerGroup can expose a single driver- side callable: split a DataProto into per-rank chunks, invoke all ranks, and concatenate results. ONE_TO_ALL instead broadcasts identical arguments, used for init and checkpoint control.

Why does GRPO in verl have no separate trainer entrypoint? Because GRPO and PPO share the loop structure; they differ in the advantage estimator, group sampling (rollout.n), and where KL enters. verl expresses all three as config on main_ppo, which is the single-controller thesis working as intended.

Why recompute log-probs after generation instead of using vLLM's? The training engine's numerics differ from the inference engine's, and the PPO/GRPO ratio should anchor on the policy as the trainer computes it. verl also offers explicit modes: bypass (trust rollout log-probs, two-policy training) and decoupled correction (importance weights between the two), making the mismatch measurable instead of silent.

Where are advantages computed, and why there? On the driver, in core_algos.py. Advantage math is O(batch) scalars, not O(model), so it costs nothing meaningful, and keeping it on the driver means new estimators are pure Python functions registered in one file rather than distributed code.

Walk through the GRPO advantage. For each prompt, n sampled responses are scored; the group mean is subtracted from each response's reward, optionally divided by the group standard deviation, and the resulting scalar is broadcast over that response's tokens. The group replaces PPO's learned critic as the baseline.

What is the resharding problem? After each update the policy exists as FSDP (or Megatron) training shards, but rollouts need it as vLLM tensor-parallel shards with fused parameter naming. verl streams parameters per tensor, all-gathered on the training side, sliced and scattered into engine memory, bucketed to bound peak memory, every step.

Why does colocated verl put vLLM to sleep during training? Both systems cannot hold full state simultaneously on one GPU. Sleep releases the engine's weights and KV cache during forward/backward; weight sync wakes weights, streams updates, then restores KV. Breaking this ordering is the classic step-boundary OOM.

When is a critic worth its memory? When rewards are dense or the token-level credit assignment of GAE materially beats an outcome-level group baseline, classically in RLHF with reward models. For verifiable outcome rewards, GRPO's group baseline usually wins the memory-for-variance trade, which is why it dominates reasoning training.

What is DataProto and why does it exist? The single envelope (tensors + non-tensor arrays + meta-info) that every worker method consumes and returns. A fixed, serializable interchange format is what lets dispatch modes split and concatenate any stage's data generically across process boundaries.

How would you place a 70B reward model on separate GPUs? Enable reward.reward_model.enable_resource_pool=True with its node/GPU counts; the ResourcePoolManager creates a second pool and maps the RewardModel role onto it. The trainer code does not change, which is the point of separating placement from algorithm.

Your reward curve is flat at zero. What do you check first? That the reward function is actually matching: run validation with dumped generations (trainer.rollout_data_dir) and eyeball whether outputs contain extractable answers in the expected format. A format mismatch between the model's output style and the verifier's regex is the most common silent failure, ahead of any RL hyperparameter.

Training reward rises but validation score does not. What is happening? Likely reward hacking or KL drift: the policy exploits the training verifier (length, format tricks) or has drifted far from the reference. Check response length trends, KL metrics, and spot-read generations; tighten the verifier or raise KL regularization.

verl versus TRL versus OpenRLHF in one breath? TRL for single-node trainer-subclass ergonomics and offline methods; OpenRLHF as a leaner Ray+vLLM midpoint; verl when you need the placement flexibility, Megatron-scale sharding, multi-backend rollout, and the recipe ecosystem, at the cost of a Ray cluster and a Hydra config tree.

Part IX: Design lessons

Separate the dataflow from its execution. The deepest idea in the codebase: describe what computes on what, and let dispatch machinery decide where. Spark did it for analytics, TensorFlow's graph did it for training, verl does it for RL loops. When a domain's algorithms churn faster than its infrastructure, this split is what keeps the churn cheap.

Make placement a config, not an architecture. Colocated hybrid, split pools, fully disaggregated async: the same trainer serves all of them because roles map to resource pools in one dictionary. Systems that hard-code placement die the first time the hardware shape changes.

Registries beat forks. Advantage estimators, reward managers, rollout backends, checkpoint engines: each is a string-keyed registry with a decorator to add entries. That is why paper reproductions land as recipes rather than as year-long divergent forks, the same pattern that keeps vLLM's model zoo and attention backends contributable.

Define one envelope for the seams. DataProto is deliberately boring: batched tensors, arrays, metadata. Every interesting boundary in the system speaks it, which makes every stage loggable, balanceable, and dispatchable by generic code. Protobufs at API boundaries and Arrow in data systems are the same move.

Make numerical mismatch observable, then optional. Training/inference policy mismatch was a folk bug for two years; v0.8's rollout-correction machinery turns it into logged importance-weight metrics with explicit modes. Upgrading a silent assumption into a measured, configurable quantity is a general pattern for hardening ML systems.

Shrink the core by exporting the periphery. Moving recipes to a submodule repository and unifying five worker implementations into one engine layer are deletions disguised as reorganizations. Healthy infrastructure projects periodically pay this cost; the alternative is a tree where nobody can find the load-bearing 10%.

Part X: Memorization framework

One sentence: verl runs the RL algorithm as one readable process that dispatches rollout, reward, and update work to worker groups wrapping vLLM/SGLang and FSDP/Megatron, and re-shards the policy's weights from training layout to inference layout every step so the loop stays on-policy.

Prompts → Rollout(n) → Reward → old/ref logp → Advantage → Update → Weight sync → (repeat)

The chain mapped to files (v0.8.0):

Loop/driver   trainer/ppo/ray_trainer.py  (fit(), single process)
Dispatch      single_controller/base/decorator.py, ray/base.py
Rollout       workers/rollout/vllm_rollout/, sglang_rollout/
Reward        workers/reward_manager/, utils/reward_score/
logp / update workers/engine_workers.py → workers/engine/{fsdp,megatron}/
Advantage     trainer/ppo/core_algos.py  (registry, on driver)
Weight sync   engine_workers.update_weights() + checkpoint_engine/
Config        trainer/config/ppo_trainer.yaml  (Hydra tree)

Memorize these:

The model fact: single controller, multi worker: control flow in one process, computation flow in worker groups, joined by @register dispatch modes and DataProto.

The batch fact: train_batch_size prompts × rollout.n samples → split into global ppo_mini_batches (an algorithm knob) → sliced into per-GPU micro-batches (a memory knob only).

The sync fact: every step: rollout sleeps → train → per-tensor all-gather from FSDP shards → bucketed stream into vLLM TP shards → offload trainer → wake KV cache.

The algorithm fact: one loop, many algorithms: adv_estimator selects grpo/gae/rloo/remax/...; GRPO = group mean baseline + KL in the loss + no critic; DrGRPO and friends are config points, recipes subclass for the rest.