verl

Part I: The mental model

python3 -m verl.trainer.main_ppo   Hydra parses ppo_trainer.yaml + CLI overrides
      |
      v
run_ppo -> ray.init() -> TaskRunner (a Ray actor)      verl/trainer/main_ppo.py
      |
      v
RayPPOTrainer.init_workers()          verl/trainer/ppo/ray_trainer.py
      |   ResourcePoolManager places WorkerGroups onto GPU pools
      v
WorkerGroups  (each is SPMD, one worker process per GPU)
   ActorRolloutRefWorker   actor (FSDP/Megatron) + rollout (vLLM/SGLang) + ref, CO-LOCATED
   CriticWorker            value function (PPO only)
   RewardModelWorker       optional model-based reward
      |
      v
RayPPOTrainer.fit()   the SINGLE CONTROLLER: a plain Python RL loop on the driver
      |   every call below ships one DataProto to the workers and gathers the result
      v
generate -> old_logprob -> ref_logprob -> values -> reward -> advantage -> update_critic -> update_actor

The one-sentence identity. verl is the RL post-training engine that treats an RLHF algorithm as a single-controller dataflow over multi-controller worker groups, so the loop reads like textbook PPO while every stage still runs at cluster scale, and it co-locates the rollout engine and the training engine on the same GPUs so no hardware sits idle. Classic RLHF stacks force a hard choice. Either you write one giant SPMD program where the PPO logic is tangled into the same code that does tensor parallelism, which is fast but nearly unreadable, or you write clean single-process code that cannot scale past one node. verl refuses the tradeoff by drawing a line down the middle of the system.

Above the line sits the single controller. The RayPPOTrainer.fit() method is a normal Python loop that you could read to someone at a whiteboard. Sample responses, score them, compute advantages, update the critic, update the actor. It holds no model weights and does almost no math. Below the line sit the worker groups, and each one is an ordinary SPMD program, one process per GPU, doing FSDP or Megatron training or vLLM generation exactly the way a single-purpose trainer would. The controller never reaches inside a worker. It only calls named methods on a worker group and passes a DataProto, and the framework takes care of splitting that batch across the group and gathering the results. This hybrid, described in the HybridFlow paper (Sheng et al., published at EuroSys 2025), is the whole idea, and everything else in verl is machinery to make it efficient.

Two consequences follow. First, the algorithm is decoupled from the engine. You can move the actor from FSDP to Megatron, or the rollout from vLLM to SGLang, without touching a line of the PPO loop, because those are worker-internal choices the controller never sees. Second, the design makes co-location a first-class option rather than an afterthought. Because generation and training are separate worker methods on the same physical GPUs, verl can let the actor's training weights and the rollout engine share memory and reshard between them, which is the single most important performance idea in the whole system and the subject of Part V. Everything here is described against a recent state of the main branch in mid 2026. The project moves fast, so where an exact path or flag is likely to drift I say so and stay at the level of role and module.

Part II: Using it

verl is a Linux-and-NVIDIA-GPUs project. It needs Ray for orchestration and a rollout backend, and the common default rollout backend is vLLM. The intended way to run it is from source against a recent release, though a PyPI package exists too.

# verl needs Linux and CUDA GPUs, plus Ray and a rollout backend
pip install verl

# from source is the usual path, since the project moves quickly
git clone https://github.com/volcengine/verl
cd verl
pip install -e .

# a rollout backend must be present; vLLM is the common default
pip install vllm       # or install sglang instead

There are maintained Docker images that pin compatible versions of PyTorch, vLLM, SGLang, flash-attention, and Megatron, and on a fresh cluster those images save real time because the rollout backends are fussy about versions. On a laptop without a GPU you can read and step through the controller code, but training needs CUDA, so the honest local workflow is reading the repository and running on a GPU box.

A good first real run is PPO on GSM8K with a tiny model, which fits on a single GPU. verl ships a data-preprocessing script that turns the dataset into parquet with a prompt column and a ground-truth answer column, and then a single Hydra command launches the whole job.

# 1. build the dataset into parquet (prompt + ground-truth answer)
python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k

# 2. launch PPO; Ray spreads the workers over the GPUs it sees
python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=gae \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  data.train_batch_size=256 \
  data.max_prompt_length=512 \
  data.max_response_length=512 \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.actor.optim.lr=1e-6 \
  actor_rollout_ref.actor.ppo_mini_batch_size=64 \
  actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=8 \
  actor_rollout_ref.rollout.name=vllm \
  actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
  critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  critic.optim.lr=1e-5 \
  trainer.n_gpus_per_node=1 \
  trainer.nnodes=1 \
  trainer.total_epochs=15

Configuration is Hydra over a base YAML, and every one of those dotted keys is a field in verl/trainer/config/ppo_trainer.yaml overridden on the command line. The top-level groups are worth learning as a map of the system. data controls the dataset and sequence lengths, actor_rollout_ref is the co-located actor-plus-rollout-plus-reference worker, critic is the value function, reward_model is the optional model-based scorer, algorithm chooses the advantage estimator and KL settings, and trainer owns the cluster shape and logging.

Switching to GRPO is a small diff, and it is the recipe most people actually want for reasoning tasks. GRPO drops the critic entirely and normalizes rewards within a group of samples drawn from the same prompt, so you change the advantage estimator, ask the rollout for several samples per prompt, and turn on the KL-in-loss term.

# GRPO: no critic block, group-normalized advantage, KL against the reference in the loss
python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=grpo \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-7B-Instruct \
  actor_rollout_ref.rollout.name=vllm \
  actor_rollout_ref.rollout.n=8 \
  actor_rollout_ref.actor.use_kl_loss=True \
  actor_rollout_ref.actor.kl_loss_coef=0.001 \
  actor_rollout_ref.actor.kl_loss_type=low_var_kl \
  trainer.n_gpus_per_node=8 \
  trainer.nnodes=1 \
  trainer.total_epochs=15

Note there is no critic.* here, because GRPO has no value function. The rollout.n=8 is the group size, the number of responses sampled per prompt whose rewards get normalized against each other, and that group is the entire mechanism that replaces the critic.

Now the mistakes beginners make. First, memory. The actor and the vLLM engine live on the same GPUs, so vLLM's gpu_memory_utilization is competing with the training weights and optimizer state, and setting it too high gives an out-of-memory error the moment training resumes after generation. The fix is to lower that fraction, or to enable the parameter and optimizer offload flags on the actor so the training state moves to CPU while vLLM generates. Second, batch-size arithmetic. data.train_batch_size is the number of prompts sampled per step, ppo_mini_batch_size sets how many samples make one gradient update inside the PPO epochs, and ppo_micro_batch_size_per_gpu is only the forward-and-backward chunk used for gradient accumulation. Confusing the three is the most common cause of a run that either wastes memory or silently changes the effective learning dynamics.

Third, the training-inference mismatch trap. The log probabilities vLLM returns during sampling are not bit-for-bit identical to the ones the training forward pass produces, so verl recomputes the behavior policy log probs with the actor's own forward before it forms the PPO ratio. If you ever see the importance ratio drift far from one on the very first inner epoch, this is usually why, and it is a property of the numerics, not a bug in your reward. Fourth, and most fundamental. Do not try to hand-place the generation and training on different GPU pools to make them tidy. Co-location is the design, and the moment you separate the rollout pool from the training pool you have paid for idle hardware and given up the resharding path that verl exists to provide.

Part III: When it is the right tool

verl is the right tool when you are doing serious RL post-training of language models and you care about throughput, scale, and the freedom to mix backends. It shines for reasoning RL with verifiable rewards, GRPO or PPO on math and code where a rule-based checker scores each answer, and it is the engine a large number of open reasoning-model reproductions are built on. It is also the natural choice when you need Megatron for a very large or mixture-of-experts model on one axis and vLLM or SGLang for fast rollout on another, because it treats both as swappable worker-internal backends rather than a fork of the whole trainer.

The honest cases for alternatives. Reach for TRL when your job is single-node or modest-scale and you want the shortest path from a Hugging Face checkpoint to a PPO, GRPO, or DPO run, because TRL's trainers are simpler to read and wire into the rest of the Transformers ecosystem. Reach for OpenRLHF when you want a Ray-plus-vLLM-plus-DeepSpeed stack that is a little smaller in surface area and you are comfortable on DeepSpeed rather than Megatron. Reach for NVIDIA's NeMo-Aligner or NeMo-RL when you already live in the NeMo and Megatron-Core world and want first-party support there. DeepSpeed-Chat is the older reference that taught the community the three-model RLHF shape, and it is worth reading for history more than for new projects. verl's distinguishing bet against all of them is the explicit single-controller programming model plus first-class co-location and weight resharding, which is exactly what you want at scale and slightly more machinery than you need for a quick single-node experiment.

The architecture-shaped warning is about where the two engines put their memory. During a step the same GPUs must hold, at different moments, the training weights and optimizer state on one side and the vLLM KV cache on the other. Get the split wrong and you do not get a clean error, you get either an out-of-memory crash or, worse, a run that technically fits but leaves almost no KV cache so generation crawls.

the co-location tension on one GPU

   [ actor params + grads + AdamW state ]   needed during training
   [ vLLM weights + KV cache            ]    needed during generation

   naive:  both resident always      -> OOM, or a tiny KV cache and slow rollout
   verl:   reshard + offload in phase -> training state parks on CPU while vLLM runs,
                                         vLLM cache frees before the optimizer step

The fix is the set of knobs Part II mentioned, vLLM's gpu_memory_utilization, the actor's parameter and optimizer offload options, and freeing the rollout cache between phases. This is the RLHF-scale version of the NFS-mounted-SQLite hazard, a configuration that runs but quietly destroys throughput, and reading a profiler or the memory logs is how you catch it.

Part IV: The full life of one training step

The specimen is one iteration of RayPPOTrainer.fit(), training a policy on GSM8K. I trace PPO with a critic, and I note where GRPO forks off. The controller code below is a faithful paraphrase of the loop in verl/trainer/ppo/ray_trainer.py, not a verbatim copy, and the method names on the worker groups are the real ones.

for batch_dict in self.train_dataloader:
    batch = DataProto.from_single_dict(batch_dict)
    gen_batch = batch.pop(["input_ids", "attention_mask", "position_ids"])

    # 1. rollout: reshard actor weights into vLLM, then sample responses
    gen = self.actor_rollout_wg.generate_sequences(gen_batch)
    batch = batch.union(gen)

    # 2. recompute the behavior-policy log probs with the TRAINING forward
    batch = batch.union(self.actor_rollout_wg.compute_log_prob(batch))

    # 3. reference-policy log probs, for the KL term
    batch = batch.union(self.ref_policy_wg.compute_ref_log_prob(batch))

    # 4. values from the critic  (PPO only; GRPO skips this)
    if self.use_critic:
        batch = batch.union(self.critic_wg.compute_values(batch))

    # 5. rewards: a rule-based verifier or a reward model
    batch = batch.union(compute_reward(batch, self.reward_fn))

    # 6. advantages: GAE for PPO, group-normalized for GRPO
    batch = compute_advantage(batch, self.config.algorithm)

    # 7 and 8. update critic, then update actor
    if self.use_critic:
        self.critic_wg.update_critic(batch)
    self.actor_rollout_wg.update_actor(batch)

Stage 1: launch and worker construction

python3 -m verl.trainer.main_ppo runs a Hydra main that loads ppo_trainer.yaml, applies your command-line overrides, and calls run_ppo. That function initializes Ray and launches a single TaskRunner Ray actor, which is where the driver logic runs so that it lives on the cluster rather than on your login shell. The runner builds a RayPPOTrainer and calls init_workers(). Inside init_workers, a ResourcePoolManager turns your requested cluster shape, n_gpus_per_node times nnodes, into one or more Ray placement groups, and each role is mapped onto a pool. In the default single-pool layout the actor-rollout-reference worker, the critic, and the reward model all land on the same GPUs, which is the co-location the whole design is built around. Each worker group is then a RayWorkerGroup, a set of Ray actors, one per GPU, that together run an SPMD program.

Stage 2: the single controller issues a call

Back on the driver, the loop begins. A batch of prompts comes off the dataloader as a plain dict and is wrapped into a DataProto, verl's transport object, which carries a batched TensorDict, a side dictionary of numpy arrays for non-tensor data like raw prompt strings, and a meta_info dict for scalars like the sampling temperature. The controller pops the fields the rollout needs and calls self.actor_rollout_wg.generate_sequences(gen_batch). This is the pivotal moment of the architecture. To the controller it looks like a plain method call that returns a DataProto. Under the hood the RayWorkerGroup consults the dispatch mode registered on that method, splits the batch into one chunk per worker, fires the remote calls, and concatenates the returned chunks. The controller stays blissfully unaware that anything was parallel.

Stage 3: generation, and the resharding that precedes it

Inside the actor-rollout worker, generate_sequences does not immediately call vLLM. First it enters a sharding manager context. On entry, that context takes the actor's current training weights, which are sharded the FSDP or Megatron way, and reshards them into the layout vLLM expects, loading them into the running inference engine without restarting it. Only then does vLLM generate, using continuous batching and PagedAttention to sample all the responses for the batch fast, and for GRPO it samples rollout.n responses per prompt. On exit, the sharding manager frees or parks the rollout state so the GPUs are ready for training math again. This enter-generate-exit dance is the beating heart of the system and gets its own deep dive. If you have read the vLLM or SGLang chapters, this is where those engines plug in, as a worker-internal backend behind rollout.name.

Stage 4: the behavior-policy log probs

The generated tokens now flow back to the controller inside the returned DataProto, unioned into the running batch. The controller then calls compute_log_prob on the same actor-rollout group. This recomputes the log probability of every generated token using the actor's training forward pass, not the sampler's numbers, and stores it as the old, behavior-policy log prob that the PPO ratio will divide by. Doing this deliberately, with the training engine, is how verl keeps the importance ratio honest despite the fact that vLLM and the FSDP forward do the math in subtly different ways. The reference policy log probs come next from a separate reference worker, or from the reference branch of the co-located worker, and feed the KL term that keeps the policy from wandering away from the base model.

Stage 5: values, rewards, and advantages

For PPO the critic worker computes a value for every token position through compute_values. GRPO skips this entirely, which is the point of GRPO, no value network to train. Rewards arrive next. In the verifiable-reward setting a reward function scores each full response against the ground-truth answer, a rule-based check for math correctness or a code test, and this runs cheaply on the driver side through a reward manager. If a learned reward model is configured instead, the reward-model worker group computes scores the same way the other workers do. With values and rewards in hand, the controller computes advantages. For PPO that is generalized advantage estimation over the per-token values. For GRPO it is the group calculation, subtract the group mean reward and divide by the group standard deviation, then broadcast that single scalar advantage across every token of the response. Both live in verl/trainer/ppo/core_algos.py.

Stage 6: the two updates

Finally the training. For PPO the critic updates first through update_critic, regressing the value head toward the returns, then the actor updates through update_actor, which runs the clipped PPO policy-gradient loss over ppo_epochs passes of ppo_mini_batch_size chunks, with ppo_micro_batch_size_per_gpu controlling gradient accumulation inside each. Both of those calls dispatch the advantage-annotated batch out to the SPMD workers, where the real FSDP or Megatron backward pass and optimizer step happen, and return metrics. The controller logs step, loss, KL, reward, and throughput, and loops. That closes one iteration, prompts in, sampled responses scored, advantages formed, and a gradient step taken on the policy, with the rollout engine and the trainer having taken turns on the very same GPUs.

Part V: Internals deep dives

Deep dive: the single controller and the dispatch decorator

The abstraction that makes the controller readable lives in verl/single_controller/. A worker method that the controller may call is marked with a decorator, and the decorator records how a batch should be split going in and reassembled coming out. The illustrative shape is this, and the exact import path may move as the package is refactored.

from verl.single_controller.base import Worker
from verl.single_controller.base.decorator import register, Dispatch

class MyActor(Worker):
    @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
    def compute_log_prob(self, data: DataProto) -> DataProto:
        # runs SPMD on every rank; each rank sees only its shard of the batch
        ...

# on the driver, the group call auto-dispatches and gathers:
#   log_probs = actor_rollout_wg.compute_log_prob(batch)   # one DataProto in, one out

When a RayWorkerGroup is built, it scans the worker class for these decorated methods and binds a matching method onto the group object. Calling actor_rollout_wg.compute_log_prob(batch) therefore does three things. It runs the dispatch function named by the mode, which for DP_COMPUTE_PROTO chunks the batch into one DataProto per data-parallel rank. It calls the method remotely on every worker with that worker's chunk. And it runs the collect function, which for the same mode concatenates the returned DataProto chunks back into one. Other modes cover the other communication shapes, ONE_TO_ALL to broadcast the same argument to every worker, and Megatron-flavored modes that understand that only the data-parallel dimension should be chunked while tensor-parallel and pipeline ranks receive the same slice. Because the split-and-gather policy is metadata attached to the method rather than code the controller writes, the RL loop can pass a whole batch to a thousand-GPU worker group and still read like it is calling a function on one machine.

The object that flows through all of this is DataProto, defined in verl/protocol.py. It is a small dataclass and it repays close reading. Its operations are exactly the ones a dataflow controller needs. chunk and concat are the split and gather that dispatch uses. union merges new columns into an existing batch, which is how each stage of Part IV adds its output, log probs, values, rewards, advantages, alongside the data that produced them. select and pop carve out the subset of fields a particular worker needs. There is also a future variant so the controller can pass a handle to a not-yet-computed result and let workers pipeline, rather than forcing the driver to block on every call.

from verl.protocol import DataProto
import torch

data = DataProto.from_dict(
    tensors={"input_ids": torch.randint(0, 100, (8, 16))},
    non_tensors={"raw_prompt": ["solve: ..."] * 8},
    meta_info={"temperature": 1.0},
)

halves = data.chunk(2)             # -> two DataProto, 4 rows each (what dispatch does)
whole  = DataProto.concat(halves)  # -> back to 8 rows      (what collect does)
whole  = whole.union(other)        # -> merge in a new column, e.g. old_log_probs

Deep dive: worker groups, roles, and resource pools

The role layer sits in verl/trainer/ppo/ray_trainer.py. A small Role enum names the parts of the algorithm, ActorRollout, Critic, RefPolicy, and RewardModel, and each maps to a worker class and a resource pool. The worker classes themselves live in verl/workers/, split by training backend. fsdp_workers.py holds the FSDP implementations, megatron_workers.py the Megatron ones, and the two expose the same method surface so the controller cannot tell them apart. The headline design choice is the actor-rollout-reference worker. Because the actor policy, the sampler that generates from it, and the reference policy all concern the same base weights, verl fuses them into one worker class that owns an FSDP or Megatron actor module, a vLLM or SGLang rollout engine, and a reference forward, all on the same GPUs. That fusion is what makes weight resharding a local operation rather than a network transfer.

The ResourcePoolManager decides which roles share GPUs. The default and most efficient layout puts every role on one global pool, so the actor-rollout worker, the critic, and the reward model all overlap on the same devices and take turns. verl also supports splitting them onto separate pools when a role is heavy enough to deserve dedicated hardware, for instance a large reward model, and it uses a colocation helper to pack several worker roles into one Ray actor when they must literally share a process. The mental model to hold is a grid. Roles are the rows, GPUs are the columns, and the resource-pool mapping decides which cells are filled, with full co-location filling every cell of every row on the same columns.

Deep dive: the hybrid engine and weight resharding

This is the idea the whole chapter has been circling, and it is what the HybridFlow paper calls the 3D-HybridEngine. During a step the same GPUs must serve two engines with incompatible layouts. The training engine shards a parameter one way, FSDP splits it across data-parallel ranks or Megatron splits it across tensor-parallel ranks, and the inference engine wants it laid out for vLLM's own tensor parallelism. A sharding manager bridges the two. It is a context manager, and the simplified usage inside the rollout path looks like this.

# simplified, inside the actor-rollout worker's generate path
with self.rollout_sharding_manager:            # __enter__: reshard training weights -> vLLM
    gen_output = self.rollout.generate_sequences(prompts)
# __exit__: free the rollout KV cache, restore the training state

The concrete managers live in verl/workers/sharding_manager/, with an FSDPVLLMShardingManager, a MegatronVLLMShardingManager, and SGLang counterparts, one per pairing of training backend and rollout backend. On __enter__ the manager gathers the training shards into the full parameters, converts names and layouts to what the inference engine expects, and updates the live vLLM or SGLang engine's weights in place, so the sampler always generates from the current policy rather than a stale copy. On __exit__ it releases the inference engine's memory, notably the KV cache, so the optimizer step that follows has room. Co-location plus resharding is why verl keeps every GPU busy. Instead of a generation cluster that idles while a training cluster works, one set of GPUs flips between the two roles, and the flip is a local memory reshuffle rather than a cross-machine weight transfer. The paper's contribution is doing that reshuffle with minimal communication by keeping as much of the resharding as possible within a node, and the practical payoff is the memory dance drawn in Part III, which those offload and cache-utilization knobs let you tune. The exact resharding internals shift as vLLM and SGLang evolve their weight-update APIs, so learn the enter-reshard-exit shape rather than any single method name.

Deep dive: the algorithms in core_algos

All the RL math is gathered in verl/trainer/ppo/core_algos.py, deliberately apart from the distributed machinery, which is why you can read the algorithm without reading a single collective. The advantage estimator is chosen by algorithm.adv_estimator, and the file implements several, generalized advantage estimation for PPO, the group-outcome estimator for GRPO, and estimators for REINFORCE-style variants like RLOO and REINFORCE++ and ReMax. The policy loss is the clipped PPO objective, with an optional dual-clip lower bound for very large ratios, and there is a matching clipped value loss for the critic. The KL term can appear in two places, and knowing which is which prevents a lot of confusion. It can be folded into the reward as a per-token penalty, the classic PPO-RLHF style controlled by algorithm.use_kl_in_reward, or it can be added directly to the actor loss, the GRPO style controlled by actor_rollout_ref.actor.use_kl_loss with its own coefficient and a low-variance estimator option.

The comparison to hold in your head is what each algorithm needs from the system. PPO trains a critic, so it pays for a whole extra model, its forward, its backward, and its optimizer state, but the critic gives per-token credit assignment through GAE. GRPO deletes the critic and buys its baseline with samples instead, drawing a group of responses per prompt and using their mean reward as the baseline, which trades critic memory and compute for extra generation. That is why GRPO runs set rollout.n to a group size larger than one and PPO does not, and it is why GRPO leans so heavily on the fast co-located rollout, since it lives or dies on how cheaply it can sample many responses. The deeper treatments of the objectives themselves live in the PPO and GRPO notes, and this chapter is about how verl schedules them across a cluster.

Part VI: Reading the repository

The repository is larger than a single trainer but it has a clear spine, and reading it in dependency order keeps you from drowning in the backend variants. Paths reflect a recent main; treat them as landmarks, since the tree is reorganized often.

Stage 0, orientation. Read the README and the docs on the programming model, then open verl/trainer/main_ppo.py. Questions to answer. What does run_ppo put inside a Ray actor and why, where does the Hydra config get assembled, and what object does the runner ultimately call fit() on.

Stage 1, the controller. Read verl/trainer/ppo/ray_trainer.py top to bottom, with init_workers and fit as the destinations. Questions. How are roles mapped onto resource pools, which worker groups exist for PPO versus GRPO, and how does each stage of the loop turn into a worker-group method call plus a union.

Stage 2, the single-controller core. verl/single_controller/, especially the base Worker and WorkerGroup, the register decorator and the Dispatch modes, and the Ray implementation of the worker group. Questions. What does DP_COMPUTE_PROTO do on the way in and the way out, how does a decorated method become a method on the group, and where would a new dispatch pattern be added.

Stage 3, DataProto. verl/protocol.py in full. Questions. What are the three payloads a DataProto carries, why is there a numpy side channel separate from the tensor batch, and what does the future variant let the controller overlap.

Stage 4, the workers. verl/workers/fsdp_workers.py first, since FSDP is the simpler backend to follow, then the parallel structure under verl/workers/, the actor/, critic/, rollout/, and reward_model/ packages. Questions. What methods does the actor-rollout-reference worker expose, how does generate_sequences wrap the rollout in a sharding manager, and what is genuinely shared when actor, rollout, and reference sit in one worker.

Stage 5, resharding and backends. verl/workers/sharding_manager/ for the hybrid engine, then verl/workers/rollout/ for the vLLM and SGLang integrations, then verl/trainer/ppo/core_algos.py for the math. Questions. What exactly happens on the manager's enter and exit, how does a rollout backend get selected by rollout.name, and which advantage estimators does core_algos implement.

Stage 6, the frontier. The recipe/ directory holds reference implementations of newer algorithms and training regimes, DAPO, PRIME, and others, each a thin layer over the same worker groups, and the Megatron worker path (verl/workers/megatron_workers.py) is where the very large and mixture-of-experts models live. Both are best met after the FSDP plus PPO plus GRPO story is solid.

Where not to start. The Megatron backend, the multi-turn and agentic rollout paths, and the async or one-step-off training modes all add real complexity that only makes sense once the synchronous FSDP loop is clear in your head. Read the plain path first, then add one axis of complexity at a time.

Part VII: Hands-on labs

Labs 1 through 3 fit on a single small GPU. Lab 4 needs none. Labs 5 and 6 want a couple of GPUs. Log formats vary with the fast pace of main, so match phases rather than exact strings.

Lab 1: one PPO run on GSM8K. Concept, the whole loop of Part IV.

python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k
python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=gae \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.rollout.name=vllm \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
  critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  trainer.n_gpus_per_node=1 trainer.nnodes=1 trainer.total_epochs=2

Watch the per-step logs and name each phase against Part IV, the generation time, the log-prob and value passes, the reward and KL, and the two update losses. The reward on GSM8K should climb as the policy learns to produce answers the checker accepts.

Lab 2: turn PPO into GRPO. Concept, dropping the critic for a group baseline.

python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=grpo \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.rollout.name=vllm \
  actor_rollout_ref.rollout.n=8 \
  actor_rollout_ref.actor.use_kl_loss=True \
  actor_rollout_ref.actor.kl_loss_coef=0.001 \
  trainer.n_gpus_per_node=1 trainer.nnodes=1 trainer.total_epochs=2

Notice there is no critic in the logs and no value loss, that generation now produces eight responses per prompt, and that the whole step spends proportionally more time in rollout. This is the compute tradeoff of GRPO made visible, more sampling in exchange for no critic.

Lab 3: swap the rollout backend. Concept, the rollout as a worker-internal choice.

# same run, SGLang instead of vLLM
python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=grpo \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.rollout.name=sglang \
  actor_rollout_ref.rollout.n=8 \
  trainer.n_gpus_per_node=1 trainer.nnodes=1 trainer.total_epochs=1

The controller code and the algorithm did not change at all, only the rollout.name. That is the decoupling of Part I in one command. Compare rollout throughput between the two backends if both install cleanly on your box.

Lab 4: DataProto with your bare hands. Concept, chunk, concat, union. CPU only.

# dataproto_lab.py
import torch
from verl.protocol import DataProto

data = DataProto.from_dict(
    tensors={"input_ids": torch.arange(32).reshape(8, 4)},
    non_tensors={"answer": [str(i) for i in range(8)]},
    meta_info={"temperature": 1.0},
)
parts = data.chunk(4)                       # simulate a 4-worker dispatch
print([p.batch["input_ids"].shape for p in parts])
back = DataProto.concat(parts)              # simulate the collect
print(back.batch["input_ids"].shape, back.non_tensor_batch["answer"][:3])
scores = DataProto.from_dict(tensors={"reward": torch.ones(8)})
merged = back.union(scores)                 # add a column, as each stage does
print(list(merged.batch.keys()))

Run it with plain python3 dataproto_lab.py. Predict the shapes before you run. This is the exact bookkeeping the dispatch layer does for you on every worker-group call, done by hand so you can see it.

Lab 5: watch co-location breathe. Concept, resharding and the memory dance.

python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=grpo \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-3B-Instruct \
  actor_rollout_ref.rollout.name=vllm \
  actor_rollout_ref.rollout.n=8 \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.5 \
  actor_rollout_ref.actor.fsdp_config.param_offload=True \
  actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
  trainer.n_gpus_per_node=2 trainer.nnodes=1 trainer.total_epochs=1

Watch GPU memory across a step with nvidia-smi in another terminal. It should rise during generation as the KV cache fills, fall as the cache is freed, and rise again as the optimizer state comes back for the update. Then push gpu_memory_utilization up until the run out-of-memories, and back it off. You are feeling the Part III tension directly.

Lab 6: a custom reward function. Concept, verifiable rewards.

# my_reward.py  -- verl calls this per sample with the response and ground truth
def compute_score(data_source, solution_str, ground_truth, extra_info=None):
    # return 1.0 for a correct final answer, else 0.0 (schema may vary by version)
    return 1.0 if ground_truth.strip() in solution_str else 0.0

python3 -m verl.trainer.main_ppo \
  algorithm.adv_estimator=grpo \
  data.train_files=$HOME/data/gsm8k/train.parquet \
  data.val_files=$HOME/data/gsm8k/test.parquet \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.rollout.name=vllm actor_rollout_ref.rollout.n=8 \
  custom_reward_function.path=$PWD/my_reward.py \
  custom_reward_function.name=compute_score \
  trainer.n_gpus_per_node=1 trainer.nnodes=1 trainer.total_epochs=1

Confirm the mean reward tracks your function. Then make the function stricter, require an exact numeric match rather than a substring, and watch the reward curve and the policy respond. The exact reward-function signature has shifted across versions, so check the reward manager for the current one before you rely on the argument names.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is verl, in one sentence?

ByteDance's HybridFlow RL post-training engine, which runs the RL algorithm as a single-controller dataflow on a driver while each heavy stage runs on an SPMD worker group, and which co-locates the rollout engine and the training engine on the same GPUs.

2. What does the single-controller / multi-controller split buy you?

Readability without giving up scale. The PPO or GRPO loop is plain sequential Python because the controller holds no weights and only calls named methods on worker groups, while each worker group is a normal one-process-per-GPU SPMD program doing FSDP, Megatron, or vLLM work. The algorithm and the engine are decoupled, so you can swap either without touching the other.

3. What is a DataProto and why does it matter?

The transport object every stage passes around, carrying a batched TensorDict, a numpy side dict for non-tensor data, and a meta_info dict. It matters because its chunk, concat, and union operations are exactly what the dispatch layer needs to split a batch across workers, gather the results, and merge each stage's new column back into the running batch.

4. What does the @register dispatch mode do?

It attaches to a worker method the policy for splitting the input and reassembling the output. DP_COMPUTE_PROTO chunks the batch into one piece per data-parallel rank on the way in and concatenates the returned pieces on the way out, so a single controller call fans out to the whole group and gathers back a single result transparently.

5. Why are actor, rollout, and reference fused into one worker?

Because they all concern the same base weights. Fusing them onto the same GPUs makes moving weights from the training layout into the inference engine a local memory operation rather than a network transfer, which is the entire reason co-location is fast.

6. What does a sharding manager do, and when?

It is a context manager around generation. On enter it reshards the actor's FSDP or Megatron weights into the layout vLLM or SGLang expects and loads them into the live engine, so sampling uses the current policy. On exit it frees the rollout KV cache and restores the training state so the optimizer step has memory.

7. Why does verl recompute the old log probs instead of trusting vLLM's?

Because the sampler's log probabilities are not numerically identical to the training forward pass, and the PPO ratio divides by the old log prob. Recomputing the behavior-policy log probs with the actor's own forward keeps the importance ratio honest and avoids a silent training-inference mismatch.

8. How does GRPO differ from PPO in what the system runs?

GRPO deletes the critic. There is no value model to forward, backprop, or store optimizer state for, and there is no GAE. Instead it samples a group of responses per prompt, uses their mean reward as the baseline, and normalizes by the group standard deviation. It trades critic memory and compute for extra generation, which is why it sets rollout.n above one and leans on fast co-located rollout.

9. Where can the KL term live, and why does it matter which?

It can be a per-token penalty folded into the reward, the classic PPO-RLHF style, or a term added straight to the actor loss, the GRPO style. It matters because the two are controlled by different config flags and combine differently with the advantage. Turning on both by accident double-counts the KL.

10. Why is co-locating rollout and training the load-bearing idea?

Because the alternative wastes hardware. Separate generation and training clusters mean one pool idles while the other works, and moving weights between them is a network transfer. Co-location keeps every GPU busy by having one set of devices flip between roles, with the flip a local reshard rather than a cross-machine copy.

11. When would you pick TRL, OpenRLHF, or NeMo-RL over verl?

TRL for single-node or modest-scale runs where the shortest path from a Hugging Face checkpoint matters more than throughput. OpenRLHF for a smaller Ray-plus-DeepSpeed stack. NeMo-Aligner or NeMo-RL when you already live in NeMo and Megatron-Core. verl wins when you want the explicit single-controller model, first-class co-location and resharding, and the freedom to mix FSDP or Megatron with vLLM or SGLang at scale.

12. A GRPO run fits in memory but generation is painfully slow. First suspect?

The co-location memory split. The training state is probably taking so much room that vLLM's KV cache is tiny, so it cannot batch many sequences at once. Lower gpu_memory_utilization conflicts by enabling parameter and optimizer offload during generation, or give the rollout more headroom, and re-measure.

13. How do you scale from one GPU to many nodes?

Mostly by changing trainer.n_gpus_per_node and trainer.nnodes and the tensor-parallel sizes of the backends. The controller loop does not change, because the worker groups absorb the scale and the dispatch layer keeps hiding it from the algorithm. Ray handles placing the workers across the cluster.

14. Where does the RL math live, and why is that separation deliberate?

In verl/trainer/ppo/core_algos.py, apart from the distributed code. The separation means you can read and modify the advantage estimators and the PPO or GRPO loss without touching a single collective, which is what makes verl a usable base for new algorithms in the recipe/ directory.

Part IX: Design lessons

Split control flow from compute. A single controller expresses the algorithm as sequential code, and SPMD worker groups do the parallel math behind named methods. This is the same instinct as an orchestrator driving stateless workers, or a query planner over execution operators. Keep the part a human must reason about small and sequential, and push the parallelism behind an interface.

Make the parallel plan metadata, not code. Attaching a dispatch mode to a method with a decorator means the controller writes one function call and the framework fills in the split and gather. Wherever a caller should stay ignorant of layout, the batching, the routing, the collection, put that policy in declarative metadata the way RPC frameworks and data-parallel libraries do.

Co-locate contended resources and reshard between phases. Two engines that never run at the same instant should share the hardware and hand it back and forth, not own separate copies. verl's hybrid engine is the pattern, and it shows up anywhere a system alternates between two costly modes, time-sharing a cache between read and write paths, or reusing a buffer pool across stages.

One transport object with algebraic operations. Everything flows as a DataProto, and chunk, concat, union, and select compose cleanly. A single well-designed data structure with a small closed set of operations beats a dozen bespoke message types, the same way a dataframe or a tensor beats ad hoc records in a pipeline.

Isolate the algorithm from the infrastructure. The RL math sits in one file with no distributed code, and the backends sit behind uniform worker interfaces. That is what lets a new algorithm be a thin recipe and a new backend be a drop-in, and it is the same mechanism-versus-policy separation that keeps business logic out of persistence code.

Configuration as a hierarchy of overrides. Hydra over a base YAML gives every knob a stable dotted name and a one-line override, so experiments are diffable and reproducible from the command that launched them. The lesson is to give configuration structure and a single source of truth rather than scattering flags.

Part X: Memorization framework

The one-sentence summary. verl runs the RL loop as a single controller on a driver, ships each stage as a DataProto to an SPMD worker group through a dispatch decorator, co-locates the actor's training engine and the vLLM or SGLang rollout on the same GPUs, and reshards weights between them with a sharding manager, so PPO and GRPO read like textbook code while running at cluster scale.

main_ppo.py -> run_ppo -> Ray -> TaskRunner -> RayPPOTrainer
  -> init_workers: ResourcePoolManager places roles on GPU pools (co-located by default)
  -> worker groups: ActorRolloutRef (FSDP/Megatron + vLLM/SGLang + ref), Critic, RewardModel
  -> fit(): generate -> old_logprob -> ref_logprob -> values -> reward -> advantage -> update
  -> each call: dispatch (chunk) -> SPMD workers -> collect (concat) a DataProto

The chain mapped to source.

launch          verl/trainer/main_ppo.py (Hydra + Ray + TaskRunner)
controller      verl/trainer/ppo/ray_trainer.py (RayPPOTrainer.fit / init_workers)
dispatch        verl/single_controller/ (Worker, WorkerGroup, register, Dispatch)
transport       verl/protocol.py (DataProto)
workers         verl/workers/fsdp_workers.py, megatron_workers.py
                actor/  critic/  rollout/  reward_model/
resharding      verl/workers/sharding_manager/ (FSDP/Megatron <-> vLLM/SGLang)
algorithms      verl/trainer/ppo/core_algos.py (GAE, GRPO, PPO loss, KL)
config          verl/trainer/config/ppo_trainer.yaml

Memorize these blocks.

• The split: single controller for the RL dataflow on the driver, multi-controller SPMD for compute in the worker groups.
• The worker groups: ActorRolloutRef (actor + rollout + reference, co-located), Critic (PPO only), RewardModel (optional).
• The transport: DataProto with chunk, concat, union, select, dispatched by @register modes like DP_COMPUTE_PROTO.
• The hybrid engine: training and rollout share GPUs, a sharding manager reshards weights into vLLM or SGLang on enter and frees the cache on exit.
• PPO vs GRPO: PPO has a critic and GAE, GRPO drops the critic and normalizes rewards within a group of rollout.n samples.

Part XI: Papers and further reading

The ideas in this walkthrough come from a small set of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Sheng et al., HybridFlow, A Flexible and Efficient RLHF Framework, EuroSys 2025. The paper verl implements, the single-controller over multi-controller hybrid and the 3D-HybridEngine resharding.
2. Schulman et al., Proximal Policy Optimization Algorithms, 2017. The clipped surrogate objective the trainer runs. Derived step by step in the PPO note and the deep reinforcement learning class on this site.
3. Shao et al., DeepSeekMath, Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. Introduces GRPO, the group-baseline objective verl popularized. The GRPO note on this site works the math.
4. Ouyang et al., Training language models to follow instructions with human feedback, 2022. The RLHF pipeline that made all of this matter.
5. DeepSeek-AI, DeepSeek-R1, Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. The reasoning-model recipe most verl reproductions chase. The training loops behind it are surveyed in the self-improving agents class.
6. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The rollout engine's core idea, covered in the vLLM walkthrough.
7. Shoeybi et al., Megatron-LM, Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019. The tensor-parallel training backend, covered in the Megatron-LM walkthrough.
8. Rajbhandari et al., ZeRO, Memory Optimizations Toward Training Trillion Parameter Models, 2019. The sharding arithmetic behind FSDP-style backends, derived in the DeepSpeed walkthrough.
9. Zhao et al., PyTorch FSDP, Experiences on Scaling Fully Sharded Data Parallel, 2023. The default training backend verl drives.
10. Moritz et al., Ray, A Distributed Framework for Emerging AI Applications, 2017. The substrate the single controller runs on, covered in the Ray walkthrough.

Part XII: Final takeaway

If the pieces below verl are the gap, the training backends it drives are their own chapters, FSDP, Megatron, and the composable parallelisms in the torchtitan walkthrough and the parallel computing class, and the rollout engines in the vLLM and SGLang chapters. The objectives themselves, the clipped policy gradient and the group baseline, are derived in the PPO and GRPO notes. Read those, then come back and read ray_trainer.py once more. It will read like plain PPO, which is the entire point.