OpenRLHF

Part I: The mental model

ray job submit ... -- python -m openrlhf.cli.train_ppo_ray
      |
      v
train(args)                build Ray placement groups, one RayActorGroup per role
      |
      +-- PolicyModelActor      actor policy,  DeepSpeed ZeRO-3      xN gpus
      +-- CriticModelActor      value model,   DeepSpeed ZeRO-3      xN gpus
      +-- ReferenceModelActor   frozen policy for the KL term        xN gpus
      +-- RewardModelActor      score head, or --reward.remote_url   xN gpus
      +-- RolloutRayActor       vLLM inference engines               xM engines
      |
      v
one PPO iteration (repeated)
   1. SamplesGenerator     -> vLLM engines generate responses (the rollout)
   2. RemoteExperienceMaker-> ref logprobs, actor logprobs, critic values, reward
   3. per-token KL, reward - kl*KL, advantages (gae / group_norm / reinforce)
   4. ppo_train            -> PolicyLoss (clipped) on actor, ValueLoss on critic
   5. broadcast_to_vllm    -> push the freshly trained actor weights into vLLM

The one-sentence identity. OpenRLHF is a Ray-orchestrated RLHF framework that splits the training loop into a generation system built on vLLM and a learning system built on DeepSpeed ZeRO-3, then reconnects them by broadcasting the updated policy weights into the inference engines after every step. A naive RLHF trainer generates responses with the same Hugging Face model it trains, calling model.generate under the training framework. That is correct and simple and slow, because autoregressive decoding under a training runtime has none of the batching tricks that make inference fast. OpenRLHF refuses that coupling. Generation is a separate service with its own memory, its own paged attention, and its own continuous batching, and the trainer's only job across the boundary is to keep that service loaded with current weights.

The load-bearing idea, said plainly. In RLHF the rollout dominates wall-clock time, so the highest-leverage move is to run rollouts on an engine that is good at inference instead of on the engine that is good at training. Every PPO or GRPO step spends most of its seconds generating long completions one token at a time. vLLM turns that phase into a throughput problem it already solves, with paged key-value caches and dynamic batching across many in-flight sequences, while DeepSpeed keeps doing what it is good at, which is sharding a large model's parameters, gradients, and optimizer state so the backward pass fits. Ray is the glue that lets these two systems live on the same GPUs or on different GPUs, whichever the cluster affords.

Two consequences follow. First, placement becomes a first-class configuration axis. The actor, critic, reference, and reward models are each a group of Ray actors, and you decide by flags whether they share GPUs or spread across the cluster, which is a very different design from a monolithic trainer that owns every device. Second, the reward signal is pluggable in a way that matters for reasoning work. A reward can come from a trained reward model, from an HTTP service, or from an ordinary Python function you write, which is how verifiable-reward training on math and code fits the same harness as classic preference-based RLHF. Everything here is verified against main in July 2026. OpenRLHF moves quickly and recently reorganized its command-line flags into grouped namespaces, so where a detail is likely to shift I say so and stay at concept level.

Part II: Using it

OpenRLHF is a Linux-and-NVIDIA-GPUs project. It leans on vLLM, DeepSpeed, Ray, and Hugging Face Transformers, and the intended way in is from PyPI or from source into a recent CUDA environment.

pip install openrlhf
# generation acceleration and the Ray path pull in vLLM:
pip install openrlhf[vllm]
# or from source, which is the honest choice while main moves fast:
git clone https://github.com/OpenRLHF/OpenRLHF
cd OpenRLHF
pip install -e .[vllm]

The non-RL trainers run under deepspeed directly. Supervised fine-tuning, reward-model training, and DPO each have a CLI module, and these are the right first runs because they exercise the model and data plumbing without Ray or vLLM in the way.

# supervised fine-tuning of a base model
deepspeed --module openrlhf.cli.train_sft \
  --actor.model_name_or_path meta-llama/Llama-3.1-8B \
  --data.prompt_dataset Open-Orca/OpenOrca \
  --ds.zero_stage 2 --ds.param_dtype bf16

# train a reward model from preference pairs
deepspeed --module openrlhf.cli.train_rm \
  --reward.model_name_or_path meta-llama/Llama-3.1-8B \
  --data.prompt_dataset OpenRLHF/preference_dataset_mixture2_and_safe_pku

The main event is Ray-based PPO with vLLM rollouts. You start a Ray cluster (a single node is fine to learn on), then submit the job. The command below places all four learned models and the vLLM engines onto the same set of eight GPUs by colocating them, which is the memory-efficient layout for a single node.

# one terminal: start Ray
ray start --head --node-ip-address 0.0.0.0

# another terminal: submit the PPO job
ray job submit --address="http://127.0.0.1:8265" \
  -- python3 -m openrlhf.cli.train_ppo_ray \
  --actor.num_nodes 1 --actor.num_gpus_per_node 8 \
  --critic.num_nodes 1 --critic.num_gpus_per_node 8 \
  --ref.num_nodes 1 --ref.num_gpus_per_node 8 \
  --reward.num_nodes 1 --reward.num_gpus_per_node 8 \
  --vllm.num_engines 4 --vllm.tensor_parallel_size 2 \
  --train.colocate_all \
  --actor.model_name_or_path OpenRLHF/Llama-3-8b-sft-mixture \
  --reward.model_name_or_path OpenRLHF/Llama-3-8b-rm-700k \
  --ds.zero_stage 3 --ds.param_dtype bf16 \
  --actor.adam.lr 5e-7 --critic.adam.lr 9e-6 \
  --algo.kl.init_coef 0.01 \
  --rollout.batch_size 1024 --train.batch_size 128 \
  --data.prompt_dataset OpenRLHF/prompt-collection-v0.1 \
  --data.input_key context_messages --data.apply_chat_template \
  --vllm.sync_backend nccl --ds.packing_samples

GRPO is the same launcher with the critic removed and the advantage estimator switched. GRPO does not learn a value function. It samples several completions per prompt and uses their reward statistics as the baseline, so you drop the --critic.* group, raise --rollout.n_samples_per_prompt, and select group_norm.

ray job submit --address="http://127.0.0.1:8265" \
  -- python3 -m openrlhf.cli.train_ppo_ray \
  --actor.num_nodes 1 --actor.num_gpus_per_node 8 \
  --ref.num_nodes 1 --ref.num_gpus_per_node 8 \
  --vllm.num_engines 4 --vllm.tensor_parallel_size 2 \
  --train.colocate_all --ds.enable_sleep --vllm.enable_sleep \
  --actor.model_name_or_path Qwen/Qwen2.5-7B \
  --reward.remote_url /path/to/reward_func.py \
  --algo.advantage.estimator group_norm \
  --rollout.n_samples_per_prompt 8 \
  --algo.kl.init_coef 1e-3 --algo.kl.use_loss \
  --rollout.batch_size 128 --train.batch_size 128 \
  --data.prompt_dataset zhuzilin/dapo-math-17k \
  --data.input_key prompt --data.label_key label \
  --ds.zero_stage 3 --ds.param_dtype bf16

The programmatic reward is worth seeing because it is why OpenRLHF is comfortable on verifiable tasks. Point --reward.remote_url at a Python file that defines a single function. It receives the full decoded queries, the original prompts, and the dataset labels, and it returns one scalar reward per query as a tensor. This is where you put a math checker, a code sandbox, or a unit-test harness.

# reward_func.py, passed as --reward.remote_url /path/to/reward_func.py
import torch

def reward_func(queries, prompts, labels):
    # queries is prompts + generated responses (decoded strings)
    # labels is the gold answer for each prompt
    rewards = []
    for q, gold in zip(queries, labels):
        answer = extract_boxed_answer(q)     # your own parsing
        rewards.append(1.0 if answer == gold else 0.0)
    return torch.tensor(rewards)

One caution on old documentation. OpenRLHF used to expose flat flags such as --actor_num_nodes, --advantage_estimator, --colocate_all_models, and --remote_rm_url, and most blog posts and older example scripts still use them. Current main groups them into namespaces, so --actor.num_nodes, --algo.advantage.estimator, --train.colocate_all, and --reward.remote_url are the shapes you will meet now. When an old tutorial fails on an unrecognized flag, this rename is almost always why, and the translation is mechanical.

Now the mistakes beginners make. First, forgetting that the reward path is required. PPO needs either a reward model through --reward.model_name_or_path or a remote reward through --reward.remote_url, and a run with neither has nothing to optimize toward. Second, colocation without sleep. When you pass --train.colocate_all to share GPUs between training and vLLM, you almost always also want --ds.enable_sleep and --vllm.enable_sleep so DeepSpeed can offload its states while vLLM generates and vLLM can release its cache while DeepSpeed trains, since otherwise both try to own the memory at once and you run out. Third, batch-size confusion. --rollout.batch_size counts prompts drawn per iteration and --train.batch_size counts samples per optimizer update, and with --rollout.n_samples_per_prompt greater than one the number of experiences is the rollout batch times the sample count, which is easy to under-provision. Fourth, and most fundamental. The vLLM engines hold their own copy of the policy weights, and they are only as current as the last weight sync, so if you disable or break the sync the model generates from stale parameters and training silently goes off-policy.

Part III: When it is the right tool

OpenRLHF is the right tool when you are doing online RL on language models at a scale where generation throughput and multi-GPU placement actually matter, and you want a codebase small enough to read and fork. That covers RLHF on 7B to 70B policies, reasoning-model training with GRPO or REINFORCE++ against verifiable rewards, and agentic rollouts where a custom environment produces the reward. Its sweet spot is a team that wants vLLM-fast generation and DeepSpeed-scale training without writing the Ray orchestration between them by hand.

The honest cases for alternatives. TRL from Hugging Face is the gentler on-ramp, tightly integrated with Transformers and PEFT, excellent for single-node experiments and for DPO-style offline methods, and it now has its own vLLM integration. Reach for it first when your job is small or when you want the shortest path from a Transformers checkpoint to a trained policy. verl from ByteDance is the most direct competitor at the high end. Its HybridFlow controller-worker model and aggressive resharding push peak throughput on very large clusters, at the cost of a heavier programming model. Reach for it when you are squeezing a thousand-GPU run and can pay for the complexity. DeepSpeed-Chat is the ancestor of this whole design and still instructive. It colocates generation and training on the same GPUs with a hybrid engine that reshards between the two phases, which OpenRLHF's paper contrasts against by decoupling generation onto vLLM instead. NeMo-Aligner is the choice when you are already on Megatron and want maximum performance on NVIDIA hardware with a framework that owns your model code. OpenRLHF sits deliberately in the middle, faster and more scalable than the simplest options and far more legible than the heaviest ones.

The architecture-shaped warning is about mapping placement onto physical GPUs and interconnect. You have two orthogonal choices, how many GPUs each role gets and whether roles share GPUs, and they interact with the vLLM tensor-parallel size and the interconnect. Colocation saves GPUs but serializes phases, since a colocated actor cannot train while its own GPUs are busy generating.

disaggregated:   every role owns its own GPUs, phases can overlap
   gpus 0-7  actor+ref (train)      gpus 8-11  critic+reward
   gpus 12-15 vLLM engines (generate)
   -> generation of the next batch can overlap training of this one
      (this is what --train.async_enable turns on)

colocated:       all roles share the same GPUs to save hardware
   gpus 0-7  actor, critic, ref, reward, AND vLLM engines
   -> must time-share: vLLM sleeps while DeepSpeed trains and vice versa
      (needs --ds.enable_sleep and --vllm.enable_sleep to fit in memory)

Neither layout is wrong. Disaggregation buys overlap and costs GPUs, colocation buys GPUs and costs overlap, and the async training mode exists precisely to reclaim the overlap that colocation gives up. The failure mode to avoid is choosing colocation for its GPU savings and then being surprised that step time does not improve when you add hardware, because the phases are still serialized on the shared devices.

Part IV: The full life of one PPO iteration

The specimen is one iteration of PPO on an 8B policy, launched with python3 -m openrlhf.cli.train_ppo_ray under a Ray job. Most of the machinery below is identical for GRPO and REINFORCE++. Where the path forks I follow both briefly.

Stage 1: train_ppo_ray and Ray placement

The entry point is train(args) in openrlhf/cli/train_ppo_ray.py. It parses the grouped config, then builds Ray placement groups, which are reservations of GPU bundles across the cluster. For each learned role it constructs a RayActorGroup (openrlhf/trainer/ray/launcher.py), a thin manager over a set of Ray actors of one type spread across the requested nodes and GPUs. The roles are the policy (PolicyModelActor), the value model (CriticModelActor), the frozen reference (ReferenceModelActor), and the reward model (RewardModelActor), each a subclass of BaseModelActor. Separately it calls create_vllm_engines (openrlhf/trainer/ray/vllm_engine.py) to stand up the inference engines as RolloutRayActor instances. The colocation flags decide whether these groups get overlapping bundles. With --train.colocate_all every group binds to the same GPUs, and with the default disaggregated layout each group gets its own. The vLLM engine handles are then passed into the policy actor's model init, because the policy is the one role that must talk to the engines to sync weights.

Stage 2: model init and the vLLM sync group

Each role's init_model_from_pretrained loads its model under a DeepspeedStrategy. The policy and critic are trainable and wrapped by DeepSpeed ZeRO-3, so their parameters, gradients, and optimizer state are sharded across that role's GPUs. The reference and reward models are inference only. The critic and reward models are not plain language models. They are built by get_llm_for_sequence_regression (openrlhf/models/model.py), which replaces the language-model head with a scalar value head (named score by default) so the network emits a number per position instead of a vocabulary distribution. The most important piece of setup happens in the policy actor. Its trainer ActorPPOTrainer (openrlhf/trainer/ray/ppo_actor.py) calls _init_vllm_sync_group, which creates a torch process group joining DeepSpeed rank 0 with every vLLM engine worker. This dedicated group is the private channel over which trained weights will later flow into the engines.

Stage 3: the rollout on vLLM

The iteration begins with generation, and generation does not happen on the training model. SamplesGenerator (openrlhf/trainer/ppo_utils/samples_generator.py) pulls a batch of prompts, expands each into n_samples_per_prompt copies, and dispatches them to the vLLM engines, which decode completions with paged attention and continuous batching across all the in-flight sequences at once. This is the phase that would have dominated a naive trainer and the whole reason vLLM is here. The engines return token sequences and their positions, and for agentic or multi-turn setups a pluggable executor (loaded from --train.agent_func_path) can drive several generate-and-act rounds before the sample is considered complete. What comes back is raw text-as-tokens with no log-probabilities and no rewards yet. Turning it into a training signal is the next stage.

Stage 4: making the experience

RemoteExperienceMaker (openrlhf/trainer/ppo_utils/experience_maker.py) takes the generated samples and fills in everything PPO needs. For each sample it requests, in parallel across the Ray actor groups, the actor's log-probabilities of the generated tokens under the current policy, the reference model's log-probabilities for the KL term, the critic's per-token values, and the reward model's score (or the remote reward if --reward.remote_url is set). Because these live on different actors, the maker dispatches the forward passes as remote calls and gathers the futures, so the reference, critic, and reward forwards can proceed concurrently rather than one after another. The results are packed into an Experience dataclass (openrlhf/trainer/ppo_utils/experience.py) whose fields carry a role tag so that sequences, log-probs, values, rewards, and masks stay aligned when batches are split and recombined.

Stage 5: KL, rewards, and advantages

With log-probs in hand the maker computes the per-token KL divergence between the current policy and the frozen reference, using one of the estimators selected by --algo.kl.estimator (the k1, k2, and k3 forms, where k3 is the low-variance unbiased estimator). The KL is folded into the reward, so the effective per-token reward is the task reward minus init_coef times the KL, which is what keeps the policy from drifting arbitrarily far from the reference to chase reward. Then compute_advantages_and_returns turns rewards and values into advantages and returns, and this is the single place the algorithm choice bites. For PPO it runs generalized advantage estimation over the critic's values. For GRPO (group_norm) it ignores the critic entirely and normalizes each sample's reward against the mean and standard deviation of its group of n_samples_per_prompt siblings. For REINFORCE++ (reinforce) it normalizes returns across the whole batch. The completed experiences go into a replay buffer (openrlhf/trainer/ppo_utils/replay_buffer.py).

Stage 6: ppo_train, the losses, and the optimizer

Now the learning system takes over. ActorPPOTrainer.ppo_train iterates over the buffer for --train.max_epochs passes, and each training_step computes the clipped policy objective with PolicyLoss (openrlhf/models/loss.py), the ratio of new to old log-probs clipped against the advantage, optionally with an explicit KL loss term when --algo.kl.use_loss is set and an entropy bonus. In parallel the critic actor runs its own CriticPPOTrainer (openrlhf/trainer/ray/ppo_critic.py) with ValueLoss, the clipped regression of predicted values toward the returns. Both updates are ordinary DeepSpeed backward-and-step calls, so ZeRO-3 all-gathers each shard's parameters for the backward, reduce-scatters the gradients, and the optimizer touches only its slice of the state. For GRPO and REINFORCE++ the critic branch is simply absent, which is much of why those methods are cheaper. They trade a learned baseline for a statistical one and delete an entire trainable model.

Stage 7: broadcasting the new weights into vLLM

The iteration is not finished when the optimizer steps, because the vLLM engines still hold the previous policy. The last act is broadcast_to_vllm. It walks the policy's named parameters and, for each one, publishes it to the engines over the sync group created in Stage 2. Under ZeRO-3 the parameter is sharded across the actor GPUs, so the code first gathers the full tensor to rank 0 with deepspeed.zero.GatheredParameters, then broadcasts it, while each vLLM worker's WorkerWrap.update_weight (openrlhf/trainer/ray/vllm_worker_wrap.py) receives the tensor by name, dtype, and shape and loads it into the running engine in place. When training and vLLM are colocated on the same physical GPUs, a faster CUDA-IPC path shares the tensors through device memory handles instead of a network broadcast. This sync is the seam of the whole design. It is the one moment the two systems touch, and it is what keeps the fast generation engine faithful to the slowly learned policy. Once it returns, the engines are current and the next iteration's rollout is on-policy again. That closes the loop of one step, prompts in, gradients across the actors, weights back out to the engines.

Part V: Internals deep dives

Deep dive: Ray placement and the actor groups

Ray is doing two jobs here, and it helps to keep them separate. The first is scheduling. A placement group reserves bundles of GPUs, and each RayActorGroup launches one Ray actor per GPU it was given, so a role with --actor.num_nodes 2 --actor.num_gpus_per_node 8 becomes sixteen PolicyModelActor processes that together form one DeepSpeed world. The second job is remote invocation. The group exposes helpers like async_run_method that call a method on every actor and return futures, which is how the driver kicks off training or how the experience maker fans forward passes out to the reference and reward groups at once. The colocation flags reduce to a placement decision. Colocated roles are assigned overlapping bundles so their actors land on the same GPUs, and disaggregated roles get disjoint bundles. Nothing about the model code changes between the two, which is exactly the property that makes placement a configuration axis rather than a rewrite. If you have met device meshes in torchtitan, this is the looser, coarser cousin. There the unit of placement is a tensor's shard, here it is a whole model replica, and the parallel computing intuition about matching communication frequency to interconnect carries straight over.

Deep dive: why decoupling generation is the whole game

It is worth being precise about why vLLM matters so much, because it is the claim the framework is built on. Consider where the time goes in one PPO step. Generation produces, say, a thousand tokens per sample autoregressively, one forward per token, with a key-value cache that grows every step. Training does a fixed handful of forward-and-backward passes over the already-generated sequence. The generation cost scales with sequence length and sample count and has terrible arithmetic intensity, since each decode step is a tall-skinny matmul that barely uses the GPU. This is exactly the regime vLLM was built for. Paged attention stores the cache in non-contiguous blocks so many sequences of different lengths share memory efficiently, and continuous batching keeps the GPU full by admitting and retiring sequences mid-flight instead of waiting for a whole batch to finish. The same reasoning drives the sister project SGLang, and the memory story behind the attention kernels is the tiling argument in the FlashAttention chapter.

The cost of decoupling is the weight sync, and it is a real cost worth respecting. Every step you must move a full copy of the policy's parameters from the training actors into the inference engines. For an 8B model in bfloat16 that is roughly sixteen gigabytes crossing the sync group each iteration. The design bet is that this one bulk transfer per step is far cheaper than the throughput you lose by generating on a training-shaped runtime, and for long completions that bet wins decisively. The ZeRO-3 wrinkle is that no single actor holds a whole parameter, so the gather-then-broadcast in broadcast_to_vllm is not incidental, it is the price of sharding the trainer. The colocated CUDA-IPC path exists to shrink that price to near zero when the engines sit on the same silicon as the trainer, since then the weights never leave the GPU, only handles to them do.

Deep dive: the algorithms and the advantage estimator

OpenRLHF keeps one training loop and swaps algorithms almost entirely through how advantages are computed. The policy loss is shared. What differs is the baseline that turns a reward into an advantage, and whether a critic exists at all.

PPO is the classic actor-critic method described in the PPO chapter. It pays for a second trainable model, the critic, and in exchange gets a per-token learned baseline that reduces variance. GRPO, covered on the GRPO page, is the reasoning-era favorite. It deletes the critic and replaces the learned baseline with the mean reward of a group of completions to the same prompt, so the advantage of a sample is just how much better than its siblings it did, normalized by the group spread. That is dramatically cheaper and works beautifully when you can afford several samples per prompt and have a clean reward, which is the verifiable-math and code setting. Dr. GRPO removes the standard-deviation normalization to avoid a length and difficulty bias that the division can introduce. REINFORCE++ is the OpenRLHF authors' own variant, a plain policy gradient hardened with PPO's engineering tricks and a global reward normalization for stability, and its baseline form borrows the group mean from GRPO for reasoning tasks. The lesson underneath the table is the one from policy gradients. All of these are the same gradient with different baselines, and the baseline only reduces variance without changing the expected update, which is why swapping estimators is a one-flag change rather than a new trainer. OpenRLHF also carries an importance-sampling correction (--algo.advantage.is_correction_enable) for the gap between the vLLM policy that generated a sample and the actor policy that trains on it, which matters most under async or partial rollouts where that gap can widen.

Deep dive: reward models, remote rewards, and the KL leash

The reward is the most swappable part of the system, and the three ways to supply it map onto three research styles. A trained reward model (--reward.model_name_or_path) is classic RLHF, a network with a scalar head fit on human preference pairs by train_rm, and it runs as its own RewardModelActor. A remote reward (--reward.remote_url pointing at an HTTP endpoint) lets the reward live in a separate service, which you launch with openrlhf.cli.serve_rm. That decoupling is useful when the reward model is large or shared across jobs, since the training run holds no reward GPUs at all and simply posts queries to the server. A programmatic reward (--reward.remote_url pointing at a Python file) is the verifiable-reward path from Part II, an ordinary reward_func(queries, prompts, labels) that returns a tensor, which is how a math grader or a code sandbox becomes the reward without training anything. All three feed the same experience maker, so the rest of the pipeline cannot tell them apart.

The reference model and the KL term deserve a note because beginners often misread them. The reference is a frozen snapshot of the starting policy, and the KL divergence between the live policy and that snapshot is a leash. Without it the policy would happily collapse onto whatever degenerate text maximizes the reward model, a phenomenon called reward hacking, so the effective reward subtracts init_coef times the KL to penalize drift. You can apply that penalty two ways in OpenRLHF, baked into the reward before advantage estimation, or added as an explicit loss term with --algo.kl.use_loss, and the controller can hold the coefficient fixed or adapt it toward a target KL. The reference model is not there to be accurate, it is there to be a fixed point the policy is not allowed to run too far from, which is why it never trains and never syncs. The relationship to the offline alternative is worth keeping in view. DPO folds this same reference-anchored objective into a single supervised loss with no rollouts at all, and OpenRLHF ships a DPO trainer for exactly the jobs that do not need online generation.

Deep dive: DeepSpeed ZeRO-3, colocation, and sleep

The training side is DeepSpeed, and ZeRO-3 is the setting that makes large policies fit. It shards parameters, gradients, and optimizer state across the actors of a role, gathering each layer's parameters just in time for its forward and backward and releasing them after, so no actor ever holds the whole model. This is the same family of idea as FSDP in torchtitan, applied here to the actor and critic. On top of it OpenRLHF adds the memory tricks that make colocation viable. --ds.adam_offload pushes optimizer state to CPU, --ds.packing_samples packs variable-length sequences to avoid padding waste, and gradient checkpointing trades compute for activation memory. The piece that makes all-on-one-node colocation actually fit is sleep. With --ds.enable_sleep and --vllm.enable_sleep the trainer offloads its states while vLLM generates, then vLLM releases its KV cache while the trainer runs, so the two systems take turns owning the GPU memory rather than fighting over it. Under the hood this is the same time-sharing that DeepSpeed-Chat's hybrid engine did, rebuilt on top of vLLM's sleep support, and it is why the single-node colocated recipe in Part II can put four models and several inference engines on eight GPUs without an out-of-memory crash.

Deep dive: async and partial rollouts

The default loop is synchronous, so in a disaggregated layout the training GPUs sit idle while generation runs. --train.async_enable breaks that lockstep and lets the vLLM engines run the next batch's rollout while the trainer still learns from the current one, with --train.async_queue_size bounding the staleness you tolerate, and partial rollout pauses and resumes long generations so one very long sample does not stall a whole batch. Async training is the framework admitting that on-policy purity is a spectrum, not a binary, and that a little staleness bought back as throughput is usually a good trade. The cost is a wider gap between the sampling and training policies, which is what the importance-sampling correction absorbs, so these knobs belong with REINFORCE++ or GRPO rather than a strict textbook PPO.

Part VI: Reading the repository

The tree is small enough to read in an afternoon, which is much of its value. All paths are as of main in July 2026, and the recent flag rename means some module internals may still be settling.

Stage 0, orientation. Read the README.md, then an example script such as examples/scripts/train_ppo_ray_hybrid_engine.sh and train_reinforce_baseline_hybrid_engine.sh. Questions. Which flags choose placement versus algorithm versus data, what does ray job submit add over a bare python -m, and which roles appear in a GRPO command that are absent versus a PPO command?

Stage 1, the driver. Read openrlhf/cli/train_ppo_ray.py and its train(args) function, then openrlhf/trainer/ray/launcher.py for RayActorGroup and the BaseModelActor hierarchy. Questions. How does a placement group become GPU bundles, how does a role become one Ray actor per GPU, and where do the vLLM engine handles get handed to the policy actor?

Stage 2, the policy actor and the sync. openrlhf/trainer/ray/ppo_actor.py is the heart of the RL path. Read _init_vllm_sync_group, ppo_train, training_step, and broadcast_to_vllm together, then the vLLM side in vllm_engine.py and vllm_worker_wrap.py. Questions. What exactly is in the sync process group, why does ZeRO-3 force a gather before the broadcast, and when does the CUDA-IPC path replace the network broadcast?

Stage 3, experience and advantages. openrlhf/trainer/ppo_utils/samples_generator.py, then experience_maker.py with make_experience and compute_advantages_and_returns as the destination, then experience.py, replay_buffer.py, and kl_controller.py. Questions. Which forward passes run on which actors, how is the KL folded into the reward, and which lines change between gae, group_norm, and reinforce?

Stage 4, models and losses. openrlhf/models/actor.py for the Actor wrapper, model.py for get_llm_for_sequence_regression and the value head, and loss.py for PolicyLoss, ValueLoss, and the DPO and reward-model losses. Questions. What makes the critic and reward models different from the policy, and how does PolicyLoss implement the clipped objective?

Stage 5, the non-RL trainers and the frontier. The train_sft, train_rm, and train_dpo CLIs and their trainers under openrlhf/trainer/, then the async path in ppo_trainer_async.py and the agent hooks in openrlhf/utils/agent.py. This is where multi-turn and environment-driven rewards live.

Where not to start. The ring-attention and sequence-balancing utilities (openrlhf/models/ring_attn_utils.py, openrlhf/utils/seqlen_balancing.py) are a performance layer that matters only for very long contexts, and the vision-language and agent executors add moving parts you should meet after the dense text PPO path is solid.

Part VII: Hands-on labs

Labs 1 and 2 need no GPU and build intuition. Labs 3 through 6 want a single multi-GPU node. Log formats vary with the fast pace of main.

Lab 1: write a reward function and predict the shape. Concept: the programmatic reward contract of Part V.

# quiz yourself before running anything
import torch
def reward_func(queries, prompts, labels):
    # queries: list of decoded prompt+response strings
    # prompts: list of the original prompts
    # labels:  list of gold answers
    return torch.tensor([float(str(g) in q) for q, g in zip(queries, labels)])
# Q: if the rollout batch is 128 prompts and n_samples_per_prompt is 8,
#    what is len(queries) on one call? (answer: it depends on batching,
#    but the tensor length must equal len(queries) every time)

The exercise is to internalize that the reward is per-query and returns exactly one scalar per query as a tensor. Get that contract wrong and the experience maker cannot align rewards to samples.

Lab 2: draw the placement for a command. Concept: roles, GPUs, and colocation from Part IV.

# given only this fragment, sketch which GPUs run what
--actor.num_gpus_per_node 8 --critic.num_gpus_per_node 8 \
--ref.num_gpus_per_node 8 --reward.num_gpus_per_node 8 \
--vllm.num_engines 4 --vllm.tensor_parallel_size 2 \
--train.colocate_all
# then remove --train.colocate_all and redraw. how many GPUs
# does each layout need at minimum?

Colocated, all five things time-share one set of eight GPUs. Disaggregated, each role wants its own, and four vLLM engines at tensor-parallel two want eight more. The point is to feel how colocation trades GPUs for serialized phases.

Lab 3: a real GRPO run on a small model. Concept: the critic-free path.

ray start --head
ray job submit --address="http://127.0.0.1:8265" \
  -- python3 -m openrlhf.cli.train_ppo_ray \
  --actor.num_nodes 1 --actor.num_gpus_per_node 4 \
  --ref.num_nodes 1 --ref.num_gpus_per_node 4 \
  --vllm.num_engines 2 --vllm.tensor_parallel_size 2 \
  --train.colocate_all --ds.enable_sleep --vllm.enable_sleep \
  --actor.model_name_or_path Qwen/Qwen2.5-1.5B-Instruct \
  --reward.remote_url /path/to/reward_func.py \
  --algo.advantage.estimator group_norm \
  --rollout.n_samples_per_prompt 8 --algo.kl.init_coef 1e-3 \
  --data.prompt_dataset zhuzilin/dapo-math-17k \
  --data.input_key prompt --data.label_key label \
  --ds.zero_stage 3 --ds.param_dtype bf16

Watch the logs for the absence of any critic actor, the group reward statistics, and the reward climbing as the policy learns to satisfy your reward_func. Then flip --algo.advantage.estimator to reinforce_baseline and compare stability.

Lab 4: prove the weight sync is happening. Concept: the seam of Part IV, Stage 7.

# run any ppo command with a tiny model, then reason about the logs:
#  - the vLLM engines are created once at startup
#  - every iteration ends with a weight broadcast into them
# disable or slow the sync in your head: what would the reward curve do
# if the engines kept generating from the initial weights forever?

The intended realization is that without the sync the rollouts stay on the initial policy while the training model moves away, so the ratio in PolicyLoss grows without bound and the run destabilizes. The sync is not an optimization, it is correctness.

Lab 5: PPO versus GRPO cost. Concept: the price of the critic.

# PPO: keep the critic
--critic.num_gpus_per_node 4 --algo.advantage.estimator gae
# GRPO: delete the critic, add samples
# (drop the --critic.* group) --algo.advantage.estimator group_norm \
--rollout.n_samples_per_prompt 8

Compare peak memory and step time. PPO carries a second trainable model and its optimizer state. GRPO carries none of that but generates several samples per prompt, so it moves the cost from training memory into generation throughput, which is precisely the resource vLLM is cheapest at.

Lab 6: colocation with and without sleep. Concept: the memory time-share of Part V.

# first without sleep, on a node that is tight on memory:
--train.colocate_all
# then add sleep and rerun:
--train.colocate_all --ds.enable_sleep --vllm.enable_sleep

Without sleep, DeepSpeed and vLLM both try to hold their memory at once and the run is likely to hit an out-of-memory error on a tight node. With sleep they take turns and it fits. This is the single most common colocation gotcha made visible.

Part VIII: Questions and model answers

Understanding checks. Answer aloud before reading.

1. What is OpenRLHF, in one sentence?

A Ray-orchestrated RLHF framework that runs generation on dedicated vLLM engines and training on DeepSpeed ZeRO-3 actors, placing the policy, critic, reference, and reward models across a cluster and syncing fresh policy weights into the engines after every step.

2. Why decouple generation from training at all?

Because rollouts dominate the wall-clock time of an RLHF step and autoregressive decoding is an inference workload, not a training one. vLLM gives paged attention and continuous batching that a training runtime lacks, so running rollouts there and only paying a weight sync per step is far faster than generating on the training model.

3. What crosses the boundary between the two systems, and how?

The policy weights. After each update broadcast_to_vllm gathers each ZeRO-3-sharded parameter to rank 0 and broadcasts it over a dedicated process group to the vLLM workers, which load it in place with update_weight. Colocated engines can use a CUDA-IPC path instead of a network broadcast.

4. How does GRPO differ from PPO in this codebase?

GRPO sets --algo.advantage.estimator group_norm, deletes the critic, and samples several completions per prompt. The advantage of a sample is its reward normalized against the mean and standard deviation of its group, so a statistical baseline replaces the learned value function and an entire trainable model disappears.

5. What is REINFORCE++ and when would you use it?

The OpenRLHF authors' policy-gradient variant, a critic-free method with global reward normalization and PPO-style engineering tricks. Its baseline form borrows the group mean from GRPO. It is a good default for reasoning tasks with verifiable rewards where you want GRPO's cheapness without committing to its exact normalization.

6. What are the three ways to supply a reward?

A trained reward model through --reward.model_name_or_path, an HTTP service through --reward.remote_url pointing at a URL served by openrlhf.cli.serve_rm, or a Python reward_func(queries, prompts, labels) through --reward.remote_url pointing at a file. All three feed the same experience maker.

7. What is the reference model for, and why does it never sync?

It is a frozen snapshot of the starting policy, used to compute the KL divergence that leashes the live policy so it cannot drift off to hack the reward. It never trains and never receives synced weights precisely because its value is being a fixed anchor, not being current.

8. Why does colocation usually need sleep mode?

Colocated training and generation share the same GPU memory, and DeepSpeed's states plus vLLM's KV cache do not both fit at full size. Sleep lets each release its memory while the other runs, so --ds.enable_sleep and --vllm.enable_sleep make the time-share fit instead of hitting out-of-memory.

9. What does async training buy and cost?

It overlaps the next rollout with the current update so disaggregated GPUs stop idling, bounded by --train.async_queue_size. The cost is off-policy staleness between the sampling and training policies, which is why it pairs with an importance-sampling correction and with methods that tolerate a little staleness.

10. Where does the algorithm choice actually take effect in the code?

Almost entirely in compute_advantages_and_returns in the experience maker. The rollout, the forward passes, and the clipped policy loss are shared. Only the baseline that converts rewards and values into advantages, and whether a critic exists, change with the estimator flag.

11. When would you pick TRL or verl over OpenRLHF?

TRL for small or single-node jobs and for offline methods like DPO, where its tight Transformers and PEFT integration is the fastest path. verl when you are pushing peak throughput on very large clusters and can absorb its heavier controller-worker model. OpenRLHF sits between them, scalable and fast but still legible.

12. A colocated run does not get faster when you add GPUs. Why might that be?

Because colocation serializes generation and training on the shared devices, so adding GPUs to a colocated layout can widen each phase without letting the phases overlap. The fix is either a disaggregated layout so the phases run on different GPUs, or async training so the next rollout overlaps the current update.

13. Why must the critic and reward models be built differently from the policy?

They emit a scalar per position, not a distribution over the vocabulary, so get_llm_for_sequence_regression replaces the language-model head with a value head. The policy keeps its full vocabulary head because it must produce token log-probabilities for the loss and for generation.

14. Why can OpenRLHF put a verifiable-reward reasoning run and a classic preference-RLHF run in the same harness?

Because the reward is abstracted behind a single interface that returns a scalar per query. A reward model, an HTTP service, and a Python grader are interchangeable behind it, so the only thing that changes between the two styles is what supplies the number, not the training loop that consumes it.

Part IX: Design lessons

Split a workload along its performance seams. RLHF has a generation half that wants an inference engine and a training half that wants a sharding engine, and OpenRLHF runs each on the system built for it rather than compromising with one runtime for both. The same instinct separates read replicas from write primaries, or an OLAP store from an OLTP one. When two phases have opposite resource profiles, give them different engines and pay only for the seam between them.

Make the seam explicit and cheap. The one place the two systems touch is a per-step weight broadcast, and the framework invests in making it fast, with a dedicated process group, a ZeRO-aware gather, and a CUDA-IPC shortcut for colocated engines. Wherever a design decouples components, the interface between them becomes the thing to engineer, not an afterthought.

Make placement a configuration, not a rewrite. Because each role is a group of Ray actors, colocated or disaggregated is a flag, and the same code runs on one node or on a hundred. Representing deployment topology as data rather than as branching code is what lets one framework serve a laptop-scale experiment and a cluster-scale run without forking.

Keep one loop, vary the baseline. PPO, GRPO, REINFORCE++, and RLOO share almost all their code and differ mainly in how advantages are computed. Recognizing that these are one algorithm with different variance-reduction baselines keeps the trainer small and makes a new method a small, testable change rather than a new system.

Abstract the reward behind one interface. A trained model, a web service, and a Python function are interchangeable reward sources because they all return a scalar per query. Putting the pluggable, research-specific part behind a narrow contract is what lets the stable core stay stable while the reward, which is where the science is, changes freely.

Admit that on-policy is a spectrum. Async and partial rollouts trade a bounded amount of staleness for throughput, and a correction term pays for the gap. Systems that let you dial a strict invariant down to a tunable one, with a knob and a compensating mechanism, tend to beat systems that treat the invariant as sacred and leave the throughput on the table.

Part X: Memorization framework

The one-sentence summary. OpenRLHF places the policy, critic, reference, and reward models as Ray actor groups, runs the rollout on separate vLLM engines, builds experiences with per-token KL and an advantage estimator that selects the algorithm, updates the actor and critic under DeepSpeed ZeRO-3, and broadcasts the new policy weights back into the engines every step.

train_ppo_ray.train(args) -> Ray placement groups -> RayActorGroup per role
  -> PolicyModelActor + CriticModelActor + ReferenceModelActor + RewardModelActor
  -> create_vllm_engines (RolloutRayActor) + _init_vllm_sync_group
  -> loop:
       SamplesGenerator (vLLM rollout)
       RemoteExperienceMaker (ref/actor logprobs, critic values, reward)
       KL + reward - kl*KL + compute_advantages_and_returns
       ppo_train: PolicyLoss (actor) + ValueLoss (critic)
       broadcast_to_vllm (gather under ZeRO-3, then push to engines)

The chain mapped to source:

launch        openrlhf/cli/train_ppo_ray.py (train)
placement     openrlhf/trainer/ray/launcher.py (RayActorGroup, BaseModelActor)
policy actor  openrlhf/trainer/ray/ppo_actor.py (ActorPPOTrainer, broadcast_to_vllm)
vllm side     openrlhf/trainer/ray/vllm_engine.py, vllm_worker_wrap.py (WorkerWrap)
rollout       openrlhf/trainer/ppo_utils/samples_generator.py
experience    openrlhf/trainer/ppo_utils/experience_maker.py, experience.py
models        openrlhf/models/actor.py, model.py (get_llm_for_sequence_regression)
losses        openrlhf/models/loss.py (PolicyLoss, ValueLoss)

Memorize these blocks:

• The five roles: policy (trains), critic (trains, PPO only), reference (frozen, KL anchor), reward (score or remote), vLLM engines (generate).
• The seam: every step ends with a weight broadcast into the engines, ZeRO-3 gathers each param first, colocation can use CUDA IPC.
• The estimators: gae (PPO, has critic), group_norm (GRPO), dr_grpo, reinforce and reinforce_baseline (REINFORCE++), rloo, all critic-free except gae.
• Placement: per-role num_nodes and num_gpus_per_node, colocate_all or colocate_actor_ref or colocate_critic_reward, colocation needs ds and vllm sleep to fit.
• Reward paths: reward.model_name_or_path, or reward.remote_url as an HTTP endpoint, or reward.remote_url as a Python reward_func(queries, prompts, labels).

Part XI: Papers and further reading

The ideas in this walkthrough come from a small set of papers, and each one rewards a direct read. Where this site derives the same idea in depth, the companion link points there.

1. Hu et al., OpenRLHF, An Easy-to-use, Scalable and High-performance RLHF Framework, 2024. The paper behind this repository, the Ray plus vLLM plus DeepSpeed design and its scheduling argument against squeezing all four models into one runtime.
2. Schulman et al., Proximal Policy Optimization Algorithms, 2017. The clipped objective that PolicyLoss implements. Derived step by step in the PPO note and the deep reinforcement learning class on this site.
3. Shao et al., DeepSeekMath, Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. Introduces GRPO, the group_norm estimator here. The GRPO note on this site works the math.
4. Hu, REINFORCE++, A Simple and Efficient Approach for Aligning Large Language Models, 2025. The OpenRLHF authors' own critic-free method, the reinforce and reinforce_baseline estimators.
5. Ahmadian et al., Back to Basics, Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, 2024. The case for simple policy gradients that the rloo estimator implements. The policy gradients note builds the underlying math.
6. Ouyang et al., Training language models to follow instructions with human feedback, 2022. The InstructGPT pipeline, SFT then reward model then PPO, that this launcher automates end to end.
7. Rafailov et al., Direct Preference Optimization, Your Language Model is Secretly a Reward Model, 2023. The offline alternative OpenRLHF ships as train_dpo, covered in the DPO note.
8. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023. The rollout engine's core idea, covered in the vLLM walkthrough.
9. Rajbhandari et al., ZeRO, Memory Optimizations Toward Training Trillion Parameter Models, 2019. The sharding that makes the training side fit, covered in the DeepSpeed walkthrough.
10. Moritz et al., Ray, A Distributed Framework for Emerging AI Applications, 2017. The actor and placement substrate, covered in the Ray walkthrough.
11. Yao et al., DeepSpeed-Chat, Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales, 2023. The ancestor design whose hybrid engine OpenRLHF contrasts with by decoupling generation onto vLLM.
12. Sheng et al., HybridFlow, A Flexible and Efficient RLHF Framework, EuroSys 2025. The competing single-controller design behind verl, covered in the verl walkthrough.

Part XII: Final takeaway

If the reinforcement-learning pieces underneath this repository are the gap, the RL section builds them from the ground up, with dedicated chapters on policy gradients, PPO, and GRPO, and the generation engine that makes the whole design pay off is the subject of the vLLM chapter. Then come back and read ppo_actor.py once more, watching the rollout, the loss, and the broadcast go by in order. It will read like two ordinary systems with one careful handshake between them, which is the entire point.