Reinforcement learning from first principles up to the methods training today's reasoning models. The foundations pages derive each algorithm and implement it in PyTorch and JAX behind a language switcher, runnable on gymnasium environments. The language-model pages cover the algorithms that fine-tune LLMs with rewards and preferences, and the library walkthroughs read the codebases that run this at scale, verl and TRL, the same way the open source chapters read vLLM and PyTorch.
Featured pageHow group relative policy optimization drops PPO's value network by normalizing rewards within a group of sampled completions, the KL-to-reference term, a worked numeric example, and the loss implemented in PyTorch and JAX.
Read the page →The core algorithms, derived and implemented on classic control environments. Each later method is a repair of the one before it.
The log-derivative trick derived line by line, reward-to-go from causality, why baselines cut variance without adding bias, and entropy regularization, with runnable CartPole implementations in both frameworks.
Bellman optimality and the tabular update worked on a gridworld, then DQN's two fixes, replay buffers and target networks, and why bootstrapping plus function approximation plus off-policy data is the deadly triad.
The critic as a learned baseline, TD error as a one-sample advantage, and generalized advantage estimation as the lambda dial between bias and variance, with the JAX version's backward lax.scan as the naturally idiomatic fit.
Why naive policy steps destroy the policy, the clipped surrogate objective worked case by case, the full rollout-GAE-minibatch recipe with the implementation details that actually matter, and the bridge from game RL to RLHF.
The algorithms that turn rewards and preferences into better models, from the RLHF pipeline to the methods behind reasoning training.
DeepSeek's replacement for PPO's value network: sample a group of completions per prompt, normalize rewards within the group to get advantages, keep the clipped ratio and the KL to the reference policy. The method behind DeepSeek-R1 and most verifiable-reward reasoning training, with the loss implemented in both frameworks.
The derivation that makes it famous: start from the KL-constrained RLHF objective, write the optimal policy in closed form, and substitute it into the Bradley-Terry preference likelihood so the reward model disappears. Offline preference tuning without rollouts, plus its limits versus online RL.
Deep walkthroughs of the codebases that run LLM reinforcement learning at scale: how to use them and how to read them.
ByteDance's HybridFlow framework, the production home of large-scale LLM RL. The walkthrough covers a GRPO run end to end, the single-controller programming model that separates the algorithm's dataflow from distributed execution, the actor, rollout, and reward worker architecture, and the training-to-inference weight-resharding problem at the center of every RLHF system.
Hugging Face's trainer family for post-training: SFT, reward modeling, DPO, GRPO, and PPO, riding on transformers, accelerate, and peft so a LoRA-based RLHF run fits on one GPU. The walkthrough covers runnable DPO and GRPO examples and a reading path through the trainer implementations, and says honestly when you have outgrown it.
Planned: OpenRLHF, CleanRL as the single-file reference implementations, and gymnax for JAX-native environments.