walkinglabs/hands-on-modern-rl

From Markov decision processes and policy optimization to reasoning models, agents, and multimodal systems

English · 中文

Train on ModelScope · Run the CartPole Notebook · ModelScope Script

Discussion Group (WeChat)

Learning Paths · Hands-On Examples · Book Features · About This Book · News · Contents · Book Structure · Experiment Code · Quick Start · Contributing

Classical & deep RL (Chapters 1–12). Start with CartPole: choose an action, observe the next state, and collect a reward. Build MDPs and value functions around that interaction, then implement DQN, policy gradients, Actor-Critic, and PPO in games and control tasks. Offline RL, imitation learning, exploration, and world models extend these foundations.

Modern RL & Agentic AI (Chapters 13–26). Apply the policy foundations to existing language models through RLHF, DPO, GRPO, and verifiable rewards. Agentic AI extends a model from answering once to calling tools, browsing, writing code, and completing multi-step tasks; the Agentic RL chapters train these decisions using feedback from entire trajectories. The later chapters cover visual and embodied tasks, training systems, and evaluation.

📣 Announcement

We sincerely thank everyone for your support of this tutorial! A new version is coming soon. Many sections are still being organized and refined, so we appreciate your patience. Suggestions and feedback are always welcome!

Note: This course was created with AI assistance and has not yet been fully reviewed. It may contain factual mistakes or code that does not run as expected. Issues and pull requests are very welcome.

• [2026-08-19] 🎮 Online Classic RL Environments and Scripts: Over the past two weeks, we have added and refined a collection of online reinforcement learning environments, training scripts, and companion notebooks. Learners can now run classic reinforcement learning experiments online, inspect training logs and evaluation results, and study the algorithms more conveniently. We also fixed many previously reported bugs in the course content, links, and experiment code.
• [2026-05-15] 📖 Full English Translation & PDF Release: Complete English translation of all chapters is now available. PDF builds for both Chinese and English editions are released automatically via CI.
• [2026-05-13] 🚀 Major Upgrade: LLM and Traditional RL Hands-on Labs: Added reproducible training examples for Agentic RL (Deep Research / rLLM) and Traditional RL (Actor-Critic continuous control). Includes complete code and fine-tuning analysis for building an Agentic training system from scratch, along with new VLM RL (GeoQA geometry reasoning) hands-on experiments!
• [2026-05-02] Initial browsable open-source release for testing and feedback.

Eight examples from the course: four environment-based experiments, followed by preference tuning, code RL, research agents, and visual reasoning. The links below open the companion code or walkthrough; the full collection is indexed in code/README.md.

WalkingLab is collaborating with ModelScope to provide online training environments for classic reinforcement learning experiments. A ModelScope Studio brings the experiment interface, runtime, and training entry point together on one page, so learners can start training in a browser and observe the agent without first configuring a local environment.

Every Studio has a companion notebook under code/online-experiments. The notebook imports the same training runtime as the Studio, exposes the experiment parameters, prints the full training log, plots checkpoint evaluations, and displays the learned-policy replay or result artifact.

CPU entries run on an ordinary notebook instance. Experiments 03, 08, 10, 11, and 12 require a scheduled ModelScope xGPU Notebook and check CUDA before training.

One continuous learning path Begin with a CartPole trial and progress through value learning, policy optimization, and modern agents.	Equations meet code Key PPO, DPO, and GRPO derivations sit beside their implementations, with every tensor accounted for.
Claims tested by experiments Real training curves, ablations, and failure signals show when an algorithm works and when it does not.	Classic RL to language models Policy gradients and PPO lead naturally into RLHF, DPO, GRPO, and RLVR.
Agents as sequential decisions Tool use, browser interaction, and code repair become problems of states, actions, trajectories, and credit assignment.	Phenomena before abstractions CartPole, LunarLander, Atari, and LLM experiments pose the problem before introducing the mathematics.

Help Wanted

Because compute resources are limited, we are seeking GPU support. If you can help with GPU access, please contact physicoada@gmail.com.

• Book Features
• About This Book
• Structure of the Book
• Experiment Code
• Recommended Learning Path
• Quick Start
• Contributing
• Citation
• Acknowledgements
• License

Reinforcement learning studies a simple but difficult problem: a system acts, observes the consequences, and uses them to improve its next action. When rewards arrive late, observations are incomplete, and each update changes the distribution of future experience, the familiar supervised-learning model of inputs and labels is no longer enough. We need a language for interaction, a way to estimate long-term return, and methods that improve a policy stably from limited data.

Hands-On Modern RL follows that problem from beginning to end. CartPole and multi-armed bandits first make states, actions, rewards, and policies observable. Markov decision processes, value functions, and Bellman equations then provide a common language. From there, the book develops DQN, policy gradients, actor-critic methods, PPO, continuous control, and offline reinforcement learning. With those foundations in place, RLHF, DPO, GRPO, and RLVR become extensions of the same sequential-decision framework rather than an isolated collection of acronyms.

The second half expands the environment to tools, browsers, code repositories, vision, and audio. An action may be a passage of text, a function call, or a sequence of interface operations. A reward may come from human preference, a rule-based verifier, or a process reward model. The setting changes, but three questions run through the entire book: How should the decision process be represented? How should an outcome be credited to earlier actions? How can we tell whether a policy has actually improved?

Each chapter follows a problem–method–experiment–reflection rhythm. A concrete task first exposes the difficulty. The concepts and equations needed to solve it come next. Runnable code, training curves, and evaluation metrics then test the argument. The chapter closes by examining assumptions, failure modes, and the limits of the method. Mathematics explains observed behavior; experiments check the mathematics.

Implementations retain the visible skeleton of each algorithm. Readers can trace trajectory collection, advantage estimation, loss construction, and metric changes—and see how reward hacking, KL drift, entropy collapse, distribution shift, or evaluation leakage can invalidate an apparently successful run.

This book is for students, researchers, and engineers with basic machine-learning experience who want a systematic understanding of modern reinforcement learning. Readers should be comfortable with Python and basic PyTorch, and should know introductory linear algebra, probability, and calculus. The mathematical appendices rebuild these tools to the depth required by the chapters, so a separate advanced-mathematics sequence is not a prerequisite.

After completing the core chapters and labs, you should be able to:

• formulate a new decision problem using MDPs, value functions, Bellman equations, and credit assignment;
• implement, read, and diagnose DQN, REINFORCE, actor-critic methods, PPO, DPO, and GRPO;
• explain how SFT, reward modeling, preference optimization, RLHF, and RLVR fit together in LLM post-training;
• design trajectories, rewards, training loops, and evaluation protocols for tool-use, code, and multimodal agents;
• identify failure modes behind training curves and test proposed improvements with controlled experiments.

This repository is an active courseware project. Content is being expanded chapter by chapter, with emphasis on correctness, runnable examples, and a stable learning path.

• Course site: walkinglabs.github.io/hands-on-modern-rl
• Source content: docs/
• Runnable examples: code/
• Local verification: npm run verify
• License: CC BY-NC-SA 4.0

Issues and pull requests are welcome for typo fixes, conceptual corrections, reproducibility improvements, references, and focused course extensions.

The course is under active development. Planned milestones:

• 2026-05-02: Initial open-source browsable release for community testing and feedback.
• 2026-05-10: Publish a first stable minor version, fix early typos, and stabilize Part 1 and Part 2 content and code.
• Late May 2026: Improve reproducible LLM RL experiments and add a full RLVR hands-on module with evaluation.
• Early June 2026: Deliver Agentic RL projects step by step, from single-tool use to complex Deep Research trajectory synthesis.
• Late June 2026: Add Unity-based embodied RL environments and trainable project examples.
• July 2026 and later: Expand multimodal frontier content with full VLM RL or Diffusion RL hands-on cases.

The book contains seven parts and twenty-six chapters. Parts I–III establish the common language and algorithmic foundations of reinforcement learning. Part IV brings those tools into LLM post-training. Parts V and VI study what changes when the action space expands to tools and multimodal environments. Part VII asks how to detect failures, build trustworthy evaluations, and move the research frontier forward. The appendices provide implementation, mathematics, and engineering references.

The book first makes an agent's failures and improvements observable, then develops the mathematical objects needed to describe long-term decisions.

When the state space grows, tables no longer suffice. This part introduces function approximation and follows the value-learning and policy-learning routes into PPO and continuous control.

When interaction is expensive, expert demonstrations are available, or a task spans multiple agents and time scales, the object of learning changes.

Language models expand an “action” into a passage of text. Policy optimization, distribution constraints, and credit assignment now lead into preference alignment, verifiable rewards, and inference-time computation.

Once agents call tools across many environment steps, the unit of training becomes a trajectory. Credit assignment, environment construction, and safety boundaries become central.

Vision, audio, robot actions, and generative models introduce new state representations, reward sources, and evaluation criteria.

A rising training reward only shows that the optimizer met its objective. This final part asks whether the objective was sound, whether the gain was real, and what new risks follow from broader capabilities.

The code/ directory contains runnable examples aligned with course chapters. Each chapter's code is intentionally compact so it can be inspected, run, and modified independently.

See code/README.md for a code index and chapter-specific dependency notes.

For a first systematic reading, follow the chapters in order. Chapters 1–4 establish the language and recursive ideas of RL; Chapters 5–9 develop the algorithmic core of deep RL; Chapters 10–12 expand the data and task settings. Together, these parts provide the foundation for the rest of the book.

Readers focused on LLM post-training can enter Chapters 13–18 after completing Chapters 6–8. Policy gradients, advantage estimation, PPO, and KL constraints directly explain the objectives used by RLHF, DPO, and GRPO. Then select the relevant topics from Chapters 19–24 for Agentic or multimodal RL. Chapters 25–26 are worth reading alongside any experiment because reward and evaluation errors affect every method in the book.

For each chapter: restate the problem it solves, derive the central equation, run at least one experiment, and change one important assumption to explain the resulting metrics. Use the appendices as references when mathematics or engineering details arise; they do not need to be read front to back first.

Published course site:

https://walkinglabs.github.io/hands-on-modern-rl/

Requirements:

• Node.js >= 18.0.0
• npm

git clone https://github.com/walkinglabs/hands-on-modern-rl.git
cd hands-on-modern-rl
npm install
npm run dev

Then open the local VitePress URL shown in the terminal, usually:

http://localhost:5173

Before submitting a pull request that changes documentation structure, theme code, navigation, build scripts, or generated assets, run:

npm run verify

This checks formatting, lints the VitePress theme, builds the site, and verifies expected build artifacts.

Most code examples use Python and are organized by chapter.

cd code
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For smaller installs, use chapter-specific requirements files:

pip install -r chapter01_cartpole/requirements.txt
python chapter01_cartpole/1-ppo_cartpole.py

Some chapters may require additional system libraries, GPU support, model downloads, or environment-specific setup. Start with Chapter 01 before running examples that involve LLMs, VLMs, or heavy simulators.

hands-on-modern-rl/
|-- docs/                      # VitePress course content
|   |-- .vitepress/            # Site config, navigation, and theme overrides
|   |-- public/                # Static assets copied into the built site
|   |-- preface/               # Course introduction and history
|   |-- chapter*/              # Main course chapters
|   |-- appendix*/             # Supplementary material and references
|   `-- summaries/             # Part-level review and summary notes
|-- code/                      # Runnable examples aligned with chapters
|-- scripts/                   # Maintenance and verification scripts
|-- package.json               # Site scripts and dependencies
|-- AGENTS.md                  # Repository maintenance guide
`-- README.md                  # Main project overview

npm run dev           # Start the local documentation server
npm run build         # Build the static site
npm run preview       # Preview the built site locally
npm run format        # Format repository files with Prettier
npm run format:check  # Check formatting
npm run lint          # Lint VitePress theme code
npm run verify        # Run format check, lint, build, and artifact verification

Contributions should make the course clearer, more accurate, easier to reproduce, or easier to navigate.

Good contributions include:

• Fixing conceptual errors, formulas, diagrams, broken links, or typos.
• Improving explanations without changing the intended learning path.
• Adding small, reproducible experiments that clarify existing chapters.
• Improving scripts, build reliability, navigation, or accessibility.
• Adding high-quality references to papers, official documentation, or widely used open-source implementations.

Please keep pull requests focused. A good PR usually changes one chapter, one experiment, one group of diagrams, or one infrastructure issue at a time.

When adding content:

1. Put course material under docs/.
2. Use kebab-case for new directories and files.
3. Prefer directory-based routes with index.md.
4. Update docs/.vitepress/config.mjs when adding navigable pages.
5. Run npm run verify before requesting review if your change touches config, theme, scripts, or generated site output.
6. Use Conventional Commits, such as docs: clarify ppo clipping or fix: repair chapter link.

For repository-specific maintenance rules, see AGENTS.md.

Our team has also created other courses. Take a look:

• Learn Harness Engineering — A course on Harness Engineering for AI coding agents. Through 12 lectures and 6 projects, it teaches you to build instructions, state management, verification, and control mechanisms that make model output reliable.
• Modern LLM Notebook — Build modern LLMs from scratch through 23 runnable Jupyter Notebooks in PyTorch, covering Tokenizer, Transformer, training, inference, alignment, and frontier topics.

For suggestions or feedback, scan the QR code to join the discussion group (WeChat):

If you use this course in teaching materials, study notes, or derivative non-commercial educational work, please cite the repository:

@misc{hands_on_modern_rl,
  title        = {Hands-On Modern RL: Practice-first reinforcement learning from CartPole to LLM post-training and agentic systems},
  author       = {WalkingLabs},
  year         = {2026},
  howpublished = {\url{https://github.com/walkinglabs/hands-on-modern-rl}},
  note         = {Open courseware repository}
}

We thank OpenAI for providing development resources and AMD for providing computing resources that support this project. Without their support, this course could not have evolved so quickly.

This course is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

You may share and adapt the material for non-commercial purposes, provided that you give appropriate credit and distribute derivative works under the same license.