NiuTrans/RL-without-Tears

Reinforcement learning (RL) has become an important training paradigm for large language models (LLMs). It is widely used for preference alignment, reasoning, and agentic learning from interaction and feedback. At its core, RL studies how an agent improves its behavior by interacting with an environment and receiving rewards. A standard objective is

where $\pi$ denotes the policy, $\tau$ denotes a trajectory, and $r_t$ is the reward received at step $t$.

For LLMs, this formulation has a natural interpretation. The LLM acts as the agent. Its next-token distribution defines the policy. Generated tokens correspond to actions, and the complete response forms a trajectory. Reward signals are then used to evaluate the quality of the generated output.

Although RL is becoming increasingly important for LLMs, many of its concepts remain unfamiliar to researchers with a background in NLP and supervised learning. Traditional RL literature often explains these concepts through robotics, control, or game-playing tasks. This makes it difficult to directly connect RL formulations with modern LLM training. This gap motivates RL without Tears: An Introduction in the Era of LLMs.

In this paper, we introduce RL from the perspective of LLM research. We explain key concepts and algorithms through LLM-oriented examples, including policy gradients, advantage estimation, importance sampling, PPO, and reward modeling. We then discuss recent advances in RL for LLMs. We further extend the discussion to reasoning models, LLM-based agents, and multimodal models.

We aim to make RL easier to understand for the LLM community. Rather than relying on traditional control examples, we use terminology and examples that are familiar to LLM researchers. We hope this paper provides an accessible introduction to RL and helps readers better understand and apply RL techniques in modern foundation models.

A simple way to access this work is through our Online Website.

• Full PDF
• Chapter 1: Introduction
• Chapter 2: Preliminary
• Chapter 3: Understanding RL in LLM Training
• Chapter 4: Improved RL for LLMs
• Chapter 5: RL for LLM Reasoning
• Chapter 6: Agentic RL
• Chapter 7: Multimodal RL
• Chapter 8: Conclusions and Future Directions
• Appendix: Datasets and Systems

Each chapter PDF is compiled independently and includes its own bibliography.

@misc{wang2026rlwithouttears,
  title  = {RL without Tears: An Introduction in the Era of LLMs},
  author = {Wang, Chenglong and Zhou, Hang and Liu, Tongran and Zhu, Jingbo and Xiao, Tong},
  year   = {2026},
  note   = {Tutorial manuscript}
}

This work is released under the CC-BY 4.0 license.

For questions or suggestions, please contact Chenglong Wang at WANGCHENGLONG@MAIL.NEU.EDU.CN.