reinforcement learning (RL)

Reinforcement learning (RL) is a machine learning paradigm where an agent interacts with an environment by observing states, executing actions, and receiving reward feedback, with the goal of improving its policy to maximize cumulative (often discounted) reward.

Many RL problems are formalized as Markov decision processes (MDPs), which are defined by a set of states, actions, transition dynamics, and reward functions. When the agent’s observations are partial or noisy, the problem may instead be modeled as a partially observable Markov decision process (POMDP).

RL techniques include:

• Value-based methods, such as Q-learning and SARSA, which learn action-value or state-value functions and derive policies from them.
• Policy-based methods, such as REINFORCE, proximal policy optimization (PPO), and group relative policy optimization (GRPO), which directly optimize a parameterized policy.
• Actor–critic algorithms that combine both: the critic estimates values, and the actor updates the policy.

In practice, modern RL systems incorporate exploration strategies such as ε-greedy, entropy bonuses, UCB, and intrinsic motivation, along with experience replay for off-policy learning, function approximation, and many stability enhancements like target networks, gradient clipping, trust regions, and KL penalties that keep the updated policy close to a reference policy.

Some RL methods are model-based, where the agent learns or uses an internal model of environment dynamics to plan actions.

Applications include game playing, robotics, autonomous control, recommendation systems, operations research, and, most visibly today, the post-training of large language models. RL systems are evaluated using metrics like return, sample efficiency, robustness, and adherence to safety constraints.

In modern LLM development, RL is the main post-training lever. Reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) fit a reward model to preference data and then optimize the language model against it with a policy-gradient algorithm. Reinforcement learning with verifiable rewards (RLVR) swaps that learned reward model for an automatic checker, like a unit test, a compiler, or an exact-match grader, so the reward reflects only whether the answer checks out. RLVR is what trains today’s reasoning models to work through a problem before answering.