reinforcement learning from human feedback (RLHF)

Reinforcement learning from human feedback (RLHF) is a training technique that aligns a large language model with human preferences by using human comparisons of model outputs to shape the model’s behavior.

A typical RLHF pipeline runs in three stages:

• Supervised fine-tuning on human-written prompts and responses to give the base model instruction-following behavior.
• Reward model training, where annotators rank pairs of model outputs and a separate model learns to predict those preferences as a scalar reward.
• Policy optimization with a reinforcement learning algorithm, typically proximal policy optimization (PPO), that maximizes the reward while a Kullback-Leibler divergence penalty discourages drift from the supervised baseline.

RLHF was popularized by InstructGPT and is used to align assistants such as ChatGPT, Claude, and Gemini.

Related methods include direct preference optimization (DPO), a direct alignment algorithm that optimizes the same preference objective with a supervised loss. It needs no separate reward model or reinforcement learning loop. Another related method, reinforcement learning from AI feedback (RLAIF), replaces human annotators with another model.

A third approach, reinforcement learning from verifiable rewards (RLVR), swaps the learned reward model for automatic checkers such as unit tests or math graders. It has become the dominant post-training method for reasoning models, usually paired with group relative policy optimization (GRPO) rather than PPO.

Known limitations of RLHF include reward hacking, sensitivity to annotator agreement, and bias inherited from the preference data.

Related Resources

Tutorial

How to Integrate ChatGPT's API With Python Projects

Learn how to use the ChatGPT Python API with the openai library to build AI-powered features in your Python applications.

intermediate ai api

For additional information on related topics, take a look at the following resources:

• Use ChatGPT to Learn Python Programming (Tutorial)
• Document Your Python Code and Projects With ChatGPT (Tutorial)
• Build an LLM RAG Chatbot With LangChain (Tutorial)
• How to Use Ollama to Run Large Language Models Locally (Tutorial)
• Python MarkItDown: Convert Documents Into LLM-Ready Markdown (Tutorial)
• Leverage OpenAI's API in Your Python Projects (Course)
• How to Integrate ChatGPT's API With Python Projects (Quiz)
• First Steps With LangChain (Course)
• Build an LLM RAG Chatbot With LangChain (Quiz)
• How to Use Ollama to Run Large Language Models Locally (Quiz)
• Python MarkItDown: Convert Documents Into LLM-Ready Markdown (Quiz)