RLHF & Alignment
AI Engineer path · lesson 101 of 124Walk through the full alignment pipeline — supervised fine-tuning, reward model training, and PPO — then see how DPO simplifies this by removing the RL loop entirely, and explore the failure modes that can make aligned models misbehave.
Key insight
RLHF's power is that it captures what humans value — not just what they can demonstrate. But it's expensive, and the demonstration stage is where the bill starts. OpenAI used 13,000 (prompt, response) pairs for InstructGPT's SFT stage. One pair can take up to 30 minutes, so at $10 a pair that is roughly $130,000 — before data design, recruiting and quality control, and with labelers of whom about 90% held a degree and more than a third a master's. The preference and PPO figures beside it are typical ranges rather than reported numbers.
The RLHF pipeline
Order the three training stages and what feeds them.
Tap a piece, then tap where it goes — or drag it. Tap the tray to take one back.
- 1Drop a step here
- 2Drop a step here
- 3Drop a step here
- 4Drop a step here
- 5Drop a step here
- 6Drop a step here
Test your knowledge
What does RLHF stand for?
The short version
Pretraining produces a brilliant but unconstrained intern. SFT gives them a style guide. RLHF is running thousands of simulated performance reviews — the intern maximizes for your feedback score. DPO skips the simulation: you directly express preferences ("I liked answer A better than B") and math does the rest.
In more depth
RLHF & Alignment
Aligning a language model means making it helpful, harmless, and honest — not just a good next-token predictor.
Three-Stage Pipeline
Stage 1 — SFT (Supervised Fine-Tuning)
Fine-tune the pretrained model on high-quality (instruction, response) demonstrations to produce a well-behaved baseline.
Stage 2 — Reward Model
Collect preference pairs: show two completions to human raters, collect "A is better than B" labels. Train a reward model RM(x, y) using a Bradley-Terry ranking objective:
loss = −log σ(RM(x, yw) − RM(x, yl))
where yw is preferred and yl is not.
Stage 3 — PPO
Use Proximal Policy Optimization to maximize expected reward while penalizing KL divergence from the SFT policy:
objective = E[RM(x, y)] − β · KL(πθ || πSFT)
DPO: Direct Preference Optimization
DPO (Rafailov et al. 2023) shows that the RLHF objective has a closed-form solution. The policy can be updated directly on preference data:
loss_DPO = −log σ(β(log πθ(yw|x)/πref(yw|x) − log πθ(yl|x)/πref(yl|x)))
No reward model, no RL loop — just a binary cross-entropy-style loss.
Failure Modes
Interview Questions
References & Further Reading
Love this visualization?
Sign up to track your progress, earn achievements, use the AI assistant, and access your personal learning dashboard.
100% free • No credit card required