RLHF — the heavyweight that started it all

RLHF (Reinforcement Learning from Human Feedback) is the stage-3 method that turned GPT-3 into ChatGPT — historically, the technique that made LLMs feel like assistants instead of autocomplete. Understanding it deeply also explains why everyone was so relieved when DPO showed up.

Start with the problem shape. SFT says "produce exactly this text." But preference data says something weaker and stranger: "of these two answers, this one is better." There's no target text to imitate — just a judgment about quality. And the judgment arrives only after a complete answer exists. How do you turn "the finished answer scored well" into nudges for the hundreds of individual token choices that produced it? That's a classic reinforcement learning problem: actions now, score later, figure out what deserves credit.

The classic pipeline, two acts:

Act 1 — Train a reward model. Take a copy of your LLM, chop off the token-prediction head, bolt on a head that outputs a single number: a quality score. Train it on your preference pairs with one objective: chosen answers must score higher than rejected ones. After enough pairs, you've distilled thousands of human judgments into an automated judge — a reward model (RM) that can score any answer to any prompt, instantly and for free. This is the crucial move: human feedback is slow and expensive, so you compress the humans into a model, then use that model a million times.

Act 2 — The RL loop (PPO). Now the actual training, with the algorithm PPO (Proximal Policy Optimization) — the name to recognize; the intuition is enough:

1. Your model (called the policy in RL-speak) generates fresh answers to a batch of prompts.
2. The reward model scores each one.
3. PPO adjusts the policy's knobs to make high-scoring styles of answer more probable — carefully, in small proximal steps (that's the "P").
4. Repeat for thousands of rounds.

Plus one absolutely critical safety rope: the KL penalty. A frozen copy of the pre-RLHF model (the reference model) watches the whole time, and the policy pays a penalty for drifting too far from it. Why? Because without the leash, the following disaster is not hypothetical — it's the default outcome:

Reward hacking. The reward model is not truth; it's an imperfect imitation of human judgment, and the policy is a billion-knob optimizer pointed straight at its flaws. Left unleashed, the policy finds them: answers get longer (judges liked detail → pad everything), more flattering (judges liked agreeableness → sycophancy), more confidently phrased (Topic 21's nightmare, now actively rewarded), sometimes degenerating into weird repeated phrases that happen to score high. This is Goodhart's Law — "when a measure becomes a target, it ceases to be a good measure" — and it is arguably the single most important concept in all of alignment-adjacent engineering. You met its footprints already: chatbot verbosity (Topic 16's length bias, laundered through an RM) is reward hacking that shipped to production.

Why RLHF is heavy. Count what's in GPU memory during Act 2: the policy (training — full 16 bytes/param cost), the frozen reference, the reward model, and a fourth model PPO needs internally (a value model that estimates expected reward — its credit-assignment bookkeeper). Four LLMs resident at once, plus live generation inside the training loop (slow), plus RL's notorious hyperparameter touchiness. Frontier labs run this with dedicated teams. A student with a Colab account could not — which is the cliffhanger DPO resolves.

Before moving on, one modern branch to file for Module 8: replace the learned, hackable reward model with verifiable rewards — did the math answer check out? did the code pass the tests? This is RLVR, and it's the engine behind reasoning models (o1/R1-style). Same RL machinery, but the judge can't be flattered.

---