DPO — the shortcut that took over

In 2023, a paper arrived with a title that summarizes itself: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." Its result reshaped open-source practice within months.

The insight, plain-language version: the RLHF setup — reward model + KL-leashed policy optimization — has enough mathematical structure that you can solve it on paper. The optimal policy and the reward model turn out to be two views of the same object, connected by a formula. So instead of building the judge and then chasing its scores through an unstable RL loop, you can skip both and write down a direct loss on the preference pairs themselves:

That's the whole algorithm. Notice what survived the collapse: the reference model still anchors everything (comparing against it is what plays the KL-leash role — the "stay yourself" rule is baked into the loss), and a knob called beta sets leash tightness (typical 0.05–0.3; higher = more conservative). What vanished: the reward model, the value model, the generation-inside-training loop, the RL instability. Here's the whole comparison in one picture:

The practical consequences of that right-hand panel are why DPO conquered open source: it trains like plain supervised learning — fixed dataset, standard loop, stable loss curves, two models in memory instead of four. And even the "two models" softens with a LoRA trick worth knowing: since your policy = base + adapter, the reference is just the same base with the adapter switched off — one copy of weights serves both roles. QLoRA + DPO on a single consumer GPU is routine. In TRL, it's deliberately anticlimactic:

from trl import DPOConfig, DPOTrainer
 
trainer = DPOTrainer(
    model=model,                      # your SFT'd model (adapter attached)
    args=DPOConfig(output_dir="dpo_out", beta=0.1, learning_rate=5e-6),
    train_dataset=pairs,              # columns: prompt, chosen, rejected
    processing_class=tok,
)
trainer.train()

Note the learning rate: ~5e-6, drastically gentler than SFT's 2e-4 — preference tuning is a finishing pass over an already-good model, and heavy hands break things fast here.

The honest comparison — because "DPO won" needs an asterisk. PPO-style RLHF is on-policy: the model learns from its own fresh outputs, exploring and getting corrected where it actually lives. DPO is offline: it studies a fixed binder of comparisons, which may describe answers unlike anything the current model would say. On-policy learning is genuinely more powerful when done well — it's why frontier labs still run RL variants, and why RLVR (verifiable rewards) dominates reasoning training. DPO's sweet spot — which happens to be your sweet spot — is: you have preference pairs, modest compute, and want stage-3 polish without an RL team. And recall Topic 16's advanced note, now fully explainable: generating your pairs from your own model's outputs (sample 4, rank, take best/worst) makes DPO quasi-on-policy — the binder describes the model's actual neighborhood. That one data decision closes much of the gap.

The variant zoo, decoded in one line each so no acronym ever intimidates you: IPO (patches a DPO overfitting quirk), ORPO (fuses SFT + preference into a single stage — no reference model at all), SimPO (drops the reference too, via length-normalized scoring), and the one with real product relevance for you: KTO, which trains on unpaired thumbs-up/thumbs-down labels instead of pairs. Sit with that: every product with a 👍/👎 button is passively collecting KTO-ready training data. If you ever ship an AI SaaS, that feedback widget is a preference-dataset factory.

The standard modern recipe, and what you'd actually run: SFT first, then DPO — teach the format with demonstrations, then polish the judgment with comparisons. Stage 2, stage 3, exactly as the Lesson 3 pipeline diagram promised.