Preference Datasets — teaching "better," not just "good"

SFT has a structural blind spot: it only ever shows positive examples. But much of what makes a model good is relative — this answer is better than that one, shorter beats rambling, admitting uncertainty beats confident nonsense, this refusal is graceful and that one is preachy. You can't demonstrate "don't do X" with a dataset that only contains things to do.

Enter the preference dataset. Each row is a triplet:

{"prompt": "My deploy failed with exit code 137, what happened?",
 "chosen":  "Exit 137 means the process was killed - almost always OOM. Check memory limits...",
 "rejected": "There could be many reasons for deployment failures. Have you tried checking logs?"}

Notice the rejected answer isn't wrong — it's just worse: vague, unhelpful, question-deflecting. That's the power of the format: it encodes a direction of improvement that no pile of good examples alone can express. Stage 3 methods (DPO, RLHF — the actual algorithms are Module 3) consume these triplets to push the model's probability mass toward chosen-like outputs and away from rejected-like ones.

Where do the pairs come from? Three sources, in increasing modernity:

1. Human annotation. Show labelers two model outputs, they pick the better one. Gold standard in theory; in practice expensive and noisy — inter-annotator agreement on such comparisons typically runs only ~65–75%. Humans genuinely disagree about "better."
2. AI feedback (RLAIF / LLM-as-judge). A strong model does the ranking, guided by a written rubric or set of principles. Dramatically cheaper, surprisingly competitive with human labels, now the workhorse. (Anthropic's Constitutional AI is the famous lineage here.)
3. Verifiable signals. For code: the chosen answer passes the tests, the rejected one doesn't. For math: correct final answer vs incorrect. Free, perfectly clean labels — use these whenever your domain allows.

Two advanced points that separate practitioners from tutorial-followers:

Length bias is real and will bite you. Both humans and LLM judges systematically prefer longer, more elaborate answers — even when the extra words add nothing. Train on such preferences naively and you manufacture a model that pads everything. (Much of the "why do chatbots ramble" phenomenon traces to exactly this.) Mitigations: length-balance your pairs, instruct judges to penalize padding, or use length-controlled evaluation.

On-policy beats off-policy. Preference pairs built from your own model's outputs (sample 4 responses from the model you're tuning, rank them, take best/worst as chosen/rejected) teach far more effectively than pairs borrowed from other models' outputs — the corrections land in the region where your model actually lives. Public datasets like UltraFeedback or Anthropic HH-RLHF are excellent for study, but the strongest results come from generating pairs on-policy. Keep this in your pocket for Module 3's DPO topic.