Lesson 6Preference Algorithms, Portable Models, and Your First Real Fine-Tune

Welcome to Lesson 6 — the back half of Module 3. Today the preference pairs you learned to build in Topic 16 finally get consumed by actual algorithms, and at the end, everything from four lessons converges into one runnable pipeline. Let's finish this module properly.

Topic 27: RLHF — the heavyweight that started it all

RLHF (Reinforcement Learning from Human Feedback) is the stage-3 method that turned GPT-3 into ChatGPT — historically, the technique that made LLMs feel like assistants instead of autocomplete. Understanding it deeply also explains why everyone was so relieved when DPO showed up.

Start with the problem shape. SFT says "produce exactly this text." But preference data says something weaker and stranger: "of these two answers, this one is better." There's no target text to imitate — just a judgment about quality. And the judgment arrives only after a complete answer exists. How do you turn "the finished answer scored well" into nudges for the hundreds of individual token choices that produced it? That's a classic reinforcement learning problem: actions now, score later, figure out what deserves credit.

The classic pipeline, two acts:

Act 1 — Train a reward model. Take a copy of your LLM, chop off the token-prediction head, bolt on a head that outputs a single number: a quality score. Train it on your preference pairs with one objective: chosen answers must score higher than rejected ones. After enough pairs, you've distilled thousands of human judgments into an automated judge — a reward model (RM) that can score any answer to any prompt, instantly and for free. This is the crucial move: human feedback is slow and expensive, so you compress the humans into a model, then use that model a million times.

Act 2 — The RL loop (PPO). Now the actual training, with the algorithm PPO (Proximal Policy Optimization) — the name to recognize; the intuition is enough:

1. Your model (called the policy in RL-speak) generates fresh answers to a batch of prompts.
2. The reward model scores each one.
3. PPO adjusts the policy's knobs to make high-scoring styles of answer more probable — carefully, in small proximal steps (that's the "P").
4. Repeat for thousands of rounds.

Plus one absolutely critical safety rope: the KL penalty. A frozen copy of the pre-RLHF model (the reference model) watches the whole time, and the policy pays a penalty for drifting too far from it. Why? Because without the leash, the following disaster is not hypothetical — it's the default outcome:

Reward hacking. The reward model is not truth; it's an imperfect imitation of human judgment, and the policy is a billion-knob optimizer pointed straight at its flaws. Left unleashed, the policy finds them: answers get longer (judges liked detail → pad everything), more flattering (judges liked agreeableness → sycophancy), more confidently phrased (Topic 21's nightmare, now actively rewarded), sometimes degenerating into weird repeated phrases that happen to score high. This is Goodhart's Law — "when a measure becomes a target, it ceases to be a good measure" — and it is arguably the single most important concept in all of alignment-adjacent engineering. You met its footprints already: chatbot verbosity (Topic 16's length bias, laundered through an RM) is reward hacking that shipped to production.

Why RLHF is heavy. Count what's in GPU memory during Act 2: the policy (training — full 16 bytes/param cost), the frozen reference, the reward model, and a fourth model PPO needs internally (a value model that estimates expected reward — its credit-assignment bookkeeper). Four LLMs resident at once, plus live generation inside the training loop (slow), plus RL's notorious hyperparameter touchiness. Frontier labs run this with dedicated teams. A student with a Colab account could not — which is the cliffhanger DPO resolves.

Before moving on, one modern branch to file for Module 8: replace the learned, hackable reward model with verifiable rewards — did the math answer check out? did the code pass the tests? This is RLVR, and it's the engine behind reasoning models (o1/R1-style). Same RL machinery, but the judge can't be flattered.

Topic 28: DPO — the shortcut that took over

In 2023, a paper arrived with a title that summarizes itself: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." Its result reshaped open-source practice within months.

The insight, plain-language version: the RLHF setup — reward model + KL-leashed policy optimization — has enough mathematical structure that you can solve it on paper. The optimal policy and the reward model turn out to be two views of the same object, connected by a formula. So instead of building the judge and then chasing its scores through an unstable RL loop, you can skip both and write down a direct loss on the preference pairs themselves:

That's the whole algorithm. Notice what survived the collapse: the reference model still anchors everything (comparing against it is what plays the KL-leash role — the "stay yourself" rule is baked into the loss), and a knob called beta sets leash tightness (typical 0.05–0.3; higher = more conservative). What vanished: the reward model, the value model, the generation-inside-training loop, the RL instability. Here's the whole comparison in one picture:

The practical consequences of that right-hand panel are why DPO conquered open source: it trains like plain supervised learning — fixed dataset, standard loop, stable loss curves, two models in memory instead of four. And even the "two models" softens with a LoRA trick worth knowing: since your policy = base + adapter, the reference is just the same base with the adapter switched off — one copy of weights serves both roles. QLoRA + DPO on a single consumer GPU is routine. In TRL, it's deliberately anticlimactic:

from trl import DPOConfig, DPOTrainer
 
trainer = DPOTrainer(
    model=model,                      # your SFT'd model (adapter attached)
    args=DPOConfig(output_dir="dpo_out", beta=0.1, learning_rate=5e-6),
    train_dataset=pairs,              # columns: prompt, chosen, rejected
    processing_class=tok,
)
trainer.train()

Note the learning rate: ~5e-6, drastically gentler than SFT's 2e-4 — preference tuning is a finishing pass over an already-good model, and heavy hands break things fast here.

The honest comparison — because "DPO won" needs an asterisk. PPO-style RLHF is on-policy: the model learns from its own fresh outputs, exploring and getting corrected where it actually lives. DPO is offline: it studies a fixed binder of comparisons, which may describe answers unlike anything the current model would say. On-policy learning is genuinely more powerful when done well — it's why frontier labs still run RL variants, and why RLVR (verifiable rewards) dominates reasoning training. DPO's sweet spot — which happens to be your sweet spot — is: you have preference pairs, modest compute, and want stage-3 polish without an RL team. And recall Topic 16's advanced note, now fully explainable: generating your pairs from your own model's outputs (sample 4, rank, take best/worst) makes DPO quasi-on-policy — the binder describes the model's actual neighborhood. That one data decision closes much of the gap.

The variant zoo, decoded in one line each so no acronym ever intimidates you: IPO (patches a DPO overfitting quirk), ORPO (fuses SFT + preference into a single stage — no reference model at all), SimPO (drops the reference too, via length-normalized scoring), and the one with real product relevance for you: KTO, which trains on unpaired thumbs-up/thumbs-down labels instead of pairs. Sit with that: every product with a 👍/👎 button is passively collecting KTO-ready training data. If you ever ship an AI SaaS, that feedback widget is a preference-dataset factory.

The standard modern recipe, and what you'd actually run: SFT first, then DPO — teach the format with demonstrations, then polish the judgment with comparisons. Stage 2, stage 3, exactly as the Lesson 3 pipeline diagram promised.

Topic 29: GGUF — the format your model wears to leave home

Everything so far lives in the training world: Hugging Face repos, a folder of files — model.safetensors shards, config.json, tokenizer files, chat template. Perfect for training and Python. Now your fine-tuned model needs to go run on a laptop, in Ollama, on someone's Mac — the local-inference world. That world runs on GGUF.

GGUF is a single-file model format built for the llama.cpp ecosystem (the C++ inference engine behind Ollama, LM Studio, and most local AI — Module 5 gives it full treatment). The design philosophy is "everything in one portable box":

• The weights — almost always quantized (Topic 24 pays off again)
• The tokenizer, baked in
• The chat template, baked in (Topic 15's silent killer, largely defused by the format itself — the file carries its own formatting instructions)
• Metadata: architecture, context length, all hyperparameters needed to run

One file. No Python, no dependencies, no folder of parts. It's also memory-mappable — the OS can page weights straight from disk without a slow loading step, which is why local models start near-instantly. If the Hugging Face folder is a project's source code, GGUF is the compiled binary you ship.

Decoding the quant names — because your first visit to a GGUF repo shows twenty files named like Q4_K_M and this table is the decoder ring:

Reading the code: Q4 = ~4 bits per weight; K = the k-quant scheme (block-wise scaling with super-blocks — Topic 24's per-region palettes, two levels deep); S/M/L = how many of the most sensitive layers get kept at higher precision (Medium is the balanced pick). You'll also see IQ variants (importance-aware quants using an imatrix — calibration data identifying which weights matter most, Topic 24's AWQ idea in GGUF clothing) squeezing better quality from very low bit-counts.

Two practical rules: file size ≈ RAM needed (plus a couple GB for context — the KV cache, formally introduced in Module 4), and the Topic 24 law holds — prefer a bigger model at Q4 over a smaller one at Q8 for the same memory.

The full journey — this is the bridge from everything you've trained to everything you'll run:

# 1. Merge your LoRA adapter into the base (16-bit)   [Topic 23's merge]
#    model = PeftModel.from_pretrained(base, "out/final_adapter")
#    model = model.merge_and_unload(); model.save_pretrained("merged/")
 
# 2. Convert HF folder -> GGUF (llama.cpp's script)
python convert_hf_to_gguf.py merged/ --outfile mymodel-f16.gguf
 
# 3. Quantize                                          [Topic 24 applied]
./llama-quantize mymodel-f16.gguf mymodel-Q4_K_M.gguf Q4_K_M
 
# 4. Serve it locally via Ollama
echo 'FROM ./mymodel-Q4_K_M.gguf' > Modelfile
ollama create mybot -f Modelfile
ollama run mybot

Four commands between "my fine-tune finished" and "my model runs on my laptop, offline, forever." When people on Hugging Face say "are there GGUFs?" under every model release — and prolific quantizers upload twenty variants within hours — this pipeline is what they ran. Per the QLoRA merge subtlety from Topic 25 and the recurring law of this course: evaluate the Q4_K_M artifact you actually ship, because it's now two transformations removed from what you trained.

Capstone: The Whole Pipeline, End to End

Module 3's promised payoff. Here is the complete journey — every stage tagged with the lesson that taught it — as one runnable script. Model choice is deliberate: Qwen2.5-1.5B-Instruct trains via QLoRA inside a free Colab T4 or your Mac's memory, in minutes-to-an-hour on a small dataset.

# pip install transformers datasets peft trl bitsandbytes
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
 
MODEL = "Qwen/Qwen2.5-1.5B-Instruct"        # instruct base [Topic 14]
 
# ── 1. DATA: your cleaned synthetic JSONL  [Lessons 3-4]
#    each line: {"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}
data = load_dataset("json", data_files="train_clean.jsonl", split="train")
data = data.train_test_split(test_size=0.1, seed=42)   # firewalled eval split [T18, T26]
 
# ── 2. BASE MODEL in 4-bit  [QLoRA, Topic 25]
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(MODEL, quantization_config=bnb,
                                             device_map="auto")
tok = AutoTokenizer.from_pretrained(MODEL)
 
# ── 3. LoRA bypasses  [Topic 23]
peft_cfg = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                      lora_dropout=0.05, task_type="CAUSAL_LM")
 
# ── 4. TRAINING with checkpoint shopping  [Topic 26]
cfg = SFTConfig(output_dir="out",
    num_train_epochs=2,                      # 1-3, no more [Topic 14]
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,           # effective batch = 16 (see note)
    learning_rate=2e-4, warmup_ratio=0.03,   # LoRA-scale LR [T23], warmup [T20]
    eval_strategy="steps",  eval_steps=25,
    save_strategy="steps",  save_steps=25,  save_total_limit=3,
    load_best_model_at_end=True, metric_for_best_model="eval_loss",
    logging_steps=10, bf16=True)
 
trainer = SFTTrainer(model=model, args=cfg, peft_config=peft_cfg,
                     train_dataset=data["train"], eval_dataset=data["test"],
                     processing_class=tok)
# SFTTrainer applies the model's own chat template and loss masking [Topic 15]
trainer.train()
trainer.save_model("out/final_adapter")      # ~50 MB of adapter [T22]

Three annotations the script can't say for itself:

Gradient accumulation — the one genuinely new concept here: with batch size 2, gradients from 8 consecutive mini-batches are accumulated before one knob-update, simulating a batch of 16 without the memory of 16. The universal trick for training big-model behavior on small-GPU budgets.

What "watching training" means: ignore train loss going down (it always does — Topic 26); watch eval_loss at each 25-step checkpoint. Smoothly falling → healthy. U-turn → you're done; load_best_model_at_end rescues the minimum automatically.

The smoke test before anything else — generate with the exact serving format:

msgs = [{"role": "user", "content": "one question from your training domain"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=300)
print(tok.decode(out[0], skip_special_tokens=True))

Read the output like a Lesson 4 critic: right format? Stops properly (EOS learned — Topic 15)? Parroting training examples verbatim (overfit — Topic 26)? Then — and this closes the loop with Topic 21 — ask it something outside its training data and watch how it fails: gracefully, or with confident fabrication your dataset accidentally taught?

From here, the exits you already know: ship as adapter (multi-tenant wardrobe, Topic 22), run DPO on top with on-policy pairs (Topic 28), or merge → GGUF → Ollama (Topic 29's four commands) and your model runs on your own machine, offline, indefinitely. One honest preview: Module 5 introduces Unsloth, which wraps this identical pipeline with major speed/memory optimizations and one-line GGUF export — I taught you the raw TRL version first because Unsloth is only magic if you know what it's automating.

Module 3: complete. You now hold the full modern fine-tuning toolkit: PEFT/LoRA/QLoRA for training within mortal hardware, checkpoint discipline for picking winners, RLHF understood and DPO wielded for preference polish, GGUF for shipping. Combined with Module 2's data craft, you can take a raw idea to a specialized local model — genuinely end to end.

Next: Module 4 — Inference & Optimization. The two flags from Lesson 2 that are still flying — the n² attention cost and one-token-per-forward-pass generation — finally get their reckoning: KV cache (why chatbots don't recompute the whole conversation every token), Flash Attention, speculative decoding, batching, model serving, GPU/VRAM fundamentals, and the latency-vs-quality economics that decide what real products ship.