Lesson 3The Data Side of Fine-Tuning

Welcome to Module 2: Datasets & Training — where you stop understanding models and start shaping them. Everything in this module hangs off one picture: the three stages a model passes through, and which kind of dataset feeds each one.

You already met this pipeline conceptually in Lesson 2. Here's the reframe that makes this module matter: when you fine-tune, you are personally running stage 2 (and optionally stage 3) of this pipeline — just on your own data, at a scale of thousands of examples instead of trillions of tokens. Same mechanism, your steering wheel.

Topic 12: Fine-Tuning Basics — when to reach for it (and when not to)

Fine-tuning = taking an already-trained model and continuing training on a small, focused dataset, so the knobs shift slightly toward your task. Two things make it different from pretraining: the dataset is tiny (hundreds to hundreds of thousands of examples, not trillions of tokens), and the learning rate — how big each knob-nudge is — is set very small. You're sculpting an existing statue, not quarrying new marble. Nudge too hard and you shatter what's already there.

The single most important decision rule in applied LLM work:

Fine-tuning is remarkably bad at reliably injecting new knowledge (Topic on continued pretraining next lesson explains why), and remarkably good at changing behavior: output format, tone, style, domain vocabulary, task reliability. If your problem is "the model doesn't know our product docs" → RAG. If your problem is "the model knows enough but won't consistently output valid JSON in our schema / won't write in our brand voice / wastes 300 tokens of preamble" → fine-tuning territory.

The cost ladder — always climb it in order:

1. Better prompting — minutes of effort. Solves most problems.
2. Few-shot examples in the prompt — show 3–5 examples of what you want. Shockingly effective.
3. RAG — hours to days. Solves knowledge problems.
4. Fine-tuning — days to weeks including data work. Solves behavior/reliability/cost problems.

Jumping straight to fine-tuning because it sounds impressive is the field's most common junior mistake. The legitimate reasons to climb to rung 4: (a) prompting has plateaued below your reliability bar, (b) your prompt has grown into a 4,000-token monster you're paying for on every request — fine-tuning can bake the prompt into the weights, (c) you want a small cheap model to replicate a big model's behavior on one narrow task (distillation, the cost play from Lesson 2), (d) latency demands a small model.

One danger to know from day one: catastrophic forgetting. Train a model hard on only your narrow data and it degrades at everything else — general reasoning, other formats, even chat ability. Like a guitarist practicing one song eight hours a day for a month: that song gets great, everything else gets rusty. Mitigations (details in Module 3): train gently (low learning rate, few passes), mix in some general instruction data with your task data, and use LoRA-style methods that touch fewer knobs.

Topic 13: SFT Datasets — the raw material

An SFT dataset (Supervised Fine-Tuning dataset) is a collection of demonstrations: here's an input, here's exactly the output we want. Single-turn form:

{"instruction": "Summarize this error log and suggest a fix", 
 "input": "TypeError: cannot read properties of undefined...", 
 "output": "The error occurs because... To fix it, ..."}

Or multi-turn conversation form (what you'll actually use for anything chat-shaped):

{"messages": [
  {"role": "system", "content": "You are a support agent for isDisposable API."},
  {"role": "user", "content": "Why is my API key rejected?"},
  {"role": "assistant", "content": "Usually one of three causes: ..."}
]}

The mechanism from Lesson 1 applies unchanged: the model does next-token prediction on the output, gets a wrongness score, knobs nudge. Which means the deepest truth about SFT data:

Not just the correct answers — the tone, the length, the hedging, the formatting quirks, the mistakes. Ten examples where the assistant starts with "Great question!" and your model will "Great question!" forever. One example with a subtly wrong answer and you've taught wrongness. There is no separate channel for "please be accurate" — demonstrations are the only channel.

How much data do you need? The landmark result here is the LIMA paper (Meta, 2023): they fine-tuned a 65B base model on just 1,000 meticulously curated examples and got quality approaching models trained on millions. The conclusion — sometimes called the superficial alignment hypothesis — is that the model already learned its capabilities during pretraining; SFT mainly teaches which format and persona to use. Practical calibration:

• Style, tone, format, persona: ~500–2,000 excellent examples often suffice.
• Reliable narrow task (classification, structured extraction, domain-specific generation): 1,000–20,000, and coverage of edge cases matters more than raw count.
• Diversity beats volume. 1,000 examples spanning many phrasings, difficulties, and edge cases outperform 10,000 near-duplicates. The FLAN research line showed training across many task types even improves performance on task types never seen — variety teaches "follow the instruction, whatever it is."

Datasets worth knowing by name (you'll see them everywhere on Hugging Face): Alpaca (52K, machine-generated — historically important, quality mediocre by today's standards), Dolly-15k (written by actual Databricks employees — human, clean, small), OpenAssistant/OASST (crowdsourced multi-turn conversations), UltraChat (large-scale synthetic dialogues), LIMA (the famous 1K). Don't train products on these directly — study them as reference anatomy for building your own.

Topic 14: Instruction Tuning — why the transformation works

Instruction tuning is the specific, most important flavor of SFT: training on (instruction → response) pairs so a base model stops continuing text and starts obeying it. It's what turns Qwen2.5-7B into Qwen2.5-7B-Instruct — the difference you saw with your own eyes in the Lesson 2 exercise.

Why does a few thousand examples flip such a fundamental behavior? Think about what the base model learned from the internet: among trillions of tokens, it saw forums where questions get answers, tutorials, Q&A sites, documentation. So the skill of answering is already inside — but it's one persona among thousands (it equally learned to continue rants, spam, and lists of exam questions). Instruction tuning doesn't teach answering; it collapses the persona distribution — massively raising the probability that "question-shaped input → helpful-answer-shaped output" wins over all the other learned continuations. That's why LIMA's 1,000 examples were enough: they were selecting a behavior, not installing one.

This gives you a sharp lens for what instruction tuning can and cannot do:

• ✅ Can: unlock and standardize behaviors the base model already latently has — formats, task-following, tone, refusing gracefully, using domain vocabulary it saw in pretraining.
• ❌ Can't: create capabilities absent from pretraining. If the base model never saw meaningful amounts of Kotlin, no amount of instruction tuning produces a good Kotlin assistant.
• ⚠️ Dangerous: training it to answer questions whose answers it doesn't actually know. Read that twice, because it's a genuinely advanced insight most practitioners miss: if your SFT data contains answers built from facts outside the model's knowledge, the gradient can't install those facts — what it can install is the behavior pattern "when asked obscure things, produce confident-sounding specific answers." You are literally training hallucination. Research on this (e.g., work following Gekhman et al., 2024) shows fine-tuning on unknown-to-the-model facts measurably increases hallucination. This is a core reason "fine-tune facts in" fails, and it previews our hallucination topic next lesson. The fix: SFT answers should either draw on knowledge the model plausibly has, or demonstrate saying "I don't know" / consulting provided context.

Practical training-process notes (full mechanics in Module 3, but these belong to the data story):

• Epochs = full passes over your dataset. Instruction tuning typically uses 1–3. More passes on a small dataset → the model starts memorizing your examples verbatim instead of learning the pattern — overfitting. Symptom: it parrots training answers word-for-word even for slightly different questions.
• Always fine-tune from the instruct version of a model when your data is small and chat-shaped — you inherit all the alignment work. Start from the base model only when you have lots of data and want full control of the persona.

Topic 15: Dataset Formatting — where fine-tunes silently die

The least glamorous topic in this module, and the cause of the majority of "my fine-tuned model is broken" posts. Attend closely.

Storage format: the universal standard is JSONL — one JSON object per line, no commas between lines, no wrapping array. Streams efficiently, appends trivially, and every training library expects it.

The chat template — the concept that actually matters. The model never sees your neat messages array. Before training or inference, the array gets flattened into one token stream using the model family's chat template — special tokens marking who's speaking. Qwen and many others use ChatML:

<|im_start|>system
You are a support agent.<|im_end|>
<|im_start|>user
Why is my API key rejected?<|im_end|>
<|im_start|>assistant
Usually one of three causes: ...<|im_end|>

Llama 3 uses entirely different markers (<|start_header_id|>user<|end_header_id|> …). These marker tokens are how the model knows a turn ended and whose turn it is. Which sets up the classic silent killer: train with one template, serve with another (or with none), and the model receives token patterns it never saw in training. Output degrades into rambling, format-breaking, never stopping — and nothing errors out. It just quietly produces garbage. When someone's fine-tune "doesn't work," template mismatch is suspect #1.

The professional habit: never hand-write templates. Let the tokenizer apply its own:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
 
text = tok.apply_chat_template(example["messages"], tokenize=False)
# ALWAYS print one example and read it with your eyes before training:
print(text)

That print is not optional ceremony. Inspecting one fully-rendered example catches template bugs, missing system prompts, and the next issue — before you burn GPU hours.

Two more formatting killers:

1. The EOS token. The end-of-sequence token (<|im_end|>, etc.) after the assistant's answer is how the model learns to stop talking. If your formatting pipeline drops it, you train a model that never learned that answers end — it generates until it hits the token limit, every time. A one-token bug with a very memorable symptom.

2. Loss masking. Recall from Lesson 1: training = predict next token, score the miss, nudge. But your training rows contain both the user's tokens and the assistant's tokens. Should the model be nudged toward producing the user's words? No — you'd be training it to imitate users (models that randomly start asking themselves questions — that's this bug). The fix is masking: loss is computed only on assistant tokens; prompt tokens are read for context but generate no learning signal. Visually:

Good news: modern training libraries (TRL, which we'll meet in Module 5) handle masking for you — assistant_only_loss style settings — but you must know it exists, because when it's misconfigured, the symptoms are subtle quality rot rather than a crash.

Last rule: train the way you'll serve. Same chat template, same system prompt (or same distribution of system prompts), same input structure. Every train/serve mismatch is a small tax on quality; template mismatch is a total one.

Topic 16: Preference Datasets — teaching "better," not just "good"

SFT has a structural blind spot: it only ever shows positive examples. But much of what makes a model good is relative — this answer is better than that one, shorter beats rambling, admitting uncertainty beats confident nonsense, this refusal is graceful and that one is preachy. You can't demonstrate "don't do X" with a dataset that only contains things to do.

Enter the preference dataset. Each row is a triplet:

{"prompt": "My deploy failed with exit code 137, what happened?",
 "chosen":  "Exit 137 means the process was killed - almost always OOM. Check memory limits...",
 "rejected": "There could be many reasons for deployment failures. Have you tried checking logs?"}

Notice the rejected answer isn't wrong — it's just worse: vague, unhelpful, question-deflecting. That's the power of the format: it encodes a direction of improvement that no pile of good examples alone can express. Stage 3 methods (DPO, RLHF — the actual algorithms are Module 3) consume these triplets to push the model's probability mass toward chosen-like outputs and away from rejected-like ones.

Where do the pairs come from? Three sources, in increasing modernity:

1. Human annotation. Show labelers two model outputs, they pick the better one. Gold standard in theory; in practice expensive and noisy — inter-annotator agreement on such comparisons typically runs only ~65–75%. Humans genuinely disagree about "better."
2. AI feedback (RLAIF / LLM-as-judge). A strong model does the ranking, guided by a written rubric or set of principles. Dramatically cheaper, surprisingly competitive with human labels, now the workhorse. (Anthropic's Constitutional AI is the famous lineage here.)
3. Verifiable signals. For code: the chosen answer passes the tests, the rejected one doesn't. For math: correct final answer vs incorrect. Free, perfectly clean labels — use these whenever your domain allows.

Two advanced points that separate practitioners from tutorial-followers:

Length bias is real and will bite you. Both humans and LLM judges systematically prefer longer, more elaborate answers — even when the extra words add nothing. Train on such preferences naively and you manufacture a model that pads everything. (Much of the "why do chatbots ramble" phenomenon traces to exactly this.) Mitigations: length-balance your pairs, instruct judges to penalize padding, or use length-controlled evaluation.

On-policy beats off-policy. Preference pairs built from your own model's outputs (sample 4 responses from the model you're tuning, rank them, take best/worst as chosen/rejected) teach far more effectively than pairs borrowed from other models' outputs — the corrections land in the region where your model actually lives. Public datasets like UltraFeedback or Anthropic HH-RLHF are excellent for study, but the strongest results come from generating pairs on-policy. Keep this in your pocket for Module 3's DPO topic.

Topic 17: Synthetic Datasets — manufacturing your training data

Everything so far assumed you have examples. Usually you have fifty, and need five thousand. The modern answer: use a strong LLM to generate training data for your target model. This is not a fringe trick — it's now the dominant data strategy across the industry, including inside frontier labs.

The founding demonstration: Stanford's Alpaca (2023). They took 175 hand-written seed examples, prompted a strong OpenAI model to generate variations (the Self-Instruct technique), got 52,000 instruction pairs for about $500 of API calls, fine-tuned Llama on them — and produced a startlingly capable model for roughly $600 total. The recipe generalizes:

handwritten seeds (50-200)
   → strong "teacher" model generates variations at scale
   → FILTER aggressively        ← the actual hard part
   → fine-tune the "student"

When the teacher is a big model and the student a small one, this is distillation — compressing expensive intelligence into a cheap model for one task. It's the standard mechanism behind "our 8B specialist beats the giant API model" from Lesson 2.

The generation techniques worth knowing by name:

• Self-Instruct: seeds → "generate 20 more tasks like these, diverse in topic and difficulty."
• Evol-Instruct (from WizardLM): take an instruction and evolve it — "rewrite this to require multi-step reasoning," "add a constraint," "make it handle an edge case." Fights the strong tendency of naive generation to produce easy, samey examples.
• Persona-driven generation: prepend varied personas ("a frustrated sysadmin at 3am," "a non-technical founder," "a student with broken English") to force diversity of phrasing and difficulty. Research like Persona Hub scaled this to a billion personas; you need twenty.
• Textbook-style synthesis: generate dense, pedagogically clean explanatory content rather than Q&A — the approach behind Microsoft's Phi models, which punched absurdly above their size by training on "textbook-quality" synthetic data.

Now the sentence to tattoo somewhere visible: generation is cheap, filtering is the product. Raw synthetic output contains duplicates, subtle errors, teacher clichés ("It's important to note that…" — you've seen the slop), and refusals-copied-as-answers. Train on it unfiltered and you distill the garbage too. The professional filter stack:

1. Exact + near-duplicate removal (embedding similarity from Lesson 2 — cluster and prune).
2. Verifiable checks wherever possible: generated code must execute and pass generated tests; math must reach the checkable answer. Hard verification is worth ten soft judgments.
3. LLM-as-judge scoring against a rubric; keep only top-scoring examples. (Yes — using a model to grade a model's homework. It works far better than it should, with known biases you now know about: length, style.)
4. Human spot-check of a random 5–10%. Your eyes calibrate the whole pipeline.

Three risks for the advanced practitioner:

• Inherited flaws: the student learns the teacher's biases, verbal tics, and mistakes with perfect fidelity. Your data ceiling is your teacher's quality ceiling.
• Model collapse: recursively training models on unfiltered output of models degrades diversity and tail knowledge over generations — a documented phenomenon. Filtering and mixing in real data are the antidotes; this is a reason curation (next lesson) never stops mattering.
• Licensing: some API providers' terms restrict using outputs to train competing models, and some open models' licenses have their own clauses. Before building a commercial product on distilled data, actually read the teacher's terms. Unsexy; occasionally existential.

Lesson 3 complete. You now understand the full data taxonomy: demonstrations (SFT), comparisons (preference), and manufactured data (synthetic) — plus the formatting layer where projects quietly die. The through-line of everything today: the model becomes your dataset. Every quality bar, every bad habit, every bias in the data reappears in the weights.

Lesson 4 finishes Module 2: data curation and cleaning (the craft of deciding what deserves to be in the dataset at all), continued pretraining (the actual way to add knowledge via training, and when it's worth it), and hallucination reduction (pulling together threads we've planted in Topics 14 and 16 into a full strategy). Then we hit Module 3 and start touching GPUs: LoRA, QLoRA, DPO, quantization — where your Lesson 2 "8× memory multiplier" flag finally pays off.

Do the Topic 17 exercise if you do only one — its output literally becomes your Module 3 training set.