SFT Datasets — the raw material

An SFT dataset (Supervised Fine-Tuning dataset) is a collection of demonstrations: here's an input, here's exactly the output we want. Single-turn form:

{"instruction": "Summarize this error log and suggest a fix", 
 "input": "TypeError: cannot read properties of undefined...", 
 "output": "The error occurs because... To fix it, ..."}

Or multi-turn conversation form (what you'll actually use for anything chat-shaped):

{"messages": [
  {"role": "system", "content": "You are a support agent for isDisposable API."},
  {"role": "user", "content": "Why is my API key rejected?"},
  {"role": "assistant", "content": "Usually one of three causes: ..."}
]}

The mechanism from Lesson 1 applies unchanged: the model does next-token prediction on the output, gets a wrongness score, knobs nudge. Which means the deepest truth about SFT data:

Not just the correct answers — the tone, the length, the hedging, the formatting quirks, the mistakes. Ten examples where the assistant starts with "Great question!" and your model will "Great question!" forever. One example with a subtly wrong answer and you've taught wrongness. There is no separate channel for "please be accurate" — demonstrations are the only channel.

How much data do you need? The landmark result here is the LIMA paper (Meta, 2023): they fine-tuned a 65B base model on just 1,000 meticulously curated examples and got quality approaching models trained on millions. The conclusion — sometimes called the superficial alignment hypothesis — is that the model already learned its capabilities during pretraining; SFT mainly teaches which format and persona to use. Practical calibration:

• Style, tone, format, persona: ~500–2,000 excellent examples often suffice.
• Reliable narrow task (classification, structured extraction, domain-specific generation): 1,000–20,000, and coverage of edge cases matters more than raw count.
• Diversity beats volume. 1,000 examples spanning many phrasings, difficulties, and edge cases outperform 10,000 near-duplicates. The FLAN research line showed training across many task types even improves performance on task types never seen — variety teaches "follow the instruction, whatever it is."

Datasets worth knowing by name (you'll see them everywhere on Hugging Face): Alpaca (52K, machine-generated — historically important, quality mediocre by today's standards), Dolly-15k (written by actual Databricks employees — human, clean, small), OpenAssistant/OASST (crowdsourced multi-turn conversations), UltraChat (large-scale synthetic dialogues), LIMA (the famous 1K). Don't train products on these directly — study them as reference anatomy for building your own.