Synthetic Datasets — manufacturing your training data

Everything so far assumed you have examples. Usually you have fifty, and need five thousand. The modern answer: use a strong LLM to generate training data for your target model. This is not a fringe trick — it's now the dominant data strategy across the industry, including inside frontier labs.

The founding demonstration: Stanford's Alpaca (2023). They took 175 hand-written seed examples, prompted a strong OpenAI model to generate variations (the Self-Instruct technique), got 52,000 instruction pairs for about $500 of API calls, fine-tuned Llama on them — and produced a startlingly capable model for roughly $600 total. The recipe generalizes:

handwritten seeds (50-200)
   → strong "teacher" model generates variations at scale
   → FILTER aggressively        ← the actual hard part
   → fine-tune the "student"

When the teacher is a big model and the student a small one, this is distillation — compressing expensive intelligence into a cheap model for one task. It's the standard mechanism behind "our 8B specialist beats the giant API model" from Lesson 2.

The generation techniques worth knowing by name:

• Self-Instruct: seeds → "generate 20 more tasks like these, diverse in topic and difficulty."
• Evol-Instruct (from WizardLM): take an instruction and evolve it — "rewrite this to require multi-step reasoning," "add a constraint," "make it handle an edge case." Fights the strong tendency of naive generation to produce easy, samey examples.
• Persona-driven generation: prepend varied personas ("a frustrated sysadmin at 3am," "a non-technical founder," "a student with broken English") to force diversity of phrasing and difficulty. Research like Persona Hub scaled this to a billion personas; you need twenty.
• Textbook-style synthesis: generate dense, pedagogically clean explanatory content rather than Q&A — the approach behind Microsoft's Phi models, which punched absurdly above their size by training on "textbook-quality" synthetic data.

Now the sentence to tattoo somewhere visible: generation is cheap, filtering is the product. Raw synthetic output contains duplicates, subtle errors, teacher clichés ("It's important to note that…" — you've seen the slop), and refusals-copied-as-answers. Train on it unfiltered and you distill the garbage too. The professional filter stack:

1. Exact + near-duplicate removal (embedding similarity from Lesson 2 — cluster and prune).
2. Verifiable checks wherever possible: generated code must execute and pass generated tests; math must reach the checkable answer. Hard verification is worth ten soft judgments.
3. LLM-as-judge scoring against a rubric; keep only top-scoring examples. (Yes — using a model to grade a model's homework. It works far better than it should, with known biases you now know about: length, style.)
4. Human spot-check of a random 5–10%. Your eyes calibrate the whole pipeline.

Three risks for the advanced practitioner:

• Inherited flaws: the student learns the teacher's biases, verbal tics, and mistakes with perfect fidelity. Your data ceiling is your teacher's quality ceiling.
• Model collapse: recursively training models on unfiltered output of models degrades diversity and tail knowledge over generations — a documented phenomenon. Filtering and mixing in real data are the antidotes; this is a reason curation (next lesson) never stops mattering.
• Licensing: some API providers' terms restrict using outputs to train competing models, and some open models' licenses have their own clauses. Before building a commercial product on distilled data, actually read the teacher's terms. Unsexy; occasionally existential.

---

Lesson 3 complete. You now understand the full data taxonomy: demonstrations (SFT), comparisons (preference), and manufactured data (synthetic) — plus the formatting layer where projects quietly die. The through-line of everything today: the model becomes your dataset. Every quality bar, every bad habit, every bias in the data reappears in the weights.

Lesson 4 finishes Module 2: data curation and cleaning (the craft of deciding what deserves to be in the dataset at all), continued pretraining (the actual way to add knowledge via training, and when it's worth it), and hallucination reduction (pulling together threads we've planted in Topics 14 and 16 into a full strategy). Then we hit Module 3 and start touching GPUs: LoRA, QLoRA, DPO, quantization — where your Lesson 2 "8× memory multiplier" flag finally pays off.

Do the Topic 17 exercise if you do only one — its output literally becomes your Module 3 training set.