Data Curation — the editor-in-chief job

Curation and cleaning get used interchangeably, but they're different jobs. Curation = deciding the composition: what goes in, in what proportions, at what difficulty. Cleaning = fixing defects in what's already in. Curation is the editor-in-chief choosing which stories run; cleaning is the proofreader fixing typos. Today's topic is the editor's chair.

Decision 1: The mixture. A dataset isn't a pile, it's a recipe with proportions. Suppose you're fine-tuning a support bot: 60% support conversations, 15% structured-output examples (your JSON schemas), 10% edge cases and graceful failures, and — crucially — 15% general instruction data (normal varied chat). That last ingredient isn't filler: it's the standard defense against catastrophic forgetting from Topic 12. Training only on your narrow task pulls the model's knobs so far toward it that general ability decays; mixing general data keeps the knobs anchored. Labs obsess over mixture ratios at pretraining scale (how much code? how much multilingual?) — the same logic applies to your 5,000 examples. Code in the mixture, interestingly, is widely believed to improve reasoning generally, not just coding.

Decision 2: Deduplication. Duplicates are the most damaging invisible flaw in datasets. Three levels:

• Exact duplicates — trivial to catch (hash each example).
• Near-duplicates — same content, trivial differences (whitespace, one synonym). Caught with fingerprinting techniques like MinHash: think of it as a smudged thumbprint that still matches even if the text changed slightly.
• Semantic duplicates — different words, same idea. Caught with embeddings from Lesson 2: embed everything, flag pairs above ~0.9 cosine similarity.

Why care so much? A duplicated example gets its knob-nudges applied twice — you've silently doubled its weight in the recipe without deciding to. At scale, duplication drives memorization (the model regurgitates training text verbatim) and wastes compute. One of the clearest findings from open pretraining efforts (Falcon's RefinedWeb, FineWeb): aggressive deduplication alone measurably improves the resulting model.

Decision 3: Quality filtering — the "better beats more" revolution. The modern approach at scale: train a small, cheap classifier to score every document (FineWeb-Edu famously scored web pages on educational value, keeping only the top slice), and train on the survivors. Result, repeated across many projects: a smaller, well-filtered dataset beats a larger unfiltered one. The Phi models from Lesson 3 are this principle taken to its extreme. At your fine-tuning scale, the equivalent is the LLM-as-judge rubric filter you built in the Topic 17 exercise.

Decision 4: Decontamination — the one beginners never think of. If your training data accidentally contains examples from your evaluation set (or public benchmarks you'll report numbers on), your eval scores become lies — the model isn't solving the test, it memorized it. This is train/test leakage, and it's rampant: benchmark questions litter the internet and sneak into scraped and synthetic data. Standard defense: n-gram overlap checks between training data and every eval you use, remove matches. When you build your own eval set (Module 10), build it first and firewall it from the training pipeline forever.

Decision 5: Difficulty distribution. All-easy examples teach nothing new; all-brutal examples make training unstable and teach the model to flail. You want a gradient — mostly solid mid-difficulty, meaningful hard tail, some easy anchors. This is why Evol-Instruct (Lesson 3) exists: naive generation clusters at "easy."

---