Dataset Cleaning — scrubbing the dirt

Now the proofreader's job. Real datasets — scraped, synthetic, or human-written — are dirty in predictable ways. Knowing the dirt taxonomy means you can hunt systematically instead of hoping.

The taxonomy of dirt:

1. Structural defects: invalid JSON, missing fields, empty outputs, and — the sneakiest — truncated outputs that end mid-sentence (usually because the generating model hit a token limit). Remember Topic 15: outputs are the tokens your model learns to produce. Train on truncated answers and you literally teach it to stop mid-thou—
2. Artifacts: leftover HTML tags, navigation boilerplate ("Click here to subscribe"), markdown debris, and the infamous assistant-isms. Scraped ChatGPT conversations and lazy synthetic data are riddled with "As an AI language model, I cannot…" — train on those and your model inherits refusals for things it should do, plus every verbal tic ("It's important to note that…", "I hope this helps!"). This is the "slop" inheritance from Topic 17, now with a cleaning procedure: a phrase blacklist.
3. Content defects: factually wrong answers, outdated information, answers that ignore the question. Hardest to catch — requires verification (execution for code, LLM-judge with the source in hand for facts).
4. PII (Personally Identifiable Information): emails, phone numbers, names, addresses that leaked in from real conversations or scraped pages. Legal and ethical hazard — models can memorize and regurgitate it. Regex catches emails/phones; NER (named-entity recognition) tools like Microsoft's Presidio catch names and addresses.
5. Language contamination: stray examples in unexpected languages, or mixed-language answers where they shouldn't be. Cheap to catch with a language-ID library.

The pipeline principle: cheap checks first, expensive checks last. Deterministic validation costs nothing; LLM-judging costs money; human review costs the most. Order accordingly:

import json, re
 
REFUSAL_TICS = ["as an ai language model", "i cannot assist", "i'm just an ai"]
 
def keep(example: dict) -> bool:
    out = example.get("messages", [{}])[-1].get("content", "")
    if not out.strip():                       return False   # empty
    if len(out) < 20 or len(out) > 8000:      return False   # length bounds
    if out.rstrip()[-1] not in ".!?`)\"]}":   return False   # truncation smell
    if any(t in out.lower() for t in REFUSAL_TICS): return False
    if re.search(r"\b[\w.+-]+@[\w-]+\.[a-z]{2,}\b", out): return False  # email PII
    return True
 
data = [json.loads(l) for l in open("raw.jsonl")]
clean = [d for d in data if keep(d)]
print(f"kept {len(clean)}/{len(data)}")      # ALWAYS look at this number

Fifteen lines, and it catches a shocking fraction of real-world dirt. Two habits elevate it to professional grade: always print what you removed (a sample of rejects — your filters might be wrongly killing good examples, e.g. that truncation check would reject perfectly valid answers ending in a code block… which is why you eyeball rejects), and track the kill rate per rule. If one rule removes 40% of your data, either your data has a systemic problem or your rule does — both are worth knowing before training.

Cleaning is a loop, not a phase. You'll clean, train, evaluate, discover a failure pattern ("the model keeps apologizing"), trace it back to dirt you missed ("ah — 200 examples with apology openers"), add a rule, re-clean, retrain. Every mature data pipeline is sediment from cycles like this.