AI Benchmarks — reading the public scoreboard
A benchmark is a standardized exam: a fixed dataset + a metric + a protocol, so any model can be scored comparably. The families worth recognizing on any model card, organized by what they probe:
| Probes | The names | Status |
|---|---|---|
| Knowledge & reasoning | MMLU → MMLU-Pro, GPQA ("Google-proof" grad questions) | MMLU aging; GPQA current |
| Math | GSM8K → MATH → AIME, FrontierMath | GSM8K saturated |
| Code | HumanEval → SWE-bench, LiveCodeBench (Topic 66) | HumanEval saturated |
| Agents | WebArena, OSWorld, τ-bench (Topics 58/61) | Active frontier |
| Long context | Needle-in-a-haystack → RULER | Needle mostly solved (retrieval ≠ reasoning over context) |
| Instruction following | IFEval | Useful, narrow |
| Everything | Humanity's Last Exam; Arena (next topic) | Current headline metrics |
Notice each row's arrow: benchmarks have lifespans, and they die of three diseases you've already studied in other organs:
- Contamination — Topic 18's train/test leakage, at ecosystem scale. Benchmark questions litter the internet, seep into pretraining corpora, and models "ace" tests they've partially memorized. Detection is genuinely hard, which is why modern designs are contamination-resistant by construction: LiveCodeBench only scores problems published after each model's cutoff — decontamination as benchmark architecture.
- Saturation — when everyone scores 90%+, the exam measures nothing (differences are noise and error-label disputes). GSM8K and HumanEval numbers on a 2026 model card are decoration, not signal.
- Goodhart — Topic 27's law, again: the moment a benchmark drives headlines and valuations, labs optimize toward it, and the score decouples from the real capability it proxied. "Benchmarketing" is the industry term, and it's earned.
So develop the professional habit of reading benchmark tables adversarially: check the protocol footnotes (5-shot vs 0-shot, with/without chain-of-thought or thinking budgets — cross-table comparisons with mismatched settings are meaningless); check who ran it (self-reported numbers on the vendor's own blog deserve one raised eyebrow); check for variance (a single run of a sampled model is one dice roll — Topic 2); prefer held-private test sets and post-cutoff designs. And hold the honest conclusion: benchmarks are decent for coarse ranking and progress-tracking across the field, and nearly useless for your decision — because your task is not MMLU, and a 2-point MMLU gap predicts nothing about which model writes better support replies for isDisposable. That gap between public scores and your answer is precisely what Topic 75 exists to close.
Summary
Benchmarks = standardized exams with lifespans, dying of contamination, saturation, and Goodhart. Read them adversarially (protocol, source, variance), use them for coarse field-level ranking — never as a substitute for evaluating your own task.
Mental model
University entrance exams. Genuinely informative at population scale, gameable by teaching-to-the-test, inflated once the questions leak to tutors — and no hiring manager confuses a test score with a work sample.
Mistakes to avoid
- Choosing your production model off a leaderboard delta. A model 2 points "worse" globally may be 20 points better on your task after fine-tuning — only your eval knows.
- Quoting saturated benchmarks as evidence. Citing HumanEval in 2026 signals exactly the secondhand knowledge this course exists to cure.
Exercise
Pull up the model cards of two current models you care about. For their headline benchmark table: identify the shot/CoT protocol of each number, find one benchmark where their settings don't match, flag any self-reported score, and find one saturated benchmark still being cited. Twenty minutes of adversarial reading — the skill itself.