Quality Benchmarking — your evals, the real skill

The module's center of gravity. Public benchmarks rank the field; human evals are slow and golden; what ships products is your own eval suite — and here's the thesis, an echo of Topic 13 you'll recognize:

Building the golden set. 30–200 examples of your actual task, each with what "correct" means. Composition matters more than count: typical cases (the daily distribution), edge cases (the weird-but-legal), known failures (every production bug becomes an eval case — your suite grows into a regression battery, exactly like software tests), and adversarial cases (Topic 55's self-red-teaming, institutionalized). Sources, best-first: real user queries, then synthetic expansion via your Topic 17 pipeline, then imagination. And the rule you wrote yourself in Topic 18: firewalled from training, forever — an eval your fine-tune trained on is a mirror, not a measurement.

Grading — the hierarchy, cheapest-and-most-trustworthy first:

1. Code graders: exact match, regex, schema validation, execution and tests. Deterministic, free, incorruptible — Topic 58's verification asymmetry as an evaluation principle. Engineer your tasks toward checkability: asking for structured output isn't just a product choice, it's what makes your evals mechanical.
2. LLM-as-judge — for the fuzzy remainder, done rigorously, because every bias from Topic 74 lives in the machine judge too, plus one new one:
   • Pairwise over absolute, with positions swapped and averaged (judges have position bias — measurably).
   • Rubric-anchored prompts with scored examples (a judge prompt is a prompt; Topic 54 applies entirely).
   • Length-bias control (Topic 16's disease, inherited).
   • Never judge with the model under test — models exhibit self-preference, rating their own style higher. Different (and strong) judge, always.
   • Meta-evaluate: before trusting the judge, score 30–50 items with humans too and measure agreement. A judge that matches humans 85%+ is a scalable instrument; an unvalidated judge is vibes with an API bill.
3. Human spot-checks — the calibration layer keeping 1 and 2 honest, per Topic 17's pattern.

Score per criterion, not one blob — correctness, groundedness, format-compliance, tone as separate numbers (Topic 52's RAG triad generalized), because "quality 7.2" diagnoses nothing while "groundedness dropped, format held" tells you exactly which knob you broke.

And the discipline that makes it all compound — eval-driven development:

This loop is what retroactively activates the entire course. Every knob you've collected — chunk size (49), quant level (24), reranker on/off (52), system-prompt edits (55), model swaps (63), routing thresholds (38), fine-tune checkpoints (26) — was introduced with the refrain "tunable once you can measure." This is the measuring. Change one thing, run the suite, read the per-criterion delta, ship or revert. It's CI for behavior, and teams that have it iterate 10× faster than teams re-vibing every change — not metaphorically, literally: their experiments conclude in minutes with numbers instead of days with arguments.

Two closing honesties. Statistics: models are sampled (Topic 2), so run the suite multiple times or at temperature 0 where appropriate, and respect the noise floor — on a 30-example set, a +3% delta is well within dice-roll territory; celebrate signal, not fluctuation. Tooling: promptfoo, Braintrust, LangSmith, Inspect all package this loop — and a 100-line Python script over JSONL does too. The artifact that matters is the golden set and the habit; the harness is commodity.

---