Proving It — Benchmarks, Judges, Stopwatches, and Invoices · Modern LLM Engineering From Scratch

Welcome to Module 10: Evaluation — the module this entire course has been foreshadowing. Count the IOUs: "measure it on your eval set" (Topics 24, 38, 54, 55), "golden set" (Topic 52), "firewall your evals" (Topic 18), "vibes aren't evidence" (everywhere). Today every IOU gets paid. This is the discipline that separates "my model seems better" from "my model is better — here's the number, here's the variance, here's what it costs." And it's genuinely the highest-leverage skill in applied AI, because everyone can call APIs; few can prove their system improved.

Topic 73: AI Benchmarks — reading the public scoreboard

A benchmark is a standardized exam: a fixed dataset + a metric + a protocol, so any model can be scored comparably. The families worth recognizing on any model card, organized by what they probe:

Probes	The names	Status
Knowledge & reasoning	MMLU → MMLU-Pro, GPQA ("Google-proof" grad questions)	MMLU aging; GPQA current
Math	GSM8K → MATH → AIME, FrontierMath	GSM8K saturated
Code	HumanEval → SWE-bench, LiveCodeBench (Topic 66)	HumanEval saturated
Agents	WebArena, OSWorld, τ-bench (Topics 58/61)	Active frontier
Long context	Needle-in-a-haystack → RULER	Needle mostly solved (retrieval ≠ reasoning over context)
Instruction following	IFEval	Useful, narrow
Everything	Humanity's Last Exam; Arena (next topic)	Current headline metrics

Notice each row's arrow: benchmarks have lifespans, and they die of three diseases you've already studied in other organs:

Contamination — Topic 18's train/test leakage, at ecosystem scale. Benchmark questions litter the internet, seep into pretraining corpora, and models "ace" tests they've partially memorized. Detection is genuinely hard, which is why modern designs are contamination-resistant by construction: LiveCodeBench only scores problems published after each model's cutoff — decontamination as benchmark architecture.
Saturation — when everyone scores 90%+, the exam measures nothing (differences are noise and error-label disputes). GSM8K and HumanEval numbers on a 2026 model card are decoration, not signal.
Goodhart — Topic 27's law, again: the moment a benchmark drives headlines and valuations, labs optimize toward it, and the score decouples from the real capability it proxied. "Benchmarketing" is the industry term, and it's earned.

So develop the professional habit of reading benchmark tables adversarially: check the protocol footnotes (5-shot vs 0-shot, with/without chain-of-thought or thinking budgets — cross-table comparisons with mismatched settings are meaningless); check who ran it (self-reported numbers on the vendor's own blog deserve one raised eyebrow); check for variance (a single run of a sampled model is one dice roll — Topic 2); prefer held-private test sets and post-cutoff designs. And hold the honest conclusion: benchmarks are decent for coarse ranking and progress-tracking across the field, and nearly useless for your decision — because your task is not MMLU, and a 2-point MMLU gap predicts nothing about which model writes better support replies for isDisposable. That gap between public scores and your answer is precisely what Topic 75 exists to close.

Summary

Benchmarks = standardized exams with lifespans, dying of contamination, saturation, and Goodhart. Read them adversarially (protocol, source, variance), use them for coarse field-level ranking — never as a substitute for evaluating your own task.

Mental model

University entrance exams. Genuinely informative at population scale, gameable by teaching-to-the-test, inflated once the questions leak to tutors — and no hiring manager confuses a test score with a work sample.

Mistakes to avoid

Choosing your production model off a leaderboard delta. A model 2 points "worse" globally may be 20 points better on your task after fine-tuning — only your eval knows.
Quoting saturated benchmarks as evidence. Citing HumanEval in 2026 signals exactly the secondhand knowledge this course exists to cure.

Exercise

Pull up the model cards of two current models you care about. For their headline benchmark table: identify the shot/CoT protocol of each number, find one benchmark where their settings don't match, flag any self-reported score, and find one saturated benchmark still being cited. Twenty minutes of adversarial reading — the skill itself.

Topic 74: Human Evals — the gold standard, with tarnish

For everything with ground truth, machines grade (next topic). But helpfulness, tone, writing quality, "which answer would you rather receive" — these have no answer key, and human judgment is the gold standard. A standard you already know is tarnished: Topic 16 taught you inter-annotator agreement on "which answer is better" runs ~65–75%. Humans genuinely disagree about quality; every human-eval design decision below exists to work around that fact.

The formats, in order of reliability:

Pairwise comparison (A/B) — the workhorse, for a deep psychometric reason: humans are bad at absolute scales ("rate this 1–10" drifts by rater, mood, and the last five items seen) and good at comparisons ("A or B?"). And notice: pairwise human evals produce (prompt, chosen, rejected) — literally Topic 16's preference data. Evaluation and training data are the same artifact viewed from two sides; a good human-eval pipeline is a DPO dataset factory running in disguise.
Rubric-anchored scoring — absolute scores made tolerable by explicit criteria and calibration examples. Use when you need per-dimension diagnostics rather than a winner.
Task success — the product-level truth: did the user accomplish the thing? Measured via A/B tests in production (retention, completion, escalation rates) and the humble 👍/👎 — which, per Topic 28, is KTO training data accumulating for free.

The famous instance: Chatbot Arena (LMArena). Anonymous crowdsourced battles — real users, real prompts, blind pairwise votes — aggregated into ratings via the Bradley-Terry model (delightfully, the same math Topic 27's reward models train on: "chosen beats rejected" as a statistical model, whether fitting a leaderboard or a reward head). Its strengths are real: blind, diverse, ungameable-by-memorization. Its weaknesses are Topic 16's biases at civilization scale: style wins votes — longer, confidently formatted, emoji-friendly answers beat terser correct ones often enough that style-controlled variants of the leaderboard had to be built — plus population bias (the voters aren't your users) and growing gaming pressure as the stakes rose. Arena is the best public vibes aggregator in existence; it is still vibes, aggregated.

Running your own human evals — the checklist that makes them science: blind (raters never know which system produced what); position-randomized (raters favor the first-shown answer — position bias is large and universal); rubric'd (define "better" or inherit 25 private definitions); multi-rater with agreement measured (if your raters agree at 60%, your eval's ceiling is 60% — report it); and powered (a 55/45 preference split needs hundreds of comparisons to distinguish from coin-flipping; ten samples prove nothing — the Topic 75 variance sermon, previewed). The costs — money, days, coordination — are exactly why the field built the machine judge that Topic 75 will teach you to trust carefully.

Summary

Human judgment anchors everything unmeasurable — via pairwise comparisons (reliable, doubles as preference data), rubric'd scoring, and production A/B. Arena industrialized it, style bias and all. Rigor = blind, position-randomized, rubric'd, multi-rater, adequately powered.

Mental model

A wine competition. Blind tasting, randomized pour order, score sheets, multiple judges — because sighted, ordered, freeform tasting measures the label, the sequence, and the judge's lunch. And even done perfectly, the judges disagree on a third of the glasses.

Mistakes to avoid

Showing evaluators outputs with model names attached. Brand priors swamp quality differences instantly; blinding is not optional hygiene, it's the whole method.
Declaring victory from 10 comparisons. At n=10, a 7–3 split is comfortably explainable by chance — collect real sample sizes or report "suggestive," not "better."

Exercise

Run a micro human eval tonight: take 10 prompts from your domain, generate answers from two models, and have two friends (or you-on-different-days, blinded via shuffled files) pick winners with positions randomized. Compute: win rate and rater agreement. Whatever the win rate, the agreement number — likely 60–80% — is the lesson: you've now felt the noise floor every eval in this field sits on.

Topic 75: Quality Benchmarking — your evals, the real skill

The module's center of gravity. Public benchmarks rank the field; human evals are slow and golden; what ships products is your own eval suite — and here's the thesis, an echo of Topic 13 you'll recognize:

The dataset is the spec (Topic 13). The eval set is the spec, enforced.

Building the golden set. 30–200 examples of your actual task, each with what "correct" means. Composition matters more than count: typical cases (the daily distribution), edge cases (the weird-but-legal), known failures (every production bug becomes an eval case — your suite grows into a regression battery, exactly like software tests), and adversarial cases (Topic 55's self-red-teaming, institutionalized). Sources, best-first: real user queries, then synthetic expansion via your Topic 17 pipeline, then imagination. And the rule you wrote yourself in Topic 18: firewalled from training, forever — an eval your fine-tune trained on is a mirror, not a measurement.

Grading — the hierarchy, cheapest-and-most-trustworthy first:

Code graders: exact match, regex, schema validation, execution and tests. Deterministic, free, incorruptible — Topic 58's verification asymmetry as an evaluation principle. Engineer your tasks toward checkability: asking for structured output isn't just a product choice, it's what makes your evals mechanical.
LLM-as-judge — for the fuzzy remainder, done rigorously, because every bias from Topic 74 lives in the machine judge too, plus one new one:
- Pairwise over absolute, with positions swapped and averaged (judges have position bias — measurably).
- Rubric-anchored prompts with scored examples (a judge prompt is a prompt; Topic 54 applies entirely).
- Length-bias control (Topic 16's disease, inherited).
- Never judge with the model under test — models exhibit self-preference, rating their own style higher. Different (and strong) judge, always.
- Meta-evaluate: before trusting the judge, score 30–50 items with humans too and measure agreement. A judge that matches humans 85%+ is a scalable instrument; an unvalidated judge is vibes with an API bill.
Human spot-checks — the calibration layer keeping 1 and 2 honest, per Topic 17's pattern.

Score per criterion, not one blob — correctness, groundedness, format-compliance, tone as separate numbers (Topic 52's RAG triad generalized), because "quality 7.2" diagnoses nothing while "groundedness dropped, format held" tells you exactly which knob you broke.

And the discipline that makes it all compound — eval-driven development:

This loop is what retroactively activates the entire course. Every knob you've collected — chunk size (49), quant level (24), reranker on/off (52), system-prompt edits (55), model swaps (63), routing thresholds (38), fine-tune checkpoints (26) — was introduced with the refrain "tunable once you can measure." This is the measuring. Change one thing, run the suite, read the per-criterion delta, ship or revert. It's CI for behavior, and teams that have it iterate 10× faster than teams re-vibing every change — not metaphorically, literally: their experiments conclude in minutes with numbers instead of days with arguments.

Two closing honesties. Statistics: models are sampled (Topic 2), so run the suite multiple times or at temperature 0 where appropriate, and respect the noise floor — on a 30-example set, a +3% delta is well within dice-roll territory; celebrate signal, not fluctuation. Tooling: promptfoo, Braintrust, LangSmith, Inspect all package this loop — and a 100-line Python script over JSONL does too. The artifact that matters is the golden set and the habit; the harness is commodity.

Summary

Your eval suite = a firewalled golden set (typical + edge + regression + adversarial cases) graded by the hierarchy (code > validated LLM-judge > human spot-checks), scored per-criterion, wired into a change→measure→ship/revert loop. It's the spec, enforced — and the activation key for every knob in this course.

Mental model

Unit tests plus CI, for a component that's probabilistic. No serious engineer edits load-bearing code without tests; no serious AI engineer edits prompts, models, or pipelines without evals. Same discipline, fuzzier assertions.

Mistakes to avoid

Judging with the model being tested, or with an unvalidated judge. Self-preference and unmeasured judge-bias produce confident, worthless numbers — meta-evaluate first, always.
Building the eval set after months of development. By then your intuitions are contaminated by what your system happens to do well. Golden set first, or at worst, now.

Exercise · the module's centerpiece — do this one

Build the real eval suite for your Lesson 6 fine-tune or Module 6 RAG system: 30 cases (20 typical, 5 edge, 5 from actual observed failures), a code grader for everything checkable, a pairwise LLM-judge (position-swapped, different model) for the rest — then meta-evaluate the judge against your own labels on 15 items and report the agreement. Finally, run the loop once for real: change one thing (a prompt line, the chunk size), rerun, and write the one-sentence verdict with numbers. You now possess the single most hireable artifact-producing skill in applied AI.

Topic 76: Speed Benchmarking — the stopwatch, held correctly

You've measured speed since Topic 34's exercise. This topic upgrades the stopwatch into methodology, because naive latency numbers are the most confidently-wrong numbers in engineering. The metrics themselves you know — TTFT (prefill's bill), TPOT/inter-token latency (decode's bill), total latency, and system throughput (tokens/sec across all streams). What's new is the discipline:

1. Percentiles, never averages. Report p50 / p95 / p99. Latency distributions have long tails — GC pauses, cache misses, prefill collisions (Topic 36's chunked-prefill problem) — and the average hides them while your users live in them: at p99 = 8s, one request in a hundred feels broken, and your heaviest users hit that daily. "Average 900ms" and "p99 8s" describe the same broken service.

2. Measure under your real load shape. Topic 36 taught the gap: single-stream and loaded-system performance differ 10–30×. So benchmark both ends and the curve between: ramp concurrency and plot latency-vs-throughput until you find the knee — the saturation point where throughput plateaus and latency turns vertical. Your capacity is the knee minus safety margin; pricing off single-stream numbers (or demoing off loaded ones) is self-deception in opposite directions.

3. Control the confounders — each one a course callback: input length (prefill scales with it — Topic 34; benchmark with your traffic's length distribution, not synthetic 100-token prompts), output length (fix it or normalize per-token), warm vs cold (first request pays model load and empty caches — Topic 37; report warm, note cold), cache state (prefix-cache hits fake TTFT miracles — Topic 55; measure hit and miss separately), and client-vs-server timing (network is in the user's number, not the engine's — know which you're quoting).

4. Reconcile against theory — the formula as a debugging instrument. You own the ceiling: bandwidth ÷ bytes (Topic 34). Measured far above ceiling → your measurement is broken (cache hit, wrong token count). Measured far below → your config is broken — offloading engaged (Topic 31), Flash Attention off, batch misconfigured. The gap between theory and measurement isn't disappointment; it's a diagnosis pointing at a specific subsystem. Tools: llama-bench (Topic 39), vLLM's benchmark scripts, k6/locust-class load generators — and the deliverable is a small honest table: config → TTFT p50/p95 → TPOT p50/p95 → max throughput within your latency SLO.

Summary

Speed benchmarking = percentiles not averages, your load shape not synthetic, confounders controlled (lengths, warmth, caches, network), the latency-throughput knee located, and every number reconciled against the Topic 34 ceiling — turning "it feels fast" into a table that survives cross-examination.

Mental model

Timing a restaurant not by one quiet-Tuesday meal but across a full Friday service: median table and the unluckiest table (p99), finding how many covers the kitchen handles before everything queues (the knee) — and if a dish arrives faster than physically cookable, suspecting it was pre-plated (cache hit), not miraculous.

Mistakes to avoid

Quoting the average. The tail is the user experience; a benchmark without p95/p99 is marketing.
Benchmarking cold-start once, or warm-cache always, without labeling which. The two differ by an order of magnitude and answer different questions — mixing them poisons every comparison downstream.

Exercise

Re-run your Topic 34 measurement, upgraded: 30 requests against your local model with your actual typical prompt length, recording TTFT and TPOT each time. Compute p50/p95, compare TPOT-p50 to your theoretical ceiling, and explain any gap over 2× using the confounder list. One table, one paragraph of reconciliation — your first methodologically defensible speed report.

Topic 77: Cost-per-Token Analysis — the invoice, decoded

The final measurement: money. Token prices look simple — $/million in, $/million out — but a product's economics live in the multipliers between "price per token" and "cost per user per month." Here's a single real-shaped request, drawn by cost rather than by tokens, which is where the surprises hide:

Read that bar the way an accountant would: the smallest token count (500 output) is the largest cost segment, because output runs ~5× input price — and the largest token count (the 2K system prompt) is nearly invisible, because caching cut it 10×. Two course lessons, now denominated in currency: Topic 37's prefix caching is a line-item discount, and Topic 67's thinking tokens — billed as output — mean reasoning models don't add 20% to your costs, they multiply your most expensive segment several-fold.

The hidden multipliers — the gap between "price per token" and "invoice," each one a callback wearing a dollar sign:

Conversation history compounds. Topic 5's whiteboard rule: every turn resends everything, so turn 12 carries turns 1–11 as input. A conversation's total cost isn't (turns × per-turn) — it's the sum of growing prefixes, roughly quadratic in conversation length. Long-chat products live or die on history pruning and summarization (Topic 53) as cost engineering, not just quality engineering.
Agent chains multiply. Topic 60's measured ~15× for multi-agent; even single-agent loops re-read accumulated context per step. An "AI feature" that's secretly a 12-step agent costs 12+ requests' worth per invocation.
Deliberate multiplication: voting (Topic 59) is N× by design; retries and fallbacks (Topic 70) add their share; consistency checks (Topic 21) are 5×. All correct choices — all invoiced.

The chain to unit economics — the calculation every AI product must survive:

$0.021/request  (the diagram)
× 200 requests/user/month  (your telemetry)
= $4.20/user/month inference cost
vs $15/month subscription → 28% of revenue on tokens alone

Twenty-eight percent gross-margin drag before servers, salaries, or marketing — that number, not the per-token price, is what the levers exist to attack. And now the levers from Topic 38 return with their price tags on: routing is the violent one — send the easy 80% to a model ~25× cheaper and blended cost becomes 0.8×($0.021/25) + 0.2×$0.021 ≈ $0.005, a 77% cut; output discipline (500→300 tokens) and prompt dieting (trim a redundant RAG chunk) are ~14% each for an afternoon's work; batch tiers halve everything offline (Topic 36); and the self-host break-even you computed in Topics 11 and 36 is the endgame lever once volume justifies ops.

Which delivers the module's unifying picture: for every candidate configuration — model × quant × prompt × pipeline — you can now produce three honest numbers: quality (Topic 75's suite), speed (Topic 76's percentiles), cost (this topic's per-request figure). Plot quality against cost and the frontier appears: some configs dominate others outright (better and cheaper — discard the dominated), and the survivors form a curve where your product picks its point deliberately. Topic 38 called this triangle a positioning problem; Module 10's achievement is making all three axes measurable, so positioning becomes arithmetic instead of argument.

Last discipline: metering in production. Topic 70's per-request cost log is the instrument; add per-feature attribution (which feature bleeds?), per-customer margin (is the enterprise tier subsidizing free users or vice versa?), and anomaly alarms — because the classic AI-product financial incident is a silent retry loop or a runaway agent discovered via the monthly invoice instead of a same-day alert.

Summary

Request cost = input×price + output×(~5×price), with caching discounting the static prefix and history/agents/voting as multipliers. Chain it to per-user-per-month, attack it with routing (biggest), dieting, caps, and batch tiers — then plot cost against your eval scores and choose on the frontier, metered live in production.

Mental model

Restaurant food-cost percentage. No chef prices a menu off ingredient prices alone — they cost each plate (portioning = output caps, prep reuse = caching, cheap-cut dishes for most orders = routing), track food-cost % weekly (metering), and re-engineer the menu when a dish's margin rots. Token economics is menu engineering.

Mistakes to avoid

Estimating costs from a single fresh request. The multipliers — history resend, agent steps, retries — routinely make real per-task cost 5–20× the naive per-request math; model the conversation and the chain, not the turn.
Discovering unit economics from the invoice. If cost-per-request isn't a logged, graphed, alarmed number from week one, your margin is an unmonitored variable in production.

Exercise · module capstone

Complete the frontier for your own product spec (the one from Topics 38/70): take three candidate configs — e.g., fine-tuned 8B self-hosted, frontier API direct, routed hybrid — and for each produce the three numbers: eval score (your Topic 75 suite), p95 latency (Topic 76 method), and cost per user per month (this topic's chain, with your realistic usage assumptions). Put them in one table, identify any dominated config, and write the one-paragraph recommendation. That table is the executive summary of everything Module 10 taught — and it's the exact artifact that decides real architecture meetings.

Module 10: complete. The measurement stack, assembled: public benchmarks read adversarially for field-level context, human judgment deployed with blinding and statistics, your own golden-set suite as the enforced spec, speed measured in percentiles against theoretical ceilings, and cost chained from tokens to margin — converging on a quality/speed/cost frontier where decisions become arithmetic. The course's longest-running refrain — measure, don't vibe — is now a toolkit you own end to end.

Next: Module 11 — Real-World Skills. The final module. Everything assembles into build playbooks: chatbots done properly (the full stack from Modules 6–10 in one product shape), AI copilots (embedding intelligence inside existing workflows), AI automation and SaaS workflows (the business shapes that actually sell), AI coding workflows (using the tools of Module 8's favorite domain like a professional), orchestration systems (Module 7 at organizational scale), and AI product thinking — the judgment layer over everything: what to build, what not to, where the moat is when everyone has the same models. One module between you and the end of the curriculum.