Human Evals — the gold standard, with tarnish

For everything with ground truth, machines grade (next topic). But helpfulness, tone, writing quality, "which answer would you rather receive" — these have no answer key, and human judgment is the gold standard. A standard you already know is tarnished: Topic 16 taught you inter-annotator agreement on "which answer is better" runs ~65–75%. Humans genuinely disagree about quality; every human-eval design decision below exists to work around that fact.

The formats, in order of reliability:

• Pairwise comparison (A/B) — the workhorse, for a deep psychometric reason: humans are bad at absolute scales ("rate this 1–10" drifts by rater, mood, and the last five items seen) and good at comparisons ("A or B?"). And notice: pairwise human evals produce (prompt, chosen, rejected) — literally Topic 16's preference data. Evaluation and training data are the same artifact viewed from two sides; a good human-eval pipeline is a DPO dataset factory running in disguise.
• Rubric-anchored scoring — absolute scores made tolerable by explicit criteria and calibration examples. Use when you need per-dimension diagnostics rather than a winner.
• Task success — the product-level truth: did the user accomplish the thing? Measured via A/B tests in production (retention, completion, escalation rates) and the humble 👍/👎 — which, per Topic 28, is KTO training data accumulating for free.

The famous instance: Chatbot Arena (LMArena). Anonymous crowdsourced battles — real users, real prompts, blind pairwise votes — aggregated into ratings via the Bradley-Terry model (delightfully, the same math Topic 27's reward models train on: "chosen beats rejected" as a statistical model, whether fitting a leaderboard or a reward head). Its strengths are real: blind, diverse, ungameable-by-memorization. Its weaknesses are Topic 16's biases at civilization scale: style wins votes — longer, confidently formatted, emoji-friendly answers beat terser correct ones often enough that style-controlled variants of the leaderboard had to be built — plus population bias (the voters aren't your users) and growing gaming pressure as the stakes rose. Arena is the best public vibes aggregator in existence; it is still vibes, aggregated.

Running your own human evals — the checklist that makes them science: blind (raters never know which system produced what); position-randomized (raters favor the first-shown answer — position bias is large and universal); rubric'd (define "better" or inherit 25 private definitions); multi-rater with agreement measured (if your raters agree at 60%, your eval's ceiling is 60% — report it); and powered (a 55/45 preference split needs hundreds of comparisons to distinguish from coin-flipping; ten samples prove nothing — the Topic 75 variance sermon, previewed). The costs — money, days, coordination — are exactly why the field built the machine judge that Topic 75 will teach you to trust carefully.

---