Speed Benchmarking — the stopwatch, held correctly

Topic 76 of 90Module 10: Evaluation3 min read

You've measured speed since Topic 34's exercise. This topic upgrades the stopwatch into methodology, because naive latency numbers are the most confidently-wrong numbers in engineering. The metrics themselves you know — TTFT (prefill's bill), TPOT/inter-token latency (decode's bill), total latency, and system throughput (tokens/sec across all streams). What's new is the discipline:

1. Percentiles, never averages. Report p50 / p95 / p99. Latency distributions have long tails — GC pauses, cache misses, prefill collisions (Topic 36's chunked-prefill problem) — and the average hides them while your users live in them: at p99 = 8s, one request in a hundred feels broken, and your heaviest users hit that daily. "Average 900ms" and "p99 8s" describe the same broken service.

2. Measure under your real load shape. Topic 36 taught the gap: single-stream and loaded-system performance differ 10–30×. So benchmark both ends and the curve between: ramp concurrency and plot latency-vs-throughput until you find the knee — the saturation point where throughput plateaus and latency turns vertical. Your capacity is the knee minus safety margin; pricing off single-stream numbers (or demoing off loaded ones) is self-deception in opposite directions.

3. Control the confounders — each one a course callback: input length (prefill scales with it — Topic 34; benchmark with your traffic's length distribution, not synthetic 100-token prompts), output length (fix it or normalize per-token), warm vs cold (first request pays model load and empty caches — Topic 37; report warm, note cold), cache state (prefix-cache hits fake TTFT miracles — Topic 55; measure hit and miss separately), and client-vs-server timing (network is in the user's number, not the engine's — know which you're quoting).

4. Reconcile against theory — the formula as a debugging instrument. You own the ceiling: bandwidth ÷ bytes (Topic 34). Measured far above ceiling → your measurement is broken (cache hit, wrong token count). Measured far below → your config is broken — offloading engaged (Topic 31), Flash Attention off, batch misconfigured. The gap between theory and measurement isn't disappointment; it's a diagnosis pointing at a specific subsystem. Tools: llama-bench (Topic 39), vLLM's benchmark scripts, k6/locust-class load generators — and the deliverable is a small honest table: config → TTFT p50/p95 → TPOT p50/p95 → max throughput within your latency SLO.

Summary

Speed benchmarking = percentiles not averages, your load shape not synthetic, confounders controlled (lengths, warmth, caches, network), the latency-throughput knee located, and every number reconciled against the Topic 34 ceiling — turning "it feels fast" into a table that survives cross-examination.

Mental model

Timing a restaurant not by one quiet-Tuesday meal but across a full Friday service: median table and the unluckiest table (p99), finding how many covers the kitchen handles before everything queues (the knee) — and if a dish arrives faster than physically cookable, suspecting it was pre-plated (cache hit), not miraculous.

Mistakes to avoid

  • Quoting the average. The tail is the user experience; a benchmark without p95/p99 is marketing.
  • Benchmarking cold-start once, or warm-cache always, without labeling which. The two differ by an order of magnitude and answer different questions — mixing them poisons every comparison downstream.

Exercise

Re-run your Topic 34 measurement, upgraded: 30 requests against your local model with your actual typical prompt length, recording TTFT and TPOT each time. Compute p50/p95, compare TPOT-p50 to your theoretical ceiling, and explain any gap over 2× using the confounder list. One table, one paragraph of reconciliation — your first methodologically defensible speed report.