Inference Optimization — the one insight that organizes everything
Synthesis time. Everything in this lesson — starved cooks, the pantry, the growing minutes, the cabinet walks — compresses into one organizing distinction and one formula you'll use for the rest of your career.
The two phases of every request (met in Topic 32, formalized now):
- Prefill: the prompt's tokens processed in parallel — huge matrix multiplies, tensor cores saturated. This phase is compute-bound: the chefs are the bottleneck, and it determines TTFT (time to first token) — the pause before the response starts, scaling with prompt length.
- Decode: one token at a time against the cache. And here's the fact that reorganizes your whole mental model: to generate each single token, the GPU must read every weight of the model out of VRAM once — all 4.5 GB of your Q4 8B, per token, every token. The arithmetic per token is trivial for the cooks; the delivery of the entire pantry contents each step is the wall. Decode is memory-bandwidth-bound, and it determines the tokens-per-second stream rate.
Which yields the formula — the single most useful equation in local AI:
decode tokens/sec ceiling ≈ memory bandwidth ÷ bytes read per token (weights + KV cache)
Worked examples, using Topic 30's table:
| Setup | Bytes/token | Bandwidth | Ceiling | Realistic |
|---|---|---|---|---|
| RTX 4090, 8B @ Q4 | ~4.5 GB | 1 TB/s | ~220 tok/s | ~120–160 |
| M4 base, 8B @ Q4 | ~4.5 GB | 120 GB/s | ~27 tok/s | ~15–20 |
| M4 base, 8B @ FP16 | ~16 GB | 120 GB/s | ~7 tok/s | ~4–6 |
| H100, 70B @ Q4 | ~40 GB | 3.35 TB/s | ~84 tok/s | ~50–65 |
(Realistic lands below ceiling due to overheads and growing KV-cache reads.) Sit with what this table quietly proves:
- Quantization is a speed technology, not just a fitting technology. Q4 vs FP16 isn't only 4× less memory — it's ~4× fewer bytes per token, hence ~4× faster decode. Topic 24's story just doubled in importance retroactively.
- Your Mac's speed was never about compute. The M4's modest tok/s comes straight from 120 GB/s of bandwidth — and the M4 Max's 546 GB/s buys ~4.5× the speed for the identical model. Mac shoppers comparing chips for local AI should read one spec line, and now you know which.
- The GPU is almost idle during single-user decode. Compute utilization runs 1–5% — those hundreds-of-ops-per-byte ratios from the Topic 30 exercise mean the cooks stand around while the pantry door cycles. Which begs the question: since the weights get read out anyway each step... couldn't that one read feed many users' next tokens simultaneously? Yes — that's batching, it's nearly free throughput, it's the economic foundation of every serving business, and it's Lesson 8's opening act.
Rounding out the toolkit, the remaining optimization families in one line each, so the landscape is mapped: kernel fusion (merge consecutive operations so intermediate results never visit HBM — Flash Attention's lesson applied everywhere; torch.compile automates much of it), CUDA graphs (pre-record the launch sequence to kill per-step CPU overhead), and speculative decoding (attack the one-token-per-step structure itself — next lesson's star).
Summary
Prefill is parallel and compute-bound (sets TTFT); decode reads all weights per token and is bandwidth-bound (sets tok/s ≈ bandwidth ÷ bytes). Quantization is therefore speed, Macs are bandwidth-limited, and idle decode compute is the free lunch batching will eat.
Mental model
Writing a story where, to choose each next word, you must re-skim the entire encyclopedia. Your thinking (compute) is instant; the librarian's cart speed (bandwidth) is your writing speed. A thinner encyclopedia (quantization) or a faster cart (better hardware) helps — or have the same skim serve twenty writers at once (batching).
Mistakes to avoid
- Buying or renting GPUs on FLOPS for a decode-heavy workload. You'd be hiring more cooks for a kitchen bottlenecked at the pantry door — bandwidth is the spec that predicts your tok/s.
- Blaming slow first tokens on decode. TTFT is prefill (compute-bound, scales with prompt size); streaming rate is decode (bandwidth-bound). Two different bottlenecks, two different fixes — diagnose before optimizing.
Exercise · the empirical payoff
Predict, then measure. Compute the ceiling for your own machine and a Q4 model you have (bandwidth ÷ file size). Then run ollama run <model> --verbose, give it a prompt, and read the reported eval rate (decode tok/s) and prompt eval rate (prefill). Compare measurement to prediction — landing within 2× on your first try is normal and feels like sorcery. Bonus: run a Q8 variant of the same model and confirm decode speed drops roughly in proportion to file size. You are now someone who predicts hardware performance from first principles.
Lesson 7 complete — and both Lesson 2 flags are down. The n² attention cost fell to IO-aware tiling; one-token-per-pass became tolerable via the KV cache and explicable via the bandwidth formula. You now hold the complete mental physics of inference: what fills memory, what moves through it, and why every number on your screen is what it is.
Lesson 8 finishes Module 4: speculative decoding (a small model drafts, the big model verifies — cheating the one-token-per-step law itself), batch inference and continuous batching (harvesting that idle 95% of compute), model serving (vLLM, PagedAttention, and what "production-grade" actually means), and latency-vs-quality tradeoffs (the decision framework that turns all this physics into product and pricing choices).