Lesson 7Why Inference Is Slow (and the Machinery That Fixes It)
Welcome to Module 4: Inference & Optimization — where the last two flags from Lesson 2 finally come down. Flag one: generation runs the entire model once per token. Flag two: attention costs n². This module is the engineering that tamed both, and it splits naturally into two lessons: today, why inference is slow and the core mechanisms that fix it (GPUs, VRAM, KV cache, Flash Attention, and the single insight that organizes all of it); next lesson, serving at scale (speculative decoding, batching, serving frameworks, and the latency-quality economics).
Fair warning: this is the module where you'll start predicting your hardware's performance with arithmetic before measuring it. By the end of today you'll be able to estimate your Mac's tokens-per-second from first principles.
Topic 30: GPU Basics — an army of line cooks
Why GPUs at all? Because a transformer's work is overwhelmingly matrix multiplication — millions of tiny multiply-and-add operations that don't depend on each other, so they can all happen at the same time.
The kitchen analogy that carries this whole topic: A CPU is 8–16 master chefs — brilliant, versatile, each able to cook any dish, handle surprises, make decisions. A GPU is 16,000 line cooks who are individually simple but all chop in perfect unison. Ask them to debate a menu (branching logic) and they're hopeless. Ask them to chop a mountain of onions (multiply these 4096×4096 matrices) and they demolish work no team of chefs could touch. LLM inference is almost entirely onion-chopping.
The anatomy that matters (three parts):
- Compute units. NVIDIA calls the simple cores CUDA cores; the ones that matter most for LLMs are tensor cores — specialized stations that perform an entire small matrix-multiply as a single action, exactly the operation transformers eat. Compute is measured in FLOPS (floating-point operations per second); modern datacenter GPUs reach ~10¹⁵ — say it as "about a thousand TFLOPS."
- VRAM — the GPU's own onboard memory (next topic).
- Memory bandwidth — the speed at which data flows from VRAM into the compute units. This one, not FLOPS, will turn out to be the star of the module. The line cooks can chop far faster than the pantry can deliver onions — and the pantry's delivery rate is the bandwidth.
Real numbers to calibrate on (rounded, worth loosely memorizing):
| Hardware | Compute (BF16) | Memory | Bandwidth |
|---|---|---|---|
| H100 (datacenter) | ~1,000 TFLOPS | 80 GB | ~3.35 TB/s |
| RTX 4090 (consumer) | ~165 TFLOPS | 24 GB | ~1 TB/s |
| Apple M4 (base) | a few TFLOPS | up to 32 GB unified | ~120 GB/s |
| M4 Max | ~2× M4 Pro | up to 128 GB unified | ~546 GB/s |
Two observations hiding in that table. First, the ratio: an H100 can perform hundreds of arithmetic operations in the time it takes to fetch one byte from its own memory. Modern GPUs are starved chefs — keeping them fed is the whole game, a fact that will explain nearly everything in Topics 33–34. Second, Apple's different bet: unified memory means your M4's GPU can access a huge pool (great — big models fit), but at modest bandwidth and compute (they run slower). "Fits" and "fast" are different axes — hold that thought.
Last piece of the landscape: CUDA, NVIDIA's programming platform. Fifteen years of libraries, tooling, and every ML framework optimized for it first — this software moat, as much as the silicon, is why NVIDIA owns AI compute, and why AMD (ROCm) and Apple (Metal/MLX) remain the "also supported" column despite good hardware.
Summary
GPUs win at LLMs because transformers are parallel matrix math. Three specs matter — compute (FLOPS), memory size (VRAM), memory bandwidth — and modern GPUs have so much compute that bandwidth is usually the real constraint.
Mental model
A kitchen with 16,000 line cooks and one pantry door. Adding cooks stopped helping long ago; the pantry door's width (bandwidth) sets the pace of service.
Mistakes to avoid
- Comparing GPUs by FLOPS alone for LLM inference. For the workload you'll actually run, bandwidth predicts speed better — Topic 34 makes this quantitative.
- Assuming Mac unified memory equals GPU-class performance because the model loads. Loading is memory size; speed is memory bandwidth — your M4 has generous size and modest bandwidth.
Exercise
Look up the specs for your own machine plus any two GPUs (say, RTX 4090 and H100): memory size, bandwidth, compute. Compute each one's ratio of compute-to-bandwidth (FLOPS ÷ bytes/sec). Notice the ratio is in the hundreds everywhere — hundreds of operations possible per byte delivered. Write that number down; Topic 34 turns it into your speed calculator.
Topic 31: VRAM Basics — the budget everything lives inside
VRAM is the GPU's own memory, physically soldered next to the chip, connected by a firehose (that 1–3 TB/s bandwidth). System RAM is a separate pool connected through a straw: PCIe, at ~32–64 GB/s — roughly 30–50× slower than VRAM bandwidth. This gap is the iron law of GPU work:
Everything the model needs during inference must live in VRAM. Spilling to system RAM ("offloading") means feeding the GPU through the straw — the moment part of a model offloads, speed falls off a cliff. (Apple's unified memory dodges this by making it all one pool — the genuine architectural advantage behind the bandwidth compromise.)
So what actually fills the budget? Four items — and this equation is the sizing tool you'll use for every deployment decision from now on:
VRAM needed = model weights + KV cache + activations/overhead (+ headroom)
- Weights: the Lesson 2 math — params × bytes/param. 8B at Q4 ≈ 4.5 GB; at FP16 ≈ 16 GB.
- KV cache: the conversation's working memory — grows with context length and with each concurrent user. Full mechanism next topic; for now, know it can rival the weights themselves.
- Activations & framework overhead: the intermediate vectors flowing through layers plus the inference engine's bookkeeping — typically ~1–2 GB.
- Headroom: fragmentation is real; plan ~10% slack or meet the OOM crash at the worst moment.
Here's the budget drawn out for a concrete case — a 24 GB GPU running an 8B model at Q4 with a 32K context:
That green zone is the interesting part: it's not waste, it's capacity — room for more concurrent users (each needing their own KV cache) or longer contexts. Serving economics, next lesson, is largely the art of filling the green zone productively. And now Topic 29's rule of thumb — "GGUF file size ≈ RAM needed, plus a couple GB for context" — decompresses into its real form: file size is the blue segment; the "couple GB" was purple and orange all along.
Summary
VRAM is the fast local pool everything must fit inside; spilling over PCIe to system RAM is a 30–50× slowdown. Budget = weights + KV cache + ~2 GB overhead + headroom, and leftover space is serving capacity, not waste.
Mental model
The kitchen's pantry. Service runs only on what's inside; anything fetched from the warehouse across town (system RAM, through the PCIe straw) stalls the line. And pantry shelves you didn't fill with ingredients can hold more orders-in-progress.
Mistakes to avoid
- Budgeting only for weights — "8B at Q4 is 4.5 GB, my 8 GB card is fine!" — then hitting OOM the first time a conversation gets long. The purple segment grows; plan for it.
- Enabling CPU offload as a casual fix and accepting a silent 10–30× slowdown. Offloading is a last resort, not a setting.
Exercise
Redraw the budget bar (on paper) for your machine three times: (a) 8B at Q4 with 8K context, (b) 14B at Q4 with 8K, (c) 8B at Q4 with 128K. Use ~128 KB/token for the KV cache (justified next topic). Which configuration breaks your budget first — and is the culprit blue or purple? You've just learned that "can I run it?" has two answers depending on how long you talk to it.
Topic 32: KV Cache — the flag comes down
Time to pay off Lesson 2's flag: generation runs the whole model once per token. Here's the horror show that would happen without today's mechanism.
Recall the machinery: to generate token 6, every token attends to previous tokens using Keys and Values (Topic 7). Naively, that means step 6 re-runs the full transformer over all six tokens — recomputing K and V vectors for tokens 1–5 that were already computed in steps 1–5. Step 100 recomputes 99 tokens' worth. Step 1,000 recomputes 999. Total work across a generation grows quadratically, almost all of it redundant.
The saving observation is beautifully simple: causal masking (Topic 7) means old tokens never see new ones — so their K and V vectors never change. Token 3's Keys and Values are identical at step 4, step 400, and step 4,000. Anything that never changes can be computed once and cached:
So the KV cache is exactly that: for every layer and every head, store each processed token's K and V vectors. Each generation step then does the minimum possible work — compute Q, K, V for the one new token, attend its Q against all cached K/V, append its own K/V to the cache, done. Per-step compute stops growing with conversation length (mostly), and generation becomes viable at all. Every chatbot you've ever used runs on this.
But the trade is memory, and the meeting-minutes analogy makes it visceral: instead of re-interviewing everyone in the room each time a new person speaks (recompute), the room keeps minutes (cache) — each new speaker just reads them. Wonderful. Except the minutes notebook grows with every utterance and starts occupying real shelf space. How much? The formula, worth deriving once:
bytes per token = 2 (K and V) × layers × KV-heads × head-dim × bytes-per-value
For Llama-2-7B era models (32 layers, 32 heads, dim 128, FP16): 2×32×32×128×2 = ~0.5 MB per token. A 4K conversation: 2 GB. A 32K conversation: 16 GB — more than the quantized model itself. The purple segment devouring the pantry.
This pain is why every modern model ships with GQA (Grouped-Query Attention) — the one architecture tweak you must know: instead of every query head having its own K/V, query heads share K/V in groups. Llama-3-8B keeps 32 query heads but only 8 KV heads — the multi-head "specialists" from Topic 7 keep their distinct questions (Q) but share notebooks (K/V) four-to-a-desk, at negligible quality cost. Cache per token: 2×32×8×128×2 = 128 KB — a 4× cut. Now 32K costs 4 GB (yesterday's diagram), 128K costs 16 GB. Further tricks stack on top: quantizing the cache itself (8-bit KV is common and cheap — Topic 24 logic applied to the notebook), and sliding-window schemes that cap how far back some layers look.
One more distinction crystallizes here, and it will organize Topic 34: processing the prompt is different from generating. The prompt's tokens all exist upfront, so their K/V can be computed in parallel in one big pass — this is prefill, and it's what you wait for before the first token appears. Then decode takes over: the one-token-at-a-time cached loop above. Two phases, utterly different characters. Hold that.
Summary
The KV cache stores every token's K/V vectors so each generation step computes only the new token — the fix for quadratic recomputation and the mechanism that makes chat possible. Price: memory that grows per token per user (≈128 KB/token with GQA), which is why long context and many users are memory problems.
Mental model
Meeting minutes. Don't re-interview the whole room per new speaker — keep notes, have each newcomer read them. GQA: note-takers share notebooks in groups of four. The notebook's size, not the speaking, becomes the constraint on very long meetings.
Mistakes to avoid
- Believing "128K context window" means you can casually use 128K. The window is what the model supports; the KV cache is what you pay — 16 GB for a full window on an 8B model, per concurrent conversation.
- Forgetting the per-user multiplier. Ten users at 8K context is ten separate caches ≈ 10 GB — sizing for one user and serving ten is a classic launch-day OOM.
Exercise
Qwen2.5-7B's config: 28 layers, 4 KV heads, head-dim 128, FP16 cache. Compute (a) bytes per token, (b) cache size for one 16K conversation, (c) how many concurrent 8K-context users fit in the green zone of yesterday's 24 GB budget diagram. (Answers to check yourself: ~56 KB/token — notice how aggressive Qwen's GQA is vs Llama's; ~0.9 GB; ~30 users.) You just did real capacity planning.
Topic 33: Flash Attention — the n² flag comes down
Now the older flag, planted in Topic 7: attention compares every token with every other token — O(n²). For years this hard-capped context windows. The fix that broke the cap is the most instructive optimization story in the field, because what it optimizes is not what anyone expected.
The surprise: Flash Attention does not reduce the n² math. All the same comparisons still happen. What it eliminates is the n² memory traffic.
To see why that's the win, one more piece of GPU anatomy — the memory hierarchy inside the GPU. VRAM (the HBM chips) is the big pool: tens of GB at ~1.5–3 TB/s. But on the chip itself sits SRAM — a tiny scratchpad, ~20 MB total, at roughly 10× the bandwidth. Whiteboard and filing cabinet: the whiteboard is lightning-fast and fits almost nothing; the cabinet holds everything and requires a walk.
Now watch standard attention at 32K context. The score matrix — every token against every token — is 32,768² ≈ 1.1 billion entries, ~2 GB in FP16, for a single head in a single layer. Nowhere near fitting on the whiteboard, so the naive algorithm computes it in pieces, files the entire giant matrix into the cabinet (HBM), then walks back to retrieve it all for the softmax, files the result, retrieves it again to multiply by V… The GPU spends its time walking to the cabinet, cooks idle, while the actual arithmetic — which tensor cores could do almost instantly — waits on memory. Attention was never compute-bound. It was IO-bound, and nobody had organized the work around that fact.
Flash Attention (Tri Dao et al., 2022) reorganizes it. Two ideas:
- Tiling: chop Q, K, V into small blocks that fit on the whiteboard. Load a block of Q and a block of K/V into SRAM, compute their piece of attention entirely on-chip, accumulate into the running output, discard, next block. The full n×n matrix is never written to HBM — it never exists anywhere.
- Online softmax — the trick that makes tiling legal. Softmax seems to need all scores before converting any to percentages (you need the total to divide by). The online variant processes blocks sequentially while keeping a running maximum and running sum, retroactively rescaling earlier contributions whenever a new block shifts the statistics. Like computing a class average while grading papers one stack at a time — keep running totals, adjust as you go, and the final number is exactly what grading all at once would give.
That last word is the part people disbelieve: Flash Attention is exact. Not an approximation, not a quality trade — the mathematically identical output, computed in a smarter order. The results: attention 2–4× faster, and attention's memory footprint drops from O(n²) to O(n) — which, together with the KV cache tricks of Topic 32, is precisely what unlocked the 128K-context era. Today it's simply the default — baked into PyTorch, every serving engine, every training framework (your Lesson 6 capstone used it without asking). Successors (FlashAttention-2, -3) refine the same idea for newer chips.
And the general lesson outranks the specific technique: on modern hardware, arithmetic is nearly free; moving data is expensive. The biggest speedups of the past few years — Flash Attention, quantization's speed side, kernel fusion — are all fundamentally about touching memory less. Optimization stopped being about doing less math and became about fewer trips to the cabinet. Carry that lens into the next topic, where it becomes a formula.
Summary
Standard attention drowned in reads/writes of the n×n score matrix to slow HBM. Flash Attention tiles the computation through fast on-chip SRAM using online softmax, never materializing the matrix — exact results, 2–4× faster, O(n) memory, long context unlocked.
Mental model
A giant calculation done chunk-by-chunk on a whiteboard with running totals, instead of printing every intermediate page and filing it in a cabinet across the room only to fetch it right back. Same final answer; the walking was the cost.
Mistakes to avoid
- Describing Flash Attention as "approximate attention" or a quality tradeoff — a common misstatement that instantly signals secondhand knowledge. It's exact; say "IO-aware."
- Assuming it repealed the n² compute. Long context still costs quadratic arithmetic in attention — Flash made the memory side stop being the bottleneck, which is why prefill on huge prompts still takes real time.
Exercise
Compute the FP16 score-matrix size (n² × 2 bytes) for one head at n = 2K, 8K, 32K, 128K. (Check: 8 MB → 128 MB → 2 GB → 32 GB.) Then note your GPU's SRAM is ~20 MB total and write one sentence on why tiling was forced, not clever. Watching the numbers cross from "fits" to "impossible" between 2K and 8K is the whole history of context windows in one column of arithmetic.
Topic 34: Inference Optimization — the one insight that organizes everything
Synthesis time. Everything in this lesson — starved cooks, the pantry, the growing minutes, the cabinet walks — compresses into one organizing distinction and one formula you'll use for the rest of your career.
The two phases of every request (met in Topic 32, formalized now):
- Prefill: the prompt's tokens processed in parallel — huge matrix multiplies, tensor cores saturated. This phase is compute-bound: the chefs are the bottleneck, and it determines TTFT (time to first token) — the pause before the response starts, scaling with prompt length.
- Decode: one token at a time against the cache. And here's the fact that reorganizes your whole mental model: to generate each single token, the GPU must read every weight of the model out of VRAM once — all 4.5 GB of your Q4 8B, per token, every token. The arithmetic per token is trivial for the cooks; the delivery of the entire pantry contents each step is the wall. Decode is memory-bandwidth-bound, and it determines the tokens-per-second stream rate.
Which yields the formula — the single most useful equation in local AI:
decode tokens/sec ceiling ≈ memory bandwidth ÷ bytes read per token (weights + KV cache)
Worked examples, using Topic 30's table:
| Setup | Bytes/token | Bandwidth | Ceiling | Realistic |
|---|---|---|---|---|
| RTX 4090, 8B @ Q4 | ~4.5 GB | 1 TB/s | ~220 tok/s | ~120–160 |
| M4 base, 8B @ Q4 | ~4.5 GB | 120 GB/s | ~27 tok/s | ~15–20 |
| M4 base, 8B @ FP16 | ~16 GB | 120 GB/s | ~7 tok/s | ~4–6 |
| H100, 70B @ Q4 | ~40 GB | 3.35 TB/s | ~84 tok/s | ~50–65 |
(Realistic lands below ceiling due to overheads and growing KV-cache reads.) Sit with what this table quietly proves:
- Quantization is a speed technology, not just a fitting technology. Q4 vs FP16 isn't only 4× less memory — it's ~4× fewer bytes per token, hence ~4× faster decode. Topic 24's story just doubled in importance retroactively.
- Your Mac's speed was never about compute. The M4's modest tok/s comes straight from 120 GB/s of bandwidth — and the M4 Max's 546 GB/s buys ~4.5× the speed for the identical model. Mac shoppers comparing chips for local AI should read one spec line, and now you know which.
- The GPU is almost idle during single-user decode. Compute utilization runs 1–5% — those hundreds-of-ops-per-byte ratios from the Topic 30 exercise mean the cooks stand around while the pantry door cycles. Which begs the question: since the weights get read out anyway each step... couldn't that one read feed many users' next tokens simultaneously? Yes — that's batching, it's nearly free throughput, it's the economic foundation of every serving business, and it's Lesson 8's opening act.
Rounding out the toolkit, the remaining optimization families in one line each, so the landscape is mapped: kernel fusion (merge consecutive operations so intermediate results never visit HBM — Flash Attention's lesson applied everywhere; torch.compile automates much of it), CUDA graphs (pre-record the launch sequence to kill per-step CPU overhead), and speculative decoding (attack the one-token-per-step structure itself — next lesson's star).
Summary
Prefill is parallel and compute-bound (sets TTFT); decode reads all weights per token and is bandwidth-bound (sets tok/s ≈ bandwidth ÷ bytes). Quantization is therefore speed, Macs are bandwidth-limited, and idle decode compute is the free lunch batching will eat.
Mental model
Writing a story where, to choose each next word, you must re-skim the entire encyclopedia. Your thinking (compute) is instant; the librarian's cart speed (bandwidth) is your writing speed. A thinner encyclopedia (quantization) or a faster cart (better hardware) helps — or have the same skim serve twenty writers at once (batching).
Mistakes to avoid
- Buying or renting GPUs on FLOPS for a decode-heavy workload. You'd be hiring more cooks for a kitchen bottlenecked at the pantry door — bandwidth is the spec that predicts your tok/s.
- Blaming slow first tokens on decode. TTFT is prefill (compute-bound, scales with prompt size); streaming rate is decode (bandwidth-bound). Two different bottlenecks, two different fixes — diagnose before optimizing.
Exercise · the empirical payoff
Predict, then measure. Compute the ceiling for your own machine and a Q4 model you have (bandwidth ÷ file size). Then run ollama run <model> --verbose, give it a prompt, and read the reported eval rate (decode tok/s) and prompt eval rate (prefill). Compare measurement to prediction — landing within 2× on your first try is normal and feels like sorcery. Bonus: run a Q8 variant of the same model and confirm decode speed drops roughly in proportion to file size. You are now someone who predicts hardware performance from first principles.
Lesson 7 complete — and both Lesson 2 flags are down. The n² attention cost fell to IO-aware tiling; one-token-per-pass became tolerable via the KV cache and explicable via the bandwidth formula. You now hold the complete mental physics of inference: what fills memory, what moves through it, and why every number on your screen is what it is.
Lesson 8 finishes Module 4: speculative decoding (a small model drafts, the big model verifies — cheating the one-token-per-step law itself), batch inference and continuous batching (harvesting that idle 95% of compute), model serving (vLLM, PagedAttention, and what "production-grade" actually means), and latency-vs-quality tradeoffs (the decision framework that turns all this physics into product and pricing choices).