Lesson 7Why Inference Is Slow (and the Machinery That Fixes It)
Welcome to Module 4: Inference & Optimization — where the last two flags from Lesson 2 finally come down. Flag one: generation runs the entire model once per token. Flag two: attention costs n². This module is the engineering that tamed both, and it splits naturally into two lessons: today, why inference is slow and the core mechanisms that fix it (GPUs, VRAM, KV cache, Flash Attention, and the single insight that organizes all of it); next lesson, serving at scale (speculative decoding, batching, serving frameworks, and the latency-quality economics).
Fair warning: this is the module where you'll start predicting your hardware's performance with arithmetic before measuring it. By the end of today you'll be able to estimate your Mac's tokens-per-second from first principles.
In this lesson
- 30GPU Basicsan army of line cooksWhy GPUs at all? Because a transformer's work is overwhelmingly matrix multiplication — millions of tiny multiply-and-add operations that…3 min
- 31VRAM Basicsthe budget everything lives insideVRAM is the GPU's own memory, physically soldered next to the chip, connected by a firehose (that 1–3 TB/s bandwidth). System RAM is a…3 min
- 32KV Cachethe flag comes downTime to pay off Lesson 2's flag: generation runs the whole model once per token. Here's the horror show that would happen without today's…4 min
- 33Flash Attentionthe n² flag comes downNow the older flag, planted in Topic 7: attention compares every token with every other token — O(n²). For years this hard-capped context…4 min
- 34Inference Optimizationthe one insight that organizes everythingSynthesis time. Everything in this lesson — starved cooks, the pantry, the growing minutes, the cabinet walks — compresses into one…5 min
Start with Topic 30: GPU Basics, or browse all 90 topics.