Lesson 8Serving at Scale — Free Speed, Free Throughput, Hard Choices
Welcome to Lesson 8 — the back half of Module 4, where the physics from Lesson 7 turns into throughput, products, and money. The setup you're carrying in: decode is memory-bound, the GPU's compute sits ~95% idle during single-stream generation, and every token costs one full read of the weights. Today's four topics are four different ways of refusing to waste that.
In this lesson
- 35Speculative Decodingcheating the one-token lawThe one-token-per-step law looks unbeatable: token N+1 depends on token N, so generation is inherently sequential. You can't parallelize…4 min
- 36Batch Inferencethe free throughputNow the biggest prize hiding in Topic 34's physics. Per decode step, the GPU reads all 4.5 GB of weights to produce one token for one user…4 min
- 37Model Servingthe restaurant around the chefA model is a chef; a serving engine is the entire restaurant — host stand, order queue, table management, expo window. Running…4 min
- 38Latency vs Quality Tradeoffsthe decision layerFinal topic of the module, and the one that turns physics into product judgment. Every deployment lives inside a three-way tension —…5 min
Start with Topic 35: Speculative Decoding, or browse all 90 topics.