Lesson 8Serving at Scale — Free Speed, Free Throughput, Hard Choices

Welcome to Lesson 8 — the back half of Module 4, where the physics from Lesson 7 turns into throughput, products, and money. The setup you're carrying in: decode is memory-bound, the GPU's compute sits ~95% idle during single-stream generation, and every token costs one full read of the weights. Today's four topics are four different ways of refusing to waste that.

Topic 35: Speculative Decoding — cheating the one-token law

The one-token-per-step law looks unbeatable: token N+1 depends on token N, so generation is inherently sequential. You can't parallelize the future… can you?

Here's the loophole, and it comes from an asymmetry you already know. Verifying tokens is cheap; generating them is expensive. Remember prefill (Topic 34): given a batch of tokens that already exist, the model can process them all in parallel in one pass — and that pass yields, at every position, the model's opinion of what token should come next. Checking "would I have written these 5 tokens?" costs one forward pass. Writing those 5 tokens the normal way costs five.

So: get someone cheap to guess the future, and use the expensive model only to check the guesses.

The mechanism:

1. A small draft model (say 1B, sharing the big model's tokenizer) quickly generates K candidate tokens — say 5. Its weight-reads are cheap: 1B vs 70B.
2. The big target model runs one forward pass over all 5 candidates in parallel — prefill-style — producing its own verdict at each position.
3. Walk left to right: accept every drafted token the target agrees with. At the first disagreement, discard the rest of the draft and emit the target's own token there instead (which the verification pass already computed — free).
4. Resume drafting from the new position. Repeat.

Watch it run:

Four tokens emitted for one big-model weight read instead of four. And that yellow box is the part that sounds too good to be true but isn't: the accept/reject rule (with a small statistical correction called rejection sampling when you're using temperature) provably produces the exact same output distribution as the big model alone. The big model never signs anything it wouldn't have written. This is not a quality trade — it's pure speed, typically 2–3×.

Why does this beat the bandwidth ceiling? It doesn't — it routes around it using Topic 34's leftover resource. Decode is memory-bound: per step, the weights get read while compute idles at ~5%. The verification pass processes 5 tokens for essentially the same memory traffic as processing 1 — the extra work lands on the idle compute. Speculative decoding is literally the conversion of wasted FLOPs into tokens per second.

The speedup depends on one number: the acceptance rate — how often the junior's guesses survive review. Predictable text (code, boilerplate, structured output, formulaic prose) drafts beautifully: 70–80% acceptance, big wins. Surprising text (high-temperature creative writing) drafts poorly, and at low acceptance the overhead can even make things slower. Which sets up the variants worth knowing:

• Separate draft model — the classic; needs a small sibling with the same tokenizer (Topic 4: tokenizers don't mix).
• Self-speculation (Medusa, EAGLE) — no second model; bolt extra lightweight prediction heads onto the big model itself so it drafts its own future. EAGLE-family methods are the current state of the art in serving engines.
• Prompt-lookup / n-gram decoding — the free gem: draft by copying from the prompt. No model at all — when the output is likely to reuse input text (RAG answers quoting context, code editing, summarization), just propose the continuation from the matching text you already have. Costs nothing, wins big exactly where products like RAG chatbots live.

One honest asterisk for the advanced file: speculative decoding shines at low batch sizes — local single-user inference, latency-critical paths. On a serving GPU already saturated with a large batch (next topic), the "idle compute" is already spoken for, and the technique's advantage shrinks. Know which regime you're in.

---

Topic 36: Batch Inference — the free throughput

Now the biggest prize hiding in Topic 34's physics. Per decode step, the GPU reads all 4.5 GB of weights to produce one token for one user while compute idles. But the matrix math being fed by that read can just as easily compute the next token for 32 users at once — same weight-read, 32 tokens out. The marginal cost of users 2 through 32 is nearly zero until you finally become compute-bound.

This single fact is the economics of the entire LLM API industry. A GPU serving one stream produces ~100 tok/s; the same GPU batching well produces thousands — throughput improvements of an order of magnitude or more, from software alone. When you wonder how API tokens can cost fractions of a cent: batching is how. The delivery van drives the same route whether it carries one package or thirty; profitable couriers never drive for one package.

But naive static batching — collect 32 requests, process them together, return when all finish — has two ugly problems: sequences finish at different times (the user who needed 20 tokens waits, done, while the 2,000-token essay grinds on — their seat sits occupied-but-idle), and new arrivals wait for the entire batch to complete before starting. Latency suffers exactly when load is high.

The fix — arguably the serving innovation of the modern era — is continuous batching (also "in-flight batching," from the Orca paper, popularized by vLLM): make admission and exit decisions at every decode step, not every batch:

A city bus, not a tour bus: passengers board and alight at every stop; seats never ride empty. Consequences worth spelling out:

• Throughput vs latency becomes a dial, not a dilemma. Bigger running batch → more tokens/sec total, each user streaming slightly slower — until the batch finally saturates compute (the cooks stop idling) and per-user speed genuinely degrades. Serving is the art of riding just below that point.
• The batch size limit is usually memory, not compute — because every passenger carries luggage: their own KV cache (Topic 32's per-user multiplier). Thirty users at long contexts can fill the green zone of your VRAM diagram before compute saturates. This makes cache memory management the real throughput frontier — which is exactly the problem the next topic's PagedAttention exists to solve.
• One scheduling subtlety for your advanced file: a new arrival's prefill is a big compute burst, and naively wedging it between decode steps makes every current user's stream visibly hiccup. Modern engines use chunked prefill — slicing the newcomer's prompt processing into pieces interleaved with decode steps — smoothing the ride for everyone.

The second meaning of "batch inference": offline bulk jobs. Not concurrent users — one owner, a million items: classify every support ticket ever, generate 50K synthetic examples (your Lesson 3 pipeline at scale), run an eval suite, summarize an archive. Nothing is latency-sensitive, so you optimize purely for throughput and cost: run a serving engine offline at maximum batch, sort inputs by length so similar-length sequences travel together, and — if using APIs — use the providers' batch tiers (both Anthropic and OpenAI offer ~50% discounts for submit-now-collect-within-24h jobs; that's the vendor sharing the batching surplus with anyone who doesn't need answers now). For any pipeline that runs overnight anyway, not using the batch tier is a voluntary 2× cost donation.

---

Topic 37: Model Serving — the restaurant around the chef

A model is a chef; a serving engine is the entire restaurant — host stand, order queue, table management, expo window. Running model.generate() in a Python loop is a chef cooking in an empty room. The serving layer is what the previous two topics require to exist: something has to schedule continuous batches, manage every diner's KV cache, stream results, and speak a standard API. Concepts today; hands-on tooling in Module 5.

The flagship idea: PagedAttention — vLLM's founding insight, and a beautiful case of stealing a 50-year-old OS trick. The problem: early engines pre-allocated each request's KV cache as one contiguous block sized for the maximum possible length — like reserving a 40-seat private room for every party because they might bring 40 people. Measured waste in pre-vLLM systems: 60–80% of KV memory held but unused. Since Topic 36 established that KV memory is the batch-size ceiling, most of the restaurant's capacity sat empty-but-reserved.

PagedAttention applies virtual memory: chop the cache into small fixed-size blocks ("pages," e.g. 16 tokens each), allocate them on demand as a sequence grows, let one request's pages scatter anywhere in VRAM, and keep a lookup table mapping logical position → physical page. Waste collapses to a few percent, effective batch size multiplies, throughput follows. Seat parties at whatever free tables exist, tracked by the host's chart — no more empty private rooms.

Paging enables a second, product-critical trick: prefix caching. If a thousand requests share the same 3,000-token system prompt (every agent product ever), compute its KV once and let all thousand requests point at the same physical pages — copy-on-write, exactly like OS memory sharing. Same for multi-turn chat: the conversation's cache survives between turns instead of being recomputed (there's Topic 5's "you re-pay for the whole whiteboard each turn" — softened, at the serving layer, and it's why providers now sell discounted "cached input tokens"). SGLang, vLLM's main rival, made radical prefix-sharing (RadixAttention) its whole founding thesis — for agent workloads with massively shared prompts, it's brutal.

The landscape, mapped once so Module 5 has pegs to hang on:

Engine	One-line identity	Reach for it when
vLLM	Open-source throughput standard; PagedAttention's home	Default for GPU serving
SGLang	Prefix-sharing specialist, very fast	Agent/structured workloads, shared prompts
TensorRT-LLM	NVIDIA's maximum-performance compiler; complex	Squeezing the last 20% at scale
llama.cpp / Ollama	CPU + Mac + consumer hardware; GGUF	Local, edge, your M4
TGI	Hugging Face's server	HF-ecosystem deployments

Everything above speaks the OpenAI-compatible API — the industry's lingua franca. Ship your app against that interface and the backend (local Ollama ↔ vLLM cluster ↔ commercial API) becomes a config change. This is the single highest-leverage architecture decision in AI product code.

The remaining ops vocabulary, one line each: streaming — tokens pushed as generated (SSE), non-negotiable for chat UX (formalized next topic); metrics — TTFT, inter-token latency, throughput, queue depth: the four dials of a serving dashboard; cold starts — loading tens of GB of weights takes tens of seconds to minutes, which makes naive request-based autoscaling miserable (hence keep-warm pools, and why "serverless GPU" is a hard product); multi-GPU — tensor parallelism splits every layer across GPUs (needs NVLink-class interconnect; how 70B+ models get served at all) while pipeline parallelism assigns different layers to different GPUs (tolerates slow links, adds latency); and multi-LoRA serving — Topic 22's wardrobe realized: engines hot-swap per-request adapters over one shared base, making per-customer fine-tunes economically absurd not to offer.

---

Topic 38: Latency vs Quality Tradeoffs — the decision layer

Final topic of the module, and the one that turns physics into product judgment. Every deployment lives inside a three-way tension — quality, latency, cost — and "make it better" without naming which axis is not an engineering request. Your job is to know the levers and match them to what the product actually needs.

The levers, ranked by violence:

1. Model size — the biggest lever by far: 70B → 8B is ~10× on cost and latency, with a quality drop whose size depends entirely on your task (tiny for narrow fine-tuned jobs — Module 3's whole thesis; large for open-ended reasoning).
2. Quantization — Topic 34 made it a speed lever: Q4 ≈ 4× faster decode than FP16, small quality cost. Nearly always taken.
3. Prompt dieting — every input token costs prefill time (TTFT) and money. Bloated system prompts, redundant RAG chunks, unpruned chat history: the cheapest optimization in the field is deleting tokens, and combined with prefix caching it compounds.
4. Output capping — decode dominates cost; verbose answers are expensive answers. "Be concise" is a latency optimization (and Topic 16 taught you why models trend verbose without it).
5. Speculative decoding, batching, caching — this lesson's machinery. Add semantic caching at the product layer: embed incoming queries (Lesson 2), serve near-duplicate questions from cached answers — FAQ-heavy products see enormous hit rates.
6. Routing and cascades — the pro move: classify each query's difficulty, send the easy 80% to a small cheap model and the hard 20% to the frontier one (Topic 11's hybrid, now with mechanism); or cascade — try small first, escalate on low confidence. Done well: ~70–90% cost reduction at near-flat quality.

The perception insight most engineers miss: perceived latency ≠ actual latency. Humans read at ~250 words/min ≈ 5–6 tokens/sec. A model streaming 20 tok/s outruns any reader — if the stream starts fast. So for chat, TTFT is almost everything: 300 ms to first token + 20 tok/s feels instant; 4 s of silence followed by 200 tok/s feels broken, despite finishing sooner. Streaming isn't a nicety; it's the difference between the same backend feeling magical or unusable.

But flip the workload and the target flips: an agent (Module 7) chains model calls, and nobody reads the intermediate steps — total completion time is all that matters, so raw tok/s and per-step token counts dominate. And agents introduce the module's last piece of mathematics, the one that should genuinely scare you:

Both budgets — latency and quality — must be engineered per step, and this arithmetic is the strongest argument for fast, cheap, reliable small models inside agents, with the big model reserved for the steps that need it.

The matrix to internalize — what to optimize, by workload:

Workload	North star	Levers that matter most
Chat UX	TTFT + streaming feel	prefix caching, prompt diet, ≥20 tok/s
Agents	total time × per-step reliability	small reliable models, token-lean steps, parallel steps where possible
Bulk/offline	cost per million tokens	max batch, batch APIs, aggressive quantization
Voice	hard TTFT budget (~300–500 ms)	small models, speculative decoding, edge/local
Code assist	quality first, then TTFT	bigger model justified; spec-decode loves code

And the closing discipline that bridges into Module 10: every trade must be measured, not vibed. "We switched to Q4 / the 8B / the cheaper route" is incomplete until finished by "...and our eval score moved by X." The teams that win at this hold a private eval set (firewalled since Topic 18) and run it against every candidate configuration — quality per dollar per second, as numbers.

---

Module 4: complete. The full inference stack now lives in your head as one connected system: bandwidth-bound decode → KV cache → Flash Attention → speculative drafts riding idle compute → continuous batches riding shared weight-reads → paged caches making the batches big → and a decision layer choosing where on the quality-latency-cost triangle each product lives.

Next: Module 5 — the Local AI Ecosystem. Every tool this course has name-dropped finally gets its hands-on treatment: llama.cpp and Ollama (where your GGUFs run), vLLM (Topic 37 made real), MLX (your M4's native framework — and why it exists), Hugging Face as the ecosystem's town square, and the fine-tuning toolchain — PEFT, TRL, Axolotl, and Unsloth (the "magic" from the Lesson 6 capstone, demystified). Less new theory, maximum practical mapping: which tool, when, and why.