Model Serving — the restaurant around the chef

A model is a chef; a serving engine is the entire restaurant — host stand, order queue, table management, expo window. Running model.generate() in a Python loop is a chef cooking in an empty room. The serving layer is what the previous two topics require to exist: something has to schedule continuous batches, manage every diner's KV cache, stream results, and speak a standard API. Concepts today; hands-on tooling in Module 5.

The flagship idea: PagedAttention — vLLM's founding insight, and a beautiful case of stealing a 50-year-old OS trick. The problem: early engines pre-allocated each request's KV cache as one contiguous block sized for the maximum possible length — like reserving a 40-seat private room for every party because they might bring 40 people. Measured waste in pre-vLLM systems: 60–80% of KV memory held but unused. Since Topic 36 established that KV memory is the batch-size ceiling, most of the restaurant's capacity sat empty-but-reserved.

PagedAttention applies virtual memory: chop the cache into small fixed-size blocks ("pages," e.g. 16 tokens each), allocate them on demand as a sequence grows, let one request's pages scatter anywhere in VRAM, and keep a lookup table mapping logical position → physical page. Waste collapses to a few percent, effective batch size multiplies, throughput follows. Seat parties at whatever free tables exist, tracked by the host's chart — no more empty private rooms.

Paging enables a second, product-critical trick: prefix caching. If a thousand requests share the same 3,000-token system prompt (every agent product ever), compute its KV once and let all thousand requests point at the same physical pages — copy-on-write, exactly like OS memory sharing. Same for multi-turn chat: the conversation's cache survives between turns instead of being recomputed (there's Topic 5's "you re-pay for the whole whiteboard each turn" — softened, at the serving layer, and it's why providers now sell discounted "cached input tokens"). SGLang, vLLM's main rival, made radical prefix-sharing (RadixAttention) its whole founding thesis — for agent workloads with massively shared prompts, it's brutal.

The landscape, mapped once so Module 5 has pegs to hang on:

Engine	One-line identity	Reach for it when
vLLM	Open-source throughput standard; PagedAttention's home	Default for GPU serving
SGLang	Prefix-sharing specialist, very fast	Agent/structured workloads, shared prompts
TensorRT-LLM	NVIDIA's maximum-performance compiler; complex	Squeezing the last 20% at scale
llama.cpp / Ollama	CPU + Mac + consumer hardware; GGUF	Local, edge, your M4
TGI	Hugging Face's server	HF-ecosystem deployments

Everything above speaks the OpenAI-compatible API — the industry's lingua franca. Ship your app against that interface and the backend (local Ollama ↔ vLLM cluster ↔ commercial API) becomes a config change. This is the single highest-leverage architecture decision in AI product code.

The remaining ops vocabulary, one line each: streaming — tokens pushed as generated (SSE), non-negotiable for chat UX (formalized next topic); metrics — TTFT, inter-token latency, throughput, queue depth: the four dials of a serving dashboard; cold starts — loading tens of GB of weights takes tens of seconds to minutes, which makes naive request-based autoscaling miserable (hence keep-warm pools, and why "serverless GPU" is a hard product); multi-GPU — tensor parallelism splits every layer across GPUs (needs NVLink-class interconnect; how 70B+ models get served at all) while pipeline parallelism assigns different layers to different GPUs (tolerates slow links, adds latency); and multi-LoRA serving — Topic 22's wardrobe realized: engines hot-swap per-request adapters over one shared base, making per-customer fine-tunes economically absurd not to offer.

---