Batch Inference — the free throughput

Now the biggest prize hiding in Topic 34's physics. Per decode step, the GPU reads all 4.5 GB of weights to produce one token for one user while compute idles. But the matrix math being fed by that read can just as easily compute the next token for 32 users at once — same weight-read, 32 tokens out. The marginal cost of users 2 through 32 is nearly zero until you finally become compute-bound.

This single fact is the economics of the entire LLM API industry. A GPU serving one stream produces ~100 tok/s; the same GPU batching well produces thousands — throughput improvements of an order of magnitude or more, from software alone. When you wonder how API tokens can cost fractions of a cent: batching is how. The delivery van drives the same route whether it carries one package or thirty; profitable couriers never drive for one package.

But naive static batching — collect 32 requests, process them together, return when all finish — has two ugly problems: sequences finish at different times (the user who needed 20 tokens waits, done, while the 2,000-token essay grinds on — their seat sits occupied-but-idle), and new arrivals wait for the entire batch to complete before starting. Latency suffers exactly when load is high.

The fix — arguably the serving innovation of the modern era — is continuous batching (also "in-flight batching," from the Orca paper, popularized by vLLM): make admission and exit decisions at every decode step, not every batch:

A city bus, not a tour bus: passengers board and alight at every stop; seats never ride empty. Consequences worth spelling out:

• Throughput vs latency becomes a dial, not a dilemma. Bigger running batch → more tokens/sec total, each user streaming slightly slower — until the batch finally saturates compute (the cooks stop idling) and per-user speed genuinely degrades. Serving is the art of riding just below that point.
• The batch size limit is usually memory, not compute — because every passenger carries luggage: their own KV cache (Topic 32's per-user multiplier). Thirty users at long contexts can fill the green zone of your VRAM diagram before compute saturates. This makes cache memory management the real throughput frontier — which is exactly the problem the next topic's PagedAttention exists to solve.
• One scheduling subtlety for your advanced file: a new arrival's prefill is a big compute burst, and naively wedging it between decode steps makes every current user's stream visibly hiccup. Modern engines use chunked prefill — slicing the newcomer's prompt processing into pieces interleaved with decode steps — smoothing the ride for everyone.

The second meaning of "batch inference": offline bulk jobs. Not concurrent users — one owner, a million items: classify every support ticket ever, generate 50K synthetic examples (your Lesson 3 pipeline at scale), run an eval suite, summarize an archive. Nothing is latency-sensitive, so you optimize purely for throughput and cost: run a serving engine offline at maximum batch, sort inputs by length so similar-length sequences travel together, and — if using APIs — use the providers' batch tiers (both Anthropic and OpenAI offer ~50% discounts for submit-now-collect-within-24h jobs; that's the vendor sharing the batching surplus with anyone who doesn't need answers now). For any pipeline that runs overnight anyway, not using the batch tier is a voluntary 2× cost donation.

---