Latency vs Quality Tradeoffs — the decision layer
Final topic of the module, and the one that turns physics into product judgment. Every deployment lives inside a three-way tension — quality, latency, cost — and "make it better" without naming which axis is not an engineering request. Your job is to know the levers and match them to what the product actually needs.
The levers, ranked by violence:
- Model size — the biggest lever by far: 70B → 8B is ~10× on cost and latency, with a quality drop whose size depends entirely on your task (tiny for narrow fine-tuned jobs — Module 3's whole thesis; large for open-ended reasoning).
- Quantization — Topic 34 made it a speed lever: Q4 ≈ 4× faster decode than FP16, small quality cost. Nearly always taken.
- Prompt dieting — every input token costs prefill time (TTFT) and money. Bloated system prompts, redundant RAG chunks, unpruned chat history: the cheapest optimization in the field is deleting tokens, and combined with prefix caching it compounds.
- Output capping — decode dominates cost; verbose answers are expensive answers. "Be concise" is a latency optimization (and Topic 16 taught you why models trend verbose without it).
- Speculative decoding, batching, caching — this lesson's machinery. Add semantic caching at the product layer: embed incoming queries (Lesson 2), serve near-duplicate questions from cached answers — FAQ-heavy products see enormous hit rates.
- Routing and cascades — the pro move: classify each query's difficulty, send the easy 80% to a small cheap model and the hard 20% to the frontier one (Topic 11's hybrid, now with mechanism); or cascade — try small first, escalate on low confidence. Done well: ~70–90% cost reduction at near-flat quality.
The perception insight most engineers miss: perceived latency ≠ actual latency. Humans read at ~250 words/min ≈ 5–6 tokens/sec. A model streaming 20 tok/s outruns any reader — if the stream starts fast. So for chat, TTFT is almost everything: 300 ms to first token + 20 tok/s feels instant; 4 s of silence followed by 200 tok/s feels broken, despite finishing sooner. Streaming isn't a nicety; it's the difference between the same backend feeling magical or unusable.
But flip the workload and the target flips: an agent (Module 7) chains model calls, and nobody reads the intermediate steps — total completion time is all that matters, so raw tok/s and per-step token counts dominate. And agents introduce the module's last piece of mathematics, the one that should genuinely scare you:
Chains compound. Ten sequential steps at 5 s each = 50 s of latency. Ten steps at 95% per-step reliability = 0.95¹⁰ ≈ 60% end-to-end success.
Both budgets — latency and quality — must be engineered per step, and this arithmetic is the strongest argument for fast, cheap, reliable small models inside agents, with the big model reserved for the steps that need it.
The matrix to internalize — what to optimize, by workload:
| Workload | North star | Levers that matter most |
|---|---|---|
| Chat UX | TTFT + streaming feel | prefix caching, prompt diet, ≥20 tok/s |
| Agents | total time × per-step reliability | small reliable models, token-lean steps, parallel steps where possible |
| Bulk/offline | cost per million tokens | max batch, batch APIs, aggressive quantization |
| Voice | hard TTFT budget (~300–500 ms) | small models, speculative decoding, edge/local |
| Code assist | quality first, then TTFT | bigger model justified; spec-decode loves code |
And the closing discipline that bridges into Module 10: every trade must be measured, not vibed. "We switched to Q4 / the 8B / the cheaper route" is incomplete until finished by "...and our eval score moved by X." The teams that win at this hold a private eval set (firewalled since Topic 18) and run it against every candidate configuration — quality per dollar per second, as numbers.
Summary
Quality–latency–cost is a triangle you position, not a problem you solve. Model size and quantization are the big levers; token dieting and caching are the free ones; routing/cascades are the pro ones. Optimize TTFT for humans, total-time-times-reliability for agents, cost for bulk — and measure every trade on your own evals.
Mental model
A courier service. Same-day bike messenger, standard post, overnight freight — "which courier is best?" is meaningless without the parcel. Product sense is matching each parcel to its tier; malpractice is shipping everything same-day (frontier model for every query) or everything freight (tiny model for legal advice).
Mistakes to avoid
- Optimizing total generation time when users perceive TTFT, or TTFT when the workload is an agent chain. Wrong north star → wasted engineering. Name the metric before touching anything.
- Defaulting every request to the best model "to be safe." That's not safety, it's an unexamined 10–30× cost multiplier — routing exists because most queries are easy.
Exercise · module capstone
Design the serving spec for a real product — take your own isDisposable support bot or any AI SaaS you'd build. Write one page: (1) workload type and north-star metric from the matrix; (2) chosen model + quantization with the Topic 34 formula predicting tok/s on your target hardware; (3) VRAM budget bar à la Topic 31 with expected concurrent users; (4) which three levers from this topic you'd pull first and the measurement that would validate each. That document is a real infra design doc — the kind that gets written before real money gets spent on GPUs.
Module 4: complete. The full inference stack now lives in your head as one connected system: bandwidth-bound decode → KV cache → Flash Attention → speculative drafts riding idle compute → continuous batches riding shared weight-reads → paged caches making the batches big → and a decision layer choosing where on the quality-latency-cost triangle each product lives.
Next: Module 5 — the Local AI Ecosystem. Every tool this course has name-dropped finally gets its hands-on treatment: llama.cpp and Ollama (where your GGUFs run), vLLM (Topic 37 made real), MLX (your M4's native framework — and why it exists), Hugging Face as the ecosystem's town square, and the fine-tuning toolchain — PEFT, TRL, Axolotl, and Unsloth (the "magic" from the Lesson 6 capstone, demystified). Less new theory, maximum practical mapping: which tool, when, and why.