API Serving — the production shell

You know the engine (vLLM, Topic 37/41) and the economics (Topic 36). What Module 4 didn't cover is the shell — everything between the user's request and any model, which every production service needs regardless of whose model answers. First, the three deployment shapes, so the vocabulary is fixed:

1. Managed API — someone else's models, zero ops (Topic 11's rent-the-restaurant).
2. Managed hosting / serverless GPU — your model (your Module 3 fine-tune) deployed on Modal/Replicate/HF-endpoints-style platforms: they run the GPUs, you get an endpoint, and scale-to-zero means idle costs nothing — with Topic 37's cold-start tax (tens of seconds loading weights) as the price of zero.
3. Dedicated self-host — rented GPUs + vLLM, full control, batch economics fully yours.

Whichever shape serves the tokens, production traffic flows through the same shell:

Walking the shell, each piece earning its box:

• Gateway — and the rule it exists to enforce: API keys never touch the client. A browser or mobile app calling a model provider directly means your key ships inside it, gets extracted within days, and funds a stranger's startup. All traffic goes through your backend, which holds the keys, authenticates users, and enforces rate limits and per-user/per-feature budgets — because in LLM products, an abuse loop isn't spam, it's a five-figure invoice. Cost controls (max_tokens caps everywhere, spend alarms) are gateway logic, not afterthoughts.
• Router — Topic 38's economics, deployed: classify difficulty, send the easy 80% to your fine-tuned 8B (marginal cost ≈ hardware ÷ Topic 36 batching), escalate the rest.
• Fallbacks — the ops truth nobody's architecture diagram admits until the first outage: providers go down and rate-limit you. Multi-provider failover with retries — same OpenAI-dialect interface everywhere (Topic 37's lingua franca making the swap trivial) — is the difference between "degraded" and "down." Tooling like LiteLLM packages gateway+router+fallback as one proxy; the concepts are exactly this diagram.
• Semantic cache — Topic 38's trick, installed at the shell: embed incoming queries, serve near-duplicates from cached answers before any model is consulted.
• Streaming passthrough — tokens must flow through your backend to the user (SSE end-to-end), because buffering the full response before forwarding destroys the TTFT experience Topic 38 declared sacred.
• Observability — log every hop: tokens in/out, latency, model chosen, cost, and for agent chains a trace linking the steps (Topic 60's debugging lesson, productionized). An LLM service without per-request cost metering is a service that discovers its unit economics from the monthly invoice.

---