Local Inference — from running models to *shipping* them

You run models locally every week now — Ollama, MLX, GGUF, the Topic 34 formula. So this topic addresses the version of "local" that's actually new: local inference as a product decision — putting a model inside software that other people install. Three problems appear the moment your users aren't you:

1. Distribution. Models are gigabytes; app stores and installers are not built for that. Three patterns: download-on-first-run (the standard — ship a small app, fetch the model with resumable downloads and a real progress UX; users forgive a one-time 4 GB download far more than a 4 GB installer), bundle a tiny model (sub-1B models are small enough to ship inside the binary for instant-on basics), or llamafile — the charming extreme: model + llama.cpp runtime fused into one executable that runs on every OS, no install at all.

2. Hardware heterogeneity. Your machine is one machine; your users have 8 GB fanless laptops and 24 GB gaming rigs. Shipping local AI means shipping a hardware-detection tier table: probe RAM/VRAM at first run, then select model size and quant accordingly — your Topic 25/31 cheat-sheet math, executed automatically per user. (Detect ≤8 GB → 3B-Q4; 16 GB → 8B-Q4; 24 GB+ → 14B; below minimum → cloud fallback.) And accept the honest cost: every user's hardware is now your test matrix — the support burden is the real price of local, more than any engineering.

3. Architecture. The clean pattern, built from pieces you own: the app talks to a localhost OpenAI-compatible endpoint (Topics 37/40's lingua franca — an embedded llama.cpp server or Ollama), which means the same application code serves local and cloud backends — and the hybrid becomes one config decision: local when hardware allows, cloud when it doesn't, user's choice surfaced as a privacy toggle.

When local-as-product wins: privacy-sensitive verticals (health, legal, anything where "your data never leaves your machine" is the headline feature), offline requirements, latency (zero network), and the economics nobody advertises — zero marginal inference cost: every token your users generate on their own silicon is a token you don't pay for, which at scale is a business model, not a feature. When it loses: the quality ceiling on weak hardware, the support matrix, and model-update logistics (shipping a new 4 GB model to 100K users is a real operation).

---