On-Device AI — phones, NPUs, and the OS as router

"On-device" sounds like "local, smaller" — but phones are a genuinely different regime, ruled by three constraints desktops ignore: battery (every joule is user-visible), thermals (no fans — sustained generation throttles within a minute or two, so design for bursts, not streams), and shared memory (a "12 GB phone" offers your model maybe 2–4 GB after the OS and apps take theirs).

Those constraints explain the hardware answer: the NPU (Neural Processing Unit — Apple's Neural Engine, Qualcomm's Hexagon, Google's Tensor cores). What it actually is, in Topic 30's vocabulary: a fixed-function matrix engine optimized for low-precision math per watt. A GPU is a flexible army of line cooks that drinks power; an NPU is a stamped assembly die that does one thing — INT8/INT4 matrix multiplies — at a fraction of the energy. Marketing quotes NPUs in TOPS (trillions of int-8 ops/sec), but you know better than to stop there: Topic 34 still governs — LLM decode is bandwidth-bound, phone memory runs ~50–100 GB/s, so the arithmetic says 1–4B quantized models at usable speeds, and no TOPS figure changes that. The NPU's real gift is doing it without cooking the battery.

The software layer is where on-device got interesting recently. Both platforms now ship an OS-level model: Apple's Foundation Models framework exposes a ~3B on-device model any app can call with a few lines of Swift (structured output, tool calling included — directly relevant to your SwiftUI work); Android's AICore serves Gemini Nano similarly. And Apple's architecture is a delight to decode with course eyes: one shared base model + swappable LoRA adapters per feature — summarization adapter, rewrite adapter, notification-priority adapter — Topic 22's wardrobe pattern, running in your pocket, chosen precisely because adapters are megabytes while models are gigabytes. For custom models beyond the OS ones: Core ML, ExecuTorch, ONNX Runtime, and LiteRT are the deployment runtimes; llama.cpp compiles for phones too.

The strategic pattern to internalize is the hybrid escalator — Topic 38's router, elevated to operating-system policy: try on-device first (private, free, instant); escalate hard queries to the cloud, ideally with privacy machinery in between (Apple's Private Cloud Compute being the reference design). On-device handles the easy 80%; the cloud handles the hard 20%; the user experiences one assistant. Every routing lesson this course taught, now baked into platforms.

---