Edge AI Basics — the far end of the spectrum

Topic 72 of 90Module 9: Deployment4 min read

Past phones lies edge AI: inference on hardware at the data source — cameras, sensors, robots, kiosks, vehicles, factory controllers. The spectrum inside the spectrum: Jetson-class boxes (GPU-equipped, tens of watts) → Raspberry-Pi-class boards (CPU, a few watts) → microcontrollers (milliwatts, kilobytes — the TinyML world). Why anyone deploys here, when the cloud exists — five reasons, each decisive in its niche:

Latency (a robot's control loop or a vehicle's perception cannot wait for a network round-trip — physics, not preference), offline (farms, ships, factories with no reliable link), privacy/regulatory (footage that legally may not leave the premises), bandwidth economics (streaming 10,000 cameras to the cloud costs more than the cameras), and fleet economics (per-device inference is capex once; cloud inference is opex forever, multiplied by the fleet).

The constraints are Module 4's, turned to eleven: memory in MB-to-low-GB, power budgets measured in watts (sometimes solar), passive cooling, and hardware that must run untouched for years. The toolkit is your existing one, pushed to extremes, plus one new word: quantization to INT8/INT4 and below (Topic 24 at maximum pressure — edge NPUs often only speak low precision); distillation into the smallest viable student (Topic 17); compilation/operator-fusion runtimes (TensorRT, LiteRT, ONNX Runtime — Topic 34's fewer-trips-to-memory lesson, done by compilers); and the newcomer — pruning: deleting weights or entire neurons outright, leaving a sparse network. Honest note for your files: pruning shines in classic vision/audio models; for LLMs it has consistently underperformed quantization and remains a minor player — know the word, reach for the crayon box first.

The LLM-at-the-edge reality check, via your own formula: a Raspberry Pi 5 has ~17 GB/s of memory bandwidth; a 1B model at Q4 is ~0.6 GB; Topic 34 says ceiling ≈ 28 tok/s, reality under 10. So: 0.5–3B models run usably on Pi/Jetson-class hardware for narrow jobs — and microcontrollers are simply not LLM territory (their AI is keyword spotting, anomaly detection, gesture recognition: brilliant, tiny, non-generative). Which makes the dominant architecture a hierarchy, and it's Topic 38's router stretched across physical space: the edge does perception and filtering (the camera's tiny model detects "person at door"; the wake-word chip hears its name on milliwatts), and only the distilled, interesting fraction escalates to a bigger model on-device, on-prem, or in the cloud. Your smart speaker is the canonical stack: microwatt wake-word at the true edge, cloud LLM for the conversation — the whole spectrum of this lesson, collaborating in one product.

Summary

Edge = inference at the data source, chosen for latency, offline, privacy, bandwidth, and fleet economics, under brutal memory/power limits. Toolkit: extreme quantization, distillation, compiled runtimes, (pruning, minorly). LLMs reach Pi/Jetson-class at 0.5–3B; below that, classic TinyML — and real systems are hierarchies: edge perceives, cloud reasons.

Mental model

Border posts and headquarters. The posts (edge) watch everything with cheap, tireless, narrow attention and radio in only what matters; headquarters (cloud) does the deep thinking on the filtered signal. Nobody streams the whole border to HQ, and nobody puts a general in every watchtower.

Mistakes to avoid

  • "Let's run the LLM on the device" as a starting point. Start from the filter: what tiny model at the edge would make 99% of cloud calls unnecessary? That's usually the product.
  • Ignoring power and thermal budgets because the demo ran fine on a bench supply for ten minutes. Edge failures are duty-cycle failures — design for the watt budget, not the demo.

Exercise

Design the hierarchy for a real system — a shop's smart CCTV that answers "did anyone linger near the storeroom after closing?": specify what runs on-camera (model class, not LLM), what triggers escalation, what runs on the on-prem box vs cloud, and estimate the bandwidth saved vs streaming everything. Then compute, with Topic 34, whether a 3B-Q4 summarizer is viable on a Jetson with ~68 GB/s. One page — and you've architected across the entire spectrum this lesson drew.

Module 9: complete. Read the spectrum back one more time — cloud API, cloud GPU, desktop, pocket, edge — and notice you can now do the arithmetic at every station: what fits, how fast, what it costs, and what shell it needs around it. Deployment stopped being a mystery the day you learned params × bytes and bandwidth ÷ bytes; today just gave the answers addresses.

Next: Module 10 — Evaluation. The module the whole course has been foreshadowing every time I said "measure it, don't vibe it": AI benchmarks (what MMLU-class scores actually mean and how they rot — contamination, saturation, Goodhart), human evals (preference testing, A/B, the annotator-agreement problem you already met), LLM-as-judge done rigorously, building your own eval sets (the real skill), cost-per-token analysis, and speed/quality benchmarking as a discipline. It's the shortest path between "my model seems better" and "my model is better, here's the number" — and it's the skill that separates engineers from demo-builders.