llama.cpp — the engine everything else wraps

Topic 39 of 90Module 5: Local AI Ecosystem2 min read

The origin story is genuinely important for understanding the ecosystem: in March 2023, Georgi Gerganov spent a weekend porting Meta's leaked-then-released LLaMA weights to plain C++ so it would run on his MacBook. No PyTorch, no Python, no CUDA — just a single compiled program. That weekend hack became the foundation of all consumer local AI.

What it is today: a dependency-free C/C++ inference engine that runs on essentially anything — CPUs (with hand-tuned SIMD kernels), Apple Metal, NVIDIA CUDA, AMD, Vulkan, phones, Raspberry Pis. It's the native home of everything from Topic 29: GGUF is its format, the K-quants are its inventions. Ollama, LM Studio, Jan, GPT4All — all of them are friendly bodies built around this engine block.

Its signature capability, which no GPU-first framework matches: partial offload. The -ngl N flag puts N layers on the GPU and runs the rest on CPU — so a model slightly too big for your VRAM still runs, just slower (you know exactly why from Topic 31: the CPU-resident layers eat the PCIe/bandwidth penalty). Graceful degradation instead of OOM.

The binaries worth knowing: llama-cli (direct chat), llama-server (an OpenAI-compatible endpoint — Topic 37's lingua franca, from a single executable), llama-quantize (you met it in Topic 29), and llama-bench (the honest way to measure your Topic 34 predictions).

Summary

The universal C++ inference engine underneath consumer local AI; GGUF's home; runs anywhere, degrades gracefully.

Mental model

The engine block an entire car industry builds brands around.

Mistakes to avoid

Reaching for it first as a beginner (Ollama wraps it better for daily use — go direct only when you need control: custom quants, benchmarking, edge targets). And forgetting -ngl exists, running fully on CPU while a perfectly good GPU idles.

Exercise

Clone and build it (cmake -B build && cmake --build build), then run llama-bench on a GGUF you already have. Compare its measured tok/s against your Topic 34 prediction and your Ollama measurement — three numbers, one theory.