GPU Basics — an army of line cooks
Why GPUs at all? Because a transformer's work is overwhelmingly matrix multiplication — millions of tiny multiply-and-add operations that don't depend on each other, so they can all happen at the same time.
The kitchen analogy that carries this whole topic: A CPU is 8–16 master chefs — brilliant, versatile, each able to cook any dish, handle surprises, make decisions. A GPU is 16,000 line cooks who are individually simple but all chop in perfect unison. Ask them to debate a menu (branching logic) and they're hopeless. Ask them to chop a mountain of onions (multiply these 4096×4096 matrices) and they demolish work no team of chefs could touch. LLM inference is almost entirely onion-chopping.
The anatomy that matters (three parts):
- Compute units. NVIDIA calls the simple cores CUDA cores; the ones that matter most for LLMs are tensor cores — specialized stations that perform an entire small matrix-multiply as a single action, exactly the operation transformers eat. Compute is measured in FLOPS (floating-point operations per second); modern datacenter GPUs reach ~10¹⁵ — say it as "about a thousand TFLOPS."
- VRAM — the GPU's own onboard memory (next topic).
- Memory bandwidth — the speed at which data flows from VRAM into the compute units. This one, not FLOPS, will turn out to be the star of the module. The line cooks can chop far faster than the pantry can deliver onions — and the pantry's delivery rate is the bandwidth.
Real numbers to calibrate on (rounded, worth loosely memorizing):
| Hardware | Compute (BF16) | Memory | Bandwidth |
|---|---|---|---|
| H100 (datacenter) | ~1,000 TFLOPS | 80 GB | ~3.35 TB/s |
| RTX 4090 (consumer) | ~165 TFLOPS | 24 GB | ~1 TB/s |
| Apple M4 (base) | a few TFLOPS | up to 32 GB unified | ~120 GB/s |
| M4 Max | ~2× M4 Pro | up to 128 GB unified | ~546 GB/s |
Two observations hiding in that table. First, the ratio: an H100 can perform hundreds of arithmetic operations in the time it takes to fetch one byte from its own memory. Modern GPUs are starved chefs — keeping them fed is the whole game, a fact that will explain nearly everything in Topics 33–34. Second, Apple's different bet: unified memory means your M4's GPU can access a huge pool (great — big models fit), but at modest bandwidth and compute (they run slower). "Fits" and "fast" are different axes — hold that thought.
Last piece of the landscape: CUDA, NVIDIA's programming platform. Fifteen years of libraries, tooling, and every ML framework optimized for it first — this software moat, as much as the silicon, is why NVIDIA owns AI compute, and why AMD (ROCm) and Apple (Metal/MLX) remain the "also supported" column despite good hardware.
Summary
GPUs win at LLMs because transformers are parallel matrix math. Three specs matter — compute (FLOPS), memory size (VRAM), memory bandwidth — and modern GPUs have so much compute that bandwidth is usually the real constraint.
Mental model
A kitchen with 16,000 line cooks and one pantry door. Adding cooks stopped helping long ago; the pantry door's width (bandwidth) sets the pace of service.
Mistakes to avoid
- Comparing GPUs by FLOPS alone for LLM inference. For the workload you'll actually run, bandwidth predicts speed better — Topic 34 makes this quantitative.
- Assuming Mac unified memory equals GPU-class performance because the model loads. Loading is memory size; speed is memory bandwidth — your M4 has generous size and modest bandwidth.
Exercise
Look up the specs for your own machine plus any two GPUs (say, RTX 4090 and H100): memory size, bandwidth, compute. Compute each one's ratio of compute-to-bandwidth (FLOPS ÷ bytes/sec). Notice the ratio is in the hundreds everywhere — hundreds of operations possible per byte delivered. Write that number down; Topic 34 turns it into your speed calculator.