Parameters — the knobs, quantified

Time to make "billions of knobs" concrete, because parameter count drives everything you'll do practically: what fits on your Mac M4, what fine-tuning costs, what inference speed you get.

A parameter is one learned number. Every weight in every attention matrix, every FFN, every embedding vector — one float each. "Llama-3-8B" = 8 billion learned floats.

Where do they live in, say, a 7B model? Roughly:

• Embedding + unembedding matrices: ~0.5B (vocab size × hidden dimension)
• Attention (all Q/K/V/output matrices, all layers): ~2B
• FFNs: ~4.5B — the majority, as promised

The math you'll use constantly — memory footprint:

Each parameter stored in standard precision (FP16/BF16 — 16-bit floats) takes 2 bytes.

7B model  × 2 bytes = 14 GB   → doesn't fit consumer GPUs comfortably
7B model  × 1 byte  (8-bit)  = 7 GB    → fits a decent GPU
7B model  × 0.5 byte (4-bit) = ~4 GB   → runs on your MacBook
70B model × 2 bytes = 140 GB  → multiple datacenter GPUs

This one multiplication — params × bytes-per-param — is the first calculation of every local-AI decision you'll ever make. (Shrinking bytes-per-param is quantization, Module 3. Add ~1–2 GB overhead for the KV cache and activations on top.)

Does bigger = better? Yes, but with two huge caveats.

First, scaling laws: performance improves smoothly and predictably with more parameters + more data + more compute. This predictability is why labs confidently spend hundreds of millions on training runs — they can forecast the result before starting.

Second, the Chinchilla insight (DeepMind, 2022): for a fixed compute budget, most early models were too big and undertrained. Better to train a smaller model on more tokens. Rule of thumb from the paper: ~20 tokens of training data per parameter for compute-optimal training. But here's the modern twist — compute-optimal isn't inference-optimal. A model gets trained once but run billions of times, so today's labs deliberately "overtrain" small models far past 20:1 (Llama-3-8B saw ~15 trillion tokens — nearly 2,000 tokens per parameter). That's why a modern 8B model demolishes a 2020-era 175B model. Parameter count alone tells you almost nothing about quality anymore; training data volume and quality matter as much.

Also distinguish: parameters (learned, fixed after training) vs hyperparameters (choices made by humans before training: how many layers, learning rate, batch size — the settings of the training process itself, not the learned values). You'll set hyperparameters yourself when fine-tuning in Module 3.

---