Quantization — shrinking the bytes

LoRA attacked the trainable parameter side of the memory bill. Quantization attacks the other side: bytes per parameter — including those 14 GB of frozen base weights, and every model you'll ever run on your own hardware.

The idea: weights are stored as 16-bit floats — each one can express billions of distinct values with fine precision. Quantization asks: what if 256 distinct values (8-bit)? What if just 16 (4-bit)? Store each weight as a tiny code pointing at one of few allowed levels:

FP32: 4 bytes/param   → 7B model = 28 GB
FP16: 2 bytes/param   → 14 GB     ← standard release format
INT8: 1 byte/param    → 7 GB
4-bit: 0.5 byte/param → ~4 GB     ← runs on a laptop

Daily-life analogy: repainting a photograph with a crayon box. The original has millions of subtle colors (FP16); you re-create it using only 16 crayons (4-bit), coloring each pixel with the nearest crayon. From across the room, remarkably faithful. Up close, small errors everywhere — the quantization error.

The engineering is in making 16 crayons hurt as little as possible. Three ideas cover most of what matters:

1. Block-wise scaling. Don't use one crayon box for the whole painting — split weights into small blocks (e.g., 64 values) and give each block its own scale: find the block's biggest value, stretch the 16 levels to exactly cover that block's range. A per-region mini-palette. Every serious format does this; the per-block scales are small extra metadata riding along with the codes.

2. The outlier problem — the deep gotcha. A handful of weights (and, at runtime, activations) are enormous compared to their neighbors — and research (the LLM.int8 work) showed these outliers carry disproportionate importance; crush them and the model visibly degrades. Naive scaling faces an ugly choice: stretch the palette to cover the outlier (wasting nearly all 16 levels on empty range, crushing the normal values into 2–3 crayons) or clip it (destroying the important value). Modern methods exist largely to dodge this: keeping outliers in higher precision, or protecting the weights that matter most.

3. Distribution-aware levels. Weights aren't uniformly spread — they bunch in a bell curve around zero. So why space the 16 levels evenly? NF4 (NormalFloat-4) — the format QLoRA introduced — places levels where the weights actually live: densely packed near zero, sparse at the extremes. A crayon set with six shades of the common colors and one of the rare ones. Free accuracy from pure statistics.

Names you'll meet on every Hugging Face model page, now decodable: bitsandbytes/NF4 (on-the-fly quantization when loading, the QLoRA workhorse); GPTQ and AWQ (careful pre-quantization using calibration data — a sample of real text — to choose levels that minimize actual output error; AWQ specifically protects weights that matter most given typical activations); GGUF K-quants (the llama.cpp family — its own topic next lesson). All are post-training quantization (PTQ): compress after training, minutes of work. The alternative, quantization-aware training, bakes robustness in during training — better at extreme compression, rare in practice.

The two rules of thumb that drive real decisions:

• 8-bit ≈ free. Quality loss is barely measurable. 4-bit ≈ cheap — small, usually acceptable degradation with good formats. Below 3-bit the crayon box gets too small and quality falls off fast.
• Given fixed memory, a bigger model at 4-bit beats a smaller model at 16-bit. A 13B-at-4-bit (~7 GB) outperforms a 7B-at-FP16 (14 GB) on most tasks. Parameters (even coarse ones) buy more than precision does. This rule quietly decides what everyone runs locally.

One subtlety that will save you confusion later: 4-bit is a storage format. During computation, each block is dequantized on the fly back to 16-bit, the math happens in 16-bit, results flow on. The GPU is doing unzip-compute-discard, layer by layer — which costs a little compute to save a lot of memory. Remember this; QLoRA depends on it in about sixty seconds.

---