Lesson 5Training Without a Datacenter

Welcome to Module 3: Fine-Tuning Methods — the module where two flags I planted in Lesson 2 finally cash out. Recall them: training needs ~8× more memory per trainable parameter than inference, and quantization shrinks bytes per parameter. Let's put real numbers on the problem first, because every method in this lesson exists to attack these numbers.

Full fine-tuning a 7B model the standard way means, per parameter: 2 bytes for the weight, 2 for its gradient, plus ~12 for the optimizer's bookkeeping (Adam keeps full-precision master weights and two running averages — the "momentum" memory from Lesson 2). Total ≈ 16 bytes/param → 7B × 16 = ~112 GB, before activations. That's a multi-GPU cluster for a small model. This module is the story of how the community got that down to a free Colab notebook — and it's split across two lessons: today the parameter-efficient training stack (adapters → LoRA → quantization → QLoRA → checkpoints), next lesson the preference algorithms (RLHF, DPO) and GGUF, capped with an end-to-end training walkthrough.