Lesson 5Training Without a Datacenter

Welcome to Module 3: Fine-Tuning Methods — the module where two flags I planted in Lesson 2 finally cash out. Recall them: training needs ~8× more memory per trainable parameter than inference, and quantization shrinks bytes per parameter. Let's put real numbers on the problem first, because every method in this lesson exists to attack these numbers.

Full fine-tuning a 7B model the standard way means, per parameter: 2 bytes for the weight, 2 for its gradient, plus ~12 for the optimizer's bookkeeping (Adam keeps full-precision master weights and two running averages — the "momentum" memory from Lesson 2). Total ≈ 16 bytes/param → 7B × 16 = ~112 GB, before activations. That's a multi-GPU cluster for a small model. This module is the story of how the community got that down to a free Colab notebook — and it's split across two lessons: today the parameter-efficient training stack (adapters → LoRA → quantization → QLoRA → checkpoints), next lesson the preference algorithms (RLHF, DPO) and GGUF, capped with an end-to-end training walkthrough.

Topic 22: Adapter Tuning & PEFT — the "don't touch the walls" idea

The family name for everything in this lesson is PEFT: Parameter-Efficient Fine-Tuning. One shared insight powers the whole family:

Freeze 7B params (14 GB, sitting quietly) and train, say, 40M new ones (40M × 16 bytes ≈ 0.6 GB of training overhead). The 112 GB monster collapses to under 20 GB. The whole trick is choosing what small thing to train so that it can meaningfully steer a frozen giant.

Daily-life analogy: Renovating a rented apartment. You can't demolish walls (the base weights are frozen — you don't own them, and touching everything is what made it expensive). But you can add furniture, lamps, and removable fixtures — small additions that completely change how the space works, and can be swapped or removed without a trace.

The family members, in historical order:

• Classic adapters (2019): insert tiny bottleneck layers between the transformer's existing layers — little "squeeze down to 64 dims, process, expand back" modules. Worked, but the inserted layers sit in the inference path forever: permanent extra latency. Furniture you can never remove from the hallway.
• Prompt tuning / prefix tuning: don't add layers at all — learn a handful of virtual tokens (trainable embedding vectors, not real words) that get prepended to every input, steering the frozen model like a magic incantation. Elegant, ultra-cheap, but weaker steering power and it eats context window.
• LoRA (2021): the one that won, so decisively that "fine-tuning" in open-source practice now means LoRA by default. Why it won is the next topic — the short version: all the steering power of adapters, with zero inference latency after a merge step.
• Modern refinements you'll see in tooling: DoRA (splits weight updates into magnitude and direction components; often edges out LoRA at the same size), rsLoRA (a scaling fix for high ranks). Know the names; reach for them when squeezing the last few percent.

One product-level consequence worth planting now: because the base stays untouched and adapters are tiny files, you can keep one base model in memory and hot-swap many adapters — a support-tone adapter, a JSON-extraction adapter, a per-customer adapter — like outfits on one mannequin. Serving frameworks exploit this (multi-LoRA serving in vLLM and friends, Module 5), and it's the economic foundation of "a fine-tune per customer" products. Full fine-tuning could never afford that: each customer would need their own 14 GB copy.

Topic 23: LoRA — the low-rank trick, properly understood

LoRA (Low-Rank Adaptation) rests on one empirical discovery about fine-tuning itself:

When you fully fine-tune a model, each weight matrix W changes by some amount ΔW. The discovery: ΔW is almost always extremely redundant — the change, though spread across millions of numbers, has very few independent directions in it. In math terms, it has low rank.

Plain-language version of "rank": think of editing a photo. You could describe your edit as a list of changes to 12 million individual pixels. But if the edit was really "brightness +10, warmth +5, contrast +8," then three sliders fully describe those 12 million pixel changes. The rank is the number of independent sliders needed to describe the change. Fine-tuning changes, it turns out, are a few-slider kind of edit — makes sense, since Module 2 taught us fine-tuning mostly selects and amplifies existing behavior rather than installing new machinery.

The mechanism. Instead of learning the full-size ΔW (4096×4096 = 16.8M numbers per matrix), LoRA learns it as a product of two thin matrices: A (4096×r) and B (r×4096), where r — the rank — is tiny, like 16. The bypass output B·A·x gets added to the frozen path W·x:

Count the savings in the diagram: the frozen matrix holds 16.8M parameters; the bypass holds 4096×16 + 16×4096 = 131K — under 1%. Apply this to every attention and MLP matrix across all 32 layers and a 7B model ends up with ~40M trainable parameters instead of 7B. Same steering wheel, 0.5% of the cost.

Three design details that reveal the elegance:

1. B starts at zero. A is initialized randomly, B as all zeros — so at step one, B·A·x = 0 and the bypass contributes nothing. Training begins from exactly the untouched base model and gradually grows a deviation. No random-initialization shock to a carefully-trained giant.
2. The α/r scaling (alpha in the formula) is just a volume knob on the bypass — how loudly the adaptation speaks over the base. Convention: set alpha = 2×r and forget about it.
3. The merge. After training, compute B·A once and add it into W permanently. The bypass vanishes; you're left with a single ordinary matrix. Zero extra inference latency — the property that killed classic adapters. Or don't merge, keep the adapter as a separate ~50 MB file, and hot-swap per Topic 22's wardrobe pattern. You choose per deployment.

The knobs you'll actually set (your first real hyperparameters, as promised in Lesson 2):

• r (rank): 8–64; start at 16. Higher = more capacity to change the model = more risk of overfitting your small dataset. Doubling r rarely doubles quality.
• target_modules: which matrices get bypasses. The original paper only did attention's Q and V; the modern default is all linear layers (attention + MLP — remember from Lesson 2 where the knowledge lives), which reliably helps.
• learning rate: ~1e-4 to 2e-4 — roughly 10× higher than full fine-tuning uses, because you're steering a tiny rudder, not the whole ship.

In code — this is the real thing, not pseudocode:

from peft import LoraConfig, get_peft_model
 
config = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                    lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 40,370,176 || all params: 7,655,986,688 || trainable%: 0.53

That printout is the whole revolution in one line.

One advanced finding to carry with you (Biderman et al., 2024, neatly titled): "LoRA learns less and forgets less." With its low-rank bottleneck, LoRA has less capacity to absorb huge amounts of genuinely new material than full fine-tuning — but by the same token it damages the base model less, naturally resisting the catastrophic forgetting we battled throughout Module 2. For most fine-tuning jobs (form, style, task reliability — where the capability already exists in the base), this trade is a straight win: you didn't need the extra capacity, and you got the protection free. For continued-pretraining-scale knowledge injection (Topic 20), it's a real limitation — another reason that job uses full training.

Topic 24: Quantization — shrinking the bytes

LoRA attacked the trainable parameter side of the memory bill. Quantization attacks the other side: bytes per parameter — including those 14 GB of frozen base weights, and every model you'll ever run on your own hardware.

The idea: weights are stored as 16-bit floats — each one can express billions of distinct values with fine precision. Quantization asks: what if 256 distinct values (8-bit)? What if just 16 (4-bit)? Store each weight as a tiny code pointing at one of few allowed levels:

FP32: 4 bytes/param   → 7B model = 28 GB
FP16: 2 bytes/param   → 14 GB     ← standard release format
INT8: 1 byte/param    → 7 GB
4-bit: 0.5 byte/param → ~4 GB     ← runs on a laptop

Daily-life analogy: repainting a photograph with a crayon box. The original has millions of subtle colors (FP16); you re-create it using only 16 crayons (4-bit), coloring each pixel with the nearest crayon. From across the room, remarkably faithful. Up close, small errors everywhere — the quantization error.

The engineering is in making 16 crayons hurt as little as possible. Three ideas cover most of what matters:

1. Block-wise scaling. Don't use one crayon box for the whole painting — split weights into small blocks (e.g., 64 values) and give each block its own scale: find the block's biggest value, stretch the 16 levels to exactly cover that block's range. A per-region mini-palette. Every serious format does this; the per-block scales are small extra metadata riding along with the codes.

2. The outlier problem — the deep gotcha. A handful of weights (and, at runtime, activations) are enormous compared to their neighbors — and research (the LLM.int8 work) showed these outliers carry disproportionate importance; crush them and the model visibly degrades. Naive scaling faces an ugly choice: stretch the palette to cover the outlier (wasting nearly all 16 levels on empty range, crushing the normal values into 2–3 crayons) or clip it (destroying the important value). Modern methods exist largely to dodge this: keeping outliers in higher precision, or protecting the weights that matter most.

3. Distribution-aware levels. Weights aren't uniformly spread — they bunch in a bell curve around zero. So why space the 16 levels evenly? NF4 (NormalFloat-4) — the format QLoRA introduced — places levels where the weights actually live: densely packed near zero, sparse at the extremes. A crayon set with six shades of the common colors and one of the rare ones. Free accuracy from pure statistics.

Names you'll meet on every Hugging Face model page, now decodable: bitsandbytes/NF4 (on-the-fly quantization when loading, the QLoRA workhorse); GPTQ and AWQ (careful pre-quantization using calibration data — a sample of real text — to choose levels that minimize actual output error; AWQ specifically protects weights that matter most given typical activations); GGUF K-quants (the llama.cpp family — its own topic next lesson). All are post-training quantization (PTQ): compress after training, minutes of work. The alternative, quantization-aware training, bakes robustness in during training — better at extreme compression, rare in practice.

The two rules of thumb that drive real decisions:

• 8-bit ≈ free. Quality loss is barely measurable. 4-bit ≈ cheap — small, usually acceptable degradation with good formats. Below 3-bit the crayon box gets too small and quality falls off fast.
• Given fixed memory, a bigger model at 4-bit beats a smaller model at 16-bit. A 13B-at-4-bit (~7 GB) outperforms a 7B-at-FP16 (14 GB) on most tasks. Parameters (even coarse ones) buy more than precision does. This rule quietly decides what everyone runs locally.

One subtlety that will save you confusion later: 4-bit is a storage format. During computation, each block is dequantized on the fly back to 16-bit, the math happens in 16-bit, results flow on. The GPU is doing unzip-compute-discard, layer by layer — which costs a little compute to save a lot of memory. Remember this; QLoRA depends on it in about sixty seconds.

Topic 25: QLoRA — the democratization event

Now snap the two pieces together. LoRA left one stubborn memory item: the frozen base still sits in GPU memory at FP16 — 14 GB for 7B before anything else. Quantization shrinks storage 4×. The obvious question: can you train LoRA adapters on top of a 4-bit base?

QLoRA (Dettmers et al., 2023) proved yes, with quality matching 16-bit LoRA. The recipe:

1. Load the base model quantized to 4-bit NF4 — frozen, as always. 7B → ~3.5 GB.
2. Attach LoRA adapters in full 16-bit — the trainable parts stay precise.
3. During training, each layer's weights are dequantized on the fly for computation (the unzip-compute-discard from Topic 24); gradients flow through the frozen 4-bit base into the adapters. The quantization error never compounds — the imprecise parts never move, and the moving parts are never imprecise.

4. Two supporting tricks from the paper: double quantization (the per-block scale factors are themselves quantized — squeezing the metadata) and paged optimizers (optimizer states can spill to CPU RAM during memory spikes instead of crashing — an overflow valve against the OOM errors that end training runs at hour six).

In code, the whole thing is a config object:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
 
bnb = BitsAndBytesConfig(load_in_4bit=True,
                         bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.bfloat16)
 
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", quantization_config=bnb)
# ...then apply the exact LoraConfig from Topic 23 on top

And here is the full arc of this lesson in one picture — where the 112 GB went:

The paper's headline demo fine-tuned a 65B model on a single 48 GB GPU — previously a many-GPU-cluster job. The downstream effect was cultural as much as technical: QLoRA is why Hugging Face hosts hundreds of thousands of community fine-tunes, why a student with a free Colab account can specialize a 7B model over a weekend, and why "fine-tune a small open model for the task" became a realistic answer for tiny teams. When you run your first training in the next lesson, it'll be QLoRA under the hood.

One deployment subtlety for the advanced file: merging (Topic 23) gets awkward with QLoRA. Your adapter was trained against the 4-bit base; cleanly merging means dequantize → merge → maybe requantize, and each step shifts things slightly. The practical guidance: either serve exactly what you trained (4-bit base + separate adapter), or merge into a 16-bit base — and whichever artifact you ship, evaluate that exact artifact, not its cousin. A theme you'll recognize: train the way you serve (Topic 15), evaluate what you deploy (Topic 24).

Topic 26: Model Checkpoints — save points and how to choose one

Last topic — humbler than the others, but it's the difference between training runs that produce a good model and runs that produce regret.

A checkpoint is a complete snapshot of training at some step: the model weights, plus — if you want to resume rather than just use — the optimizer states, learning-rate schedule position, and random-number state. Save it to disk; training can be reconstructed from that exact moment.

(One vocabulary note: the word also means "released model snapshot" — when Hugging Face pages say "load the checkpoint," they mean the published weights. Same concept, different context.)

Why checkpoints matter — reason 1, the boring one: crashes. Long runs die — out-of-memory at hour six, driver hiccups, and above all spot instances: cloud GPUs rented at ~⅓ price with the catch that the provider can yank them anytime. Checkpointing every N steps converts "yanked at hour five" from a disaster into a two-minute resume — which means checkpoint discipline is literally a 3× cost reduction on training compute, because it's what makes cheap interruptible GPUs usable. Unsexy skill, real money.

Why checkpoints matter — reason 2, the deep one: the best model is usually not the last one. Training does not monotonically improve the model. Watch two curves during any run:

• Training loss: wrongness on the data being trained on. Falls, keeps falling — the model can always fit the training set harder, eventually by memorizing it.
• Validation loss (eval loss): wrongness on a held-out split the model never trains on. Falls… then flattens… then turns around and rises. That U-turn is overfitting happening live — Topic 14's parroting, now visible as a number: the model is trading general skill for memorization of your training examples.

The best model lives at the bottom of the validation curve — often well before the final step. So the professional workflow is checkpoint shopping: save every N steps, evaluate each on the validation split, deploy the one with the lowest validation loss (or best task metric), and treat everything after the U-turn as compost. In Hugging Face's trainer this is three arguments doing the whole job:

TrainingArguments(
    eval_strategy="steps", eval_steps=50,
    save_strategy="steps", save_steps=50,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    save_total_limit=3,          # keep best few, auto-delete the rest
)

Note the quiet dependency on Module 2: this only works if your validation split is trustworthy — held out before training, deduplicated against the training set (Topic 18's near-dup checks apply between splits too — a near-duplicate leaking across gives that example a memorized, fraudulent score), and firewalled forever.

And one more LoRA dividend: a full-fine-tune checkpoint with optimizer states for 7B is on the order of ~100 GB — save ten and you've filled a terabyte. A LoRA checkpoint is adapters only: tens of megabytes. Checkpoint every 50 steps, keep them all, email one to a friend. Every practice in this topic gets 1000× cheaper under the methods of this lesson — the module's ideas compounding with each other.

Lesson 5 complete. The arc, compressed: full fine-tuning costs 16 bytes per trainable parameter → freeze everything and train low-rank bypasses (LoRA, <1% trainable, mergeable to zero latency) → quantize the frozen part to 4-bit (QLoRA) → 112 GB becomes 6 GB → checkpoint the run and shop for the validation-loss minimum. You now hold the complete mechanics of modern accessible fine-tuning.

Lesson 6 finishes Module 3: RLHF (reward models and why the classic pipeline is heavy), DPO (the elegant shortcut that consumes your Topic 16 preference pairs directly — and why it took over), GGUF (the file format your quantized models wear when they leave home for llama.cpp and Ollama), and then the payoff: an end-to-end fine-tune walkthrough — your Lesson 3 synthetic dataset, QLoRA, real code, checkpoint selection, the whole pipeline you've been assembling for four lessons finally run as one.