QLoRA — the democratization event

Now snap the two pieces together. LoRA left one stubborn memory item: the frozen base still sits in GPU memory at FP16 — 14 GB for 7B before anything else. Quantization shrinks storage 4×. The obvious question: can you train LoRA adapters on top of a 4-bit base?

QLoRA (Dettmers et al., 2023) proved yes, with quality matching 16-bit LoRA. The recipe:

1. Load the base model quantized to 4-bit NF4 — frozen, as always. 7B → ~3.5 GB.
2. Attach LoRA adapters in full 16-bit — the trainable parts stay precise.
3. During training, each layer's weights are dequantized on the fly for computation (the unzip-compute-discard from Topic 24); gradients flow through the frozen 4-bit base into the adapters. The quantization error never compounds — the imprecise parts never move, and the moving parts are never imprecise.

4. Two supporting tricks from the paper: double quantization (the per-block scale factors are themselves quantized — squeezing the metadata) and paged optimizers (optimizer states can spill to CPU RAM during memory spikes instead of crashing — an overflow valve against the OOM errors that end training runs at hour six).

In code, the whole thing is a config object:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
 
bnb = BitsAndBytesConfig(load_in_4bit=True,
                         bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.bfloat16)
 
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", quantization_config=bnb)
# ...then apply the exact LoraConfig from Topic 23 on top

And here is the full arc of this lesson in one picture — where the 112 GB went:

The paper's headline demo fine-tuned a 65B model on a single 48 GB GPU — previously a many-GPU-cluster job. The downstream effect was cultural as much as technical: QLoRA is why Hugging Face hosts hundreds of thousands of community fine-tunes, why a student with a free Colab account can specialize a 7B model over a weekend, and why "fine-tune a small open model for the task" became a realistic answer for tiny teams. When you run your first training in the next lesson, it'll be QLoRA under the hood.

One deployment subtlety for the advanced file: merging (Topic 23) gets awkward with QLoRA. Your adapter was trained against the 4-bit base; cleanly merging means dequantize → merge → maybe requantize, and each step shifts things slightly. The practical guidance: either serve exactly what you trained (4-bit base + separate adapter), or merge into a 16-bit base — and whichever artifact you ship, evaluate that exact artifact, not its cousin. A theme you'll recognize: train the way you serve (Topic 15), evaluate what you deploy (Topic 24).