Model Checkpoints — save points and how to choose one

Last topic — humbler than the others, but it's the difference between training runs that produce a good model and runs that produce regret.

A checkpoint is a complete snapshot of training at some step: the model weights, plus — if you want to resume rather than just use — the optimizer states, learning-rate schedule position, and random-number state. Save it to disk; training can be reconstructed from that exact moment.

(One vocabulary note: the word also means "released model snapshot" — when Hugging Face pages say "load the checkpoint," they mean the published weights. Same concept, different context.)

Why checkpoints matter — reason 1, the boring one: crashes. Long runs die — out-of-memory at hour six, driver hiccups, and above all spot instances: cloud GPUs rented at ~⅓ price with the catch that the provider can yank them anytime. Checkpointing every N steps converts "yanked at hour five" from a disaster into a two-minute resume — which means checkpoint discipline is literally a 3× cost reduction on training compute, because it's what makes cheap interruptible GPUs usable. Unsexy skill, real money.

Why checkpoints matter — reason 2, the deep one: the best model is usually not the last one. Training does not monotonically improve the model. Watch two curves during any run:

• Training loss: wrongness on the data being trained on. Falls, keeps falling — the model can always fit the training set harder, eventually by memorizing it.
• Validation loss (eval loss): wrongness on a held-out split the model never trains on. Falls… then flattens… then turns around and rises. That U-turn is overfitting happening live — Topic 14's parroting, now visible as a number: the model is trading general skill for memorization of your training examples.

The best model lives at the bottom of the validation curve — often well before the final step. So the professional workflow is checkpoint shopping: save every N steps, evaluate each on the validation split, deploy the one with the lowest validation loss (or best task metric), and treat everything after the U-turn as compost. In Hugging Face's trainer this is three arguments doing the whole job:

TrainingArguments(
    eval_strategy="steps", eval_steps=50,
    save_strategy="steps", save_steps=50,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    save_total_limit=3,          # keep best few, auto-delete the rest
)

Note the quiet dependency on Module 2: this only works if your validation split is trustworthy — held out before training, deduplicated against the training set (Topic 18's near-dup checks apply between splits too — a near-duplicate leaking across gives that example a memorized, fraudulent score), and firewalled forever.

And one more LoRA dividend: a full-fine-tune checkpoint with optimizer states for 7B is on the order of ~100 GB — save ten and you've filled a terabyte. A LoRA checkpoint is adapters only: tens of megabytes. Checkpoint every 50 steps, keep them all, email one to a friend. Every practice in this topic gets 1000× cheaper under the methods of this lesson — the module's ideas compounding with each other.

Lesson 5 complete. The arc, compressed: full fine-tuning costs 16 bytes per trainable parameter → freeze everything and train low-rank bypasses (LoRA, <1% trainable, mergeable to zero latency) → quantize the frozen part to 4-bit (QLoRA) → 112 GB becomes 6 GB → checkpoint the run and shop for the validation-loss minimum. You now hold the complete mechanics of modern accessible fine-tuning.

Lesson 6 finishes Module 3: RLHF (reward models and why the classic pipeline is heavy), DPO (the elegant shortcut that consumes your Topic 16 preference pairs directly — and why it took over), GGUF (the file format your quantized models wear when they leave home for llama.cpp and Ollama), and then the payoff: an end-to-end fine-tune walkthrough — your Lesson 3 synthetic dataset, QLoRA, real code, checkpoint selection, the whole pipeline you've been assembling for four lessons finally run as one.