GGUF — the format your model wears to leave home

Everything so far lives in the training world: Hugging Face repos, a folder of files — model.safetensors shards, config.json, tokenizer files, chat template. Perfect for training and Python. Now your fine-tuned model needs to go run on a laptop, in Ollama, on someone's Mac — the local-inference world. That world runs on GGUF.

GGUF is a single-file model format built for the llama.cpp ecosystem (the C++ inference engine behind Ollama, LM Studio, and most local AI — Module 5 gives it full treatment). The design philosophy is "everything in one portable box":

• The weights — almost always quantized (Topic 24 pays off again)
• The tokenizer, baked in
• The chat template, baked in (Topic 15's silent killer, largely defused by the format itself — the file carries its own formatting instructions)
• Metadata: architecture, context length, all hyperparameters needed to run

One file. No Python, no dependencies, no folder of parts. It's also memory-mappable — the OS can page weights straight from disk without a slow loading step, which is why local models start near-instantly. If the Hugging Face folder is a project's source code, GGUF is the compiled binary you ship.

Decoding the quant names — because your first visit to a GGUF repo shows twenty files named like Q4_K_M and this table is the decoder ring:

Reading the code: Q4 = ~4 bits per weight; K = the k-quant scheme (block-wise scaling with super-blocks — Topic 24's per-region palettes, two levels deep); S/M/L = how many of the most sensitive layers get kept at higher precision (Medium is the balanced pick). You'll also see IQ variants (importance-aware quants using an imatrix — calibration data identifying which weights matter most, Topic 24's AWQ idea in GGUF clothing) squeezing better quality from very low bit-counts.

Two practical rules: file size ≈ RAM needed (plus a couple GB for context — the KV cache, formally introduced in Module 4), and the Topic 24 law holds — prefer a bigger model at Q4 over a smaller one at Q8 for the same memory.

The full journey — this is the bridge from everything you've trained to everything you'll run:

# 1. Merge your LoRA adapter into the base (16-bit)   [Topic 23's merge]
#    model = PeftModel.from_pretrained(base, "out/final_adapter")
#    model = model.merge_and_unload(); model.save_pretrained("merged/")
 
# 2. Convert HF folder -> GGUF (llama.cpp's script)
python convert_hf_to_gguf.py merged/ --outfile mymodel-f16.gguf
 
# 3. Quantize                                          [Topic 24 applied]
./llama-quantize mymodel-f16.gguf mymodel-Q4_K_M.gguf Q4_K_M
 
# 4. Serve it locally via Ollama
echo 'FROM ./mymodel-Q4_K_M.gguf' > Modelfile
ollama create mybot -f Modelfile
ollama run mybot

Four commands between "my fine-tune finished" and "my model runs on my laptop, offline, forever." When people on Hugging Face say "are there GGUFs?" under every model release — and prolific quantizers upload twenty variants within hours — this pipeline is what they ran. Per the QLoRA merge subtlety from Topic 25 and the recurring law of this course: evaluate the Q4_K_M artifact you actually ship, because it's now two transformations removed from what you trained.

Capstone: The Whole Pipeline, End to End

Module 3's promised payoff. Here is the complete journey — every stage tagged with the lesson that taught it — as one runnable script. Model choice is deliberate: Qwen2.5-1.5B-Instruct trains via QLoRA inside a free Colab T4 or your Mac's memory, in minutes-to-an-hour on a small dataset.

# pip install transformers datasets peft trl bitsandbytes
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
 
MODEL = "Qwen/Qwen2.5-1.5B-Instruct"        # instruct base [Topic 14]
 
# ── 1. DATA: your cleaned synthetic JSONL  [Lessons 3-4]
#    each line: {"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}
data = load_dataset("json", data_files="train_clean.jsonl", split="train")
data = data.train_test_split(test_size=0.1, seed=42)   # firewalled eval split [T18, T26]
 
# ── 2. BASE MODEL in 4-bit  [QLoRA, Topic 25]
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(MODEL, quantization_config=bnb,
                                             device_map="auto")
tok = AutoTokenizer.from_pretrained(MODEL)
 
# ── 3. LoRA bypasses  [Topic 23]
peft_cfg = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                      lora_dropout=0.05, task_type="CAUSAL_LM")
 
# ── 4. TRAINING with checkpoint shopping  [Topic 26]
cfg = SFTConfig(output_dir="out",
    num_train_epochs=2,                      # 1-3, no more [Topic 14]
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,           # effective batch = 16 (see note)
    learning_rate=2e-4, warmup_ratio=0.03,   # LoRA-scale LR [T23], warmup [T20]
    eval_strategy="steps",  eval_steps=25,
    save_strategy="steps",  save_steps=25,  save_total_limit=3,
    load_best_model_at_end=True, metric_for_best_model="eval_loss",
    logging_steps=10, bf16=True)
 
trainer = SFTTrainer(model=model, args=cfg, peft_config=peft_cfg,
                     train_dataset=data["train"], eval_dataset=data["test"],
                     processing_class=tok)
# SFTTrainer applies the model's own chat template and loss masking [Topic 15]
trainer.train()
trainer.save_model("out/final_adapter")      # ~50 MB of adapter [T22]

Three annotations the script can't say for itself:

Gradient accumulation — the one genuinely new concept here: with batch size 2, gradients from 8 consecutive mini-batches are accumulated before one knob-update, simulating a batch of 16 without the memory of 16. The universal trick for training big-model behavior on small-GPU budgets.

What "watching training" means: ignore train loss going down (it always does — Topic 26); watch eval_loss at each 25-step checkpoint. Smoothly falling → healthy. U-turn → you're done; load_best_model_at_end rescues the minimum automatically.

The smoke test before anything else — generate with the exact serving format:

msgs = [{"role": "user", "content": "one question from your training domain"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=300)
print(tok.decode(out[0], skip_special_tokens=True))

Read the output like a Lesson 4 critic: right format? Stops properly (EOS learned — Topic 15)? Parroting training examples verbatim (overfit — Topic 26)? Then — and this closes the loop with Topic 21 — ask it something outside its training data and watch how it fails: gracefully, or with confident fabrication your dataset accidentally taught?

From here, the exits you already know: ship as adapter (multi-tenant wardrobe, Topic 22), run DPO on top with on-policy pairs (Topic 28), or merge → GGUF → Ollama (Topic 29's four commands) and your model runs on your own machine, offline, indefinitely. One honest preview: Module 5 introduces Unsloth, which wraps this identical pipeline with major speed/memory optimizations and one-line GGUF export — I taught you the raw TRL version first because Unsloth is only magic if you know what it's automating.

Module 3: complete. You now hold the full modern fine-tuning toolkit: PEFT/LoRA/QLoRA for training within mortal hardware, checkpoint discipline for picking winners, RLHF understood and DPO wielded for preference polish, GGUF for shipping. Combined with Module 2's data craft, you can take a raw idea to a specialized local model — genuinely end to end.

Next: Module 4 — Inference & Optimization. The two flags from Lesson 2 that are still flying — the n² attention cost and one-token-per-forward-pass generation — finally get their reckoning: KV cache (why chatbots don't recompute the whole conversation every token), Flash Attention, speculative decoding, batching, model serving, GPU/VRAM fundamentals, and the latency-vs-quality economics that decide what real products ship.