Training vs Inference — building the brain vs using it
Two completely different modes of a model's existence. Confusing them causes more beginner misconceptions than anything else.
Training = adjusting the parameters. Inference = using frozen parameters to generate. Everything differs between them:
| Training | Inference | |
|---|---|---|
| Parameters | Being updated | Frozen solid |
| Direction | Forward pass + backward pass (compute blame, update knobs) | Forward only |
| Memory | Params + gradients + optimizer states ≈ 8× more than inference | Params + KV cache |
| Hardware | Thousands of GPUs, months | One GPU (or your Mac), milliseconds |
| Cost | Tens of millions of dollars, once | Fractions of a cent, per request |
| Who does it | Labs (and you, at small scale in Module 3) | Everyone, constantly |
The memory row deserves explanation because it shocks people: training a 7B model in standard fashion needs not 14 GB but ~80–120 GB. Why? For every parameter you also store its gradient (which direction to nudge it — the blame signal from backpropagation) and optimizer states (the Adam optimizer keeps two running averages per parameter to make updates smoother — think momentum, like remembering which direction you've been nudging so you don't zigzag). That's roughly 16 bytes per parameter for training vs 2 for inference. This 8× gap is exactly the problem LoRA and QLoRA solve in Module 3 — remember this number.
The modern training pipeline — a preview map of Module 2–3, because these stages are the vocabulary of the whole field:
- Pretraining: next-token prediction on trillions of tokens of raw internet/books/code. Produces a base model — a brilliant text-completer with zero manners. Ask it "What is Kubernetes?" and it might continue "…is a question often asked in interviews. Other common questions include…" — because that's a plausible continuation, and continuation is all it knows.
- Supervised Fine-Tuning (SFT): train on curated (instruction → good response) pairs. Teaches the format of being an assistant: when given a question, answer it.
- Preference tuning (RLHF/DPO): teach it which answers humans prefer — helpful over evasive, honest over confident nonsense.
Base model = raw talent. SFT = job training. Preference tuning = professional polish. When you download models, you'll see this in the names: Llama-3-8B (base) vs Llama-3-8B-Instruct (steps 2–3 applied). Grabbing the base by accident and wondering why it won't answer questions is a rite of passage — skip it.
One more critical point, repeated from Lesson 1 because it matters: your conversations are inference. The model learns nothing from them. Any "learning" during a chat is just information sitting in the context window (the whiteboard), gone when the conversation ends.
Summary
Training updates parameters (expensive, memory-hungry, done rarely); inference uses frozen parameters (cheap, fast, done constantly). Modern models pass through pretraining → SFT → preference tuning.
Mental model
Medical school vs seeing patients. School (training): years, enormous cost, the doctor's knowledge is being changed. Practice (inference): the knowledge is fixed; each patient is a fast, cheap application of it. Patients don't rewire the doctor's brain.
Mistakes to avoid
- "The model is learning from my corrections in this chat!" It isn't. Context window, not learning.
- Estimating fine-tuning memory from inference math. Training needs ~8× more per trainable parameter — the whole reason parameter-efficient methods exist.
Exercise
Find a base model and its instruct version (e.g., on Hugging Face: Qwen2.5-7B vs Qwen2.5-7B-Instruct — many have free hosted demos). Send both the same question. Watch the base model continue your text while the instruct model answers it. Ten minutes, and you'll never confuse the two again — this is the exact difference your own SFT runs in Module 3 will create.