Lesson 13The Model Zoo

Welcome to Module 8: Model Types — a fast, high-density tour of the zoo. Six species: VLMs (how images become tokens), SLMs (why small became a strategy rather than a compromise), dense and MoE architectures taught together, coding models, and reasoning models. Watch for the pattern: every exotic species here decomposes into fundamentals you already own.

Topic 62: VLMs — how images become tokens

A transformer processes a sequence of vectors. It does not care what those vectors originally were. That single fact is the whole of vision-language modelling: get an image into vector form the model already understands, and every mechanism from Lesson 2 works unchanged.

The three parts, decoded:

• Patches — "visual words." A Vision Transformer (ViT) splits the image into a grid of small tiles (commonly 14×14 or 16×16 pixels) and treats each as one unit — the visual analog of Topic 3's tokens. LEGO bricks again, cut from pixels.
• Vision encoder. A transformer over those patches (attention between image regions instead of between words), typically pretrained CLIP-style: on hundreds of millions of (image, caption) pairs, pushing each image's vector toward its caption's vector — Lesson 2's "similar meaning, nearby coordinates" built across modalities. A photo of a dog and the text "a dog" end up neighbors.
• Projector. Vision-space and LLM-embedding-space are different coordinate systems (Topic 6's incompatible-maps warning), so a small trained adapter — often just an MLP — translates one into the other. The famous LLaVA recipe proved how little glue is needed: frozen CLIP + tiny projector + Llama, and suddenly the LLM answers questions about pictures. That simplicity is the deep lesson: vision wasn't bolted on as a new faculty; images were translated into the language the model already spoke.

After the merge, there is no special vision machinery — one transformer, one sequence, text tokens attending to image tokens exactly as they attend to each other (Topic 7). "What color is the car?" works because the word "car" literally attends to the patches containing it.

Engineering realities: images are expensive context — a single image typically costs ~500–1,500+ tokens (high-resolution images get split into tiles, each encoded separately, multiplying cost), so your Topic 5/38 budget discipline applies with force. And the strengths/weaknesses follow directly from the architecture: VLMs excel at description, reading text in images, charts, and UI understanding (Topic 61's screenshots, now explained) — but are weak at precise counting and exact spatial coordinates, because the world reaches them pre-chunked into patches: fine detail below patch resolution is simply not in the sequence. That's the mechanical reason browser agents' pixel-clicks miss, completing last lesson's story. Last distinction to keep crisp: VLMs understand images; generating them is a different architecture family (diffusion models) — don't conflate.

---

Topic 63: SLMs — small as a strategy

Small Language Models — roughly 0.5B to 14B parameters, runnable on consumer hardware — used to be understood as "big models, but worse." That framing died, and you already know each nail in its coffin:

1. Overtraining (Topic 9): Chinchilla-optimal was compute-optimal, not inference-optimal — so labs now train small models on trillions of tokens, thousands of tokens per parameter, making modern 8Bs stronger than 2020's 175B.
2. Distillation (Topic 17): frontier teachers manufacture textbook-quality training data for small students — the Phi lineage's whole thesis.
3. The specialist theorem (Topics 11–12, this course's recurring business case): a fine-tuned 8B beats a general giant on the narrow task it was tuned for, at ~1/50th the cost.

So the modern question isn't "how much worse is the small model?" but "which jobs are small-model jobs?" — and Modules 4–7 already gave you the list: the easy 80% behind a router or cascade (Topic 38's economics); agent steps, where low latency and high per-step reliability beat raw brilliance (Topic 58's compounding math loves fast, dependable steps); draft models for speculative decoding (Topic 35 — an SLM's literal job description); guardrails and classifiers (input filtering, routing decisions, injection screening — sub-second, always-on jobs); and on-device deployment for privacy, offline operation, and zero-network latency (Module 9's subject).

The honest limits, with their standard remedies: SLMs hold less world knowledge (→ pair with RAG — the Module 6 marriage: facts from retrieval, fluency from the model; a 3B with good retrieval beats a 70B recalling from stale weights on your documents), reason less deeply over long horizons (→ decompose via Topic 59's workflows, or route hard cases up), and buckle under 40-rule instruction stacks (→ fine-tune the behavior in instead — Topic 12's rung 4, and precisely where your Module 3 skills earn money).

Names to know: Qwen's 0.5B–7B line, Llama 3.2 1B/3B, Phi, Gemma, SmolLM — a tier moving faster than any other, because every technique in this course (distillation, overtraining, quantization, fine-tuning) compounds hardest at small scale.

---

Topic 64: Dense Models — the honest baseline

A short but load-bearing topic. A dense model is one where every parameter participates in every token's forward pass — every FFN neuron fires, every attention matrix multiplies, for every single token. Every model this course has discussed so far — Llama, Qwen's dense line, everything you've fine-tuned — is dense. It's not a variant; it's the default the next topic departs from.

Density's virtues are the quiet ones: predictability and simplicity. The Topic 34 speed formula applies directly (bandwidth ÷ total bytes); the Topic 31 memory budget is exact; quantization behaves uniformly; LoRA targets every layer cleanly (Topic 23's all-linear just works); serving engines need no special handling. When you fine-tune, deploy, and debug a dense model, the whole of Modules 3–5 applies without asterisks.

Density's problem is the scaling bill, and you can state it precisely from Topic 34: to get smarter, add parameters — but every token pays for all of them. Decode reads every weight per token, so capability scales the bandwidth bill linearly. Want 10× the stored knowledge? Pay 10× the memory traffic on every token — including the easy ones. "The" costs as much compute as a theorem. A 400B dense model isn't just expensive to train; it's expensive to whisper.

Which raises the question that defines the next topic: the FFNs — Topic 8 told you they're the knowledge warehouse and ~⅔ of all parameters — must all of them really activate for every token? Does predicting "the" need the entire warehouse walked?

---

Topic 65: MoE — the warehouse with a triage desk

Mixture of Experts answers Topic 64's closing question with surgical precision. The surgery site is exactly where you'd predict: the FFNs — the knowledge warehouse, two-thirds of the parameters. MoE replaces each layer's single FFN with many parallel FFNs ("experts") plus a tiny router network that, per token, picks which few experts (typically 1–2, or 8 of many in fine-grained designs) actually run. Attention stays shared and dense; only the warehouse got subdivided:

This splits one number into two, and the split is the entire concept: total parameters (all experts — the knowledge stored) versus active parameters (the chosen few — the compute spent per token). Real examples: Mixtral 8×7B — 47B total, ~13B active. DeepSeek-V3 — 671B total, 37B active. Qwen3-235B-A22B — the naming now self-decoding: 235B total, 22B Active. The frontier is overwhelmingly MoE; the labels finally tell you why.

The bargain, stated in Module 4's language: the model knows like a 671B (scaling laws reward total capacity — Topic 9) but runs like a 37B — decode's per-token weight-read (Topic 34) touches only the active subset, so speed ≈ bandwidth ÷ active-param bytes. But memory is unforgiving: the router's choice isn't known until the token arrives, so every expert must sit loaded — VRAM scales with total. Capacity of a giant, speed of a mid-size, memory bill of the giant. That asymmetry produces a delightful practical corollary for your hardware specifically: MoE is unified-memory-friendly — Macs have exactly the profile MoE wants (lots of RAM to house all experts, modest bandwidth that only the small active set must cross per token), which is why big MoE models run surprisingly well on high-memory Mac Studios that could never touch an equally-capable dense model.

Three honest complications:

1. "Expert" oversells it. Routing is learned, per-token, per-layer — and interpretability work finds experts specializing in things like punctuation, syntax patterns, or token statistics, not "the math expert" and "the French expert." The metaphor is organizational, not semantic; don't anthropomorphize the warehouse sections (the same discipline as Topic 7's attention-maps warning).
2. Load balancing is a real training battle. Left alone, routing collapses rich-get-richer — a few experts absorb all traffic while others atrophy. Training adds auxiliary balancing losses to keep the warehouse evenly used; it's a genuine source of MoE training instability.
3. The bargain shifts with batch size (advanced, but you have the tools): a single stream touches few experts per step — maximal savings. A large batch (Topic 36) scatters tokens across many experts, so most of the warehouse gets read anyway; MoE's per-token savings partially dissolve at high serving load, changing the serving economics in ways engineers must actually model. Also: fine-tuning MoE is fussier (routers + LoRA interact non-trivially) — one more reason dense stays the default at fine-tunable scales.

---

Topic 66: Coding Models — where the world grades homework

Why did code become the field's favorite domain — the place where models improved fastest and products (Copilot, Claude Code, Cursor) matured first? Four reasons, three of which you can already recite:

1. Verification asymmetry, maximal (Topics 58, 27): code has free, instant, objective verifiers — compilers, type-checkers, test suites, execution. That makes it RLVR heaven (reward = tests pass, unhackable) and agent heaven (every step checkable, converting compounding failure into retry-until-verified).
2. Training data abundance: decades of public code with commits, issues, and reviews — process data, not just artifacts.
3. Economic pull: developers pay, measure productivity, and tolerate iteration.
4. The bonus: code in the pretraining mixture improves general reasoning (Topic 18's note) — structured, causal, consequence-laden text turns out to be excellent brain food.

What makes a model a "coding model" — a recipe you fully possess: heavy code in the data mixture, often via continued pretraining at Topic 20 scale (Code Llama = Llama 2 + ~500B code tokens — the sandwich, deployed); long context for repositories; and one genuinely new training format: FIM — fill-in-the-middle. Standard training predicts left-to-right, but real coding means inserting code between existing lines. FIM restructures training examples with special tokens — here's the prefix, here's the suffix, now produce the middle — teaching bidirectional-ish completion within the causal architecture. FIM is what powers every Tab-completion you've ever accepted.

Which surfaces the key product split — two different animals sharing the word "coding model":

	Completion models	Agentic/chat coding models
Job	Tab-complete as you type	"Fix this issue across the repo"
Training	FIM-heavy	Instruction + tool-use + RL on real tasks
Latency budget	<500 ms — SLM territory (Topic 63)	Minutes acceptable — frontier territory
Shape	One fast infill	Topic 58's loop: read, edit, run tests, iterate

The agentic side is trained increasingly on execution feedback — RL over real repository tasks where reward is the test suite going green — which is why the benchmark that matters shifted from HumanEval (small function puzzles, now saturated and contamination-riddled) to SWE-bench (resolve actual GitHub issues in real repos, end-to-end) and contamination-resistant designs like LiveCodeBench that only use problems published after model cutoffs — Topic 18's decontamination principle, weaponized into benchmark design. Names on the open side: Qwen-Coder, DeepSeek-Coder, Codestral.

---

Topic 67: Reasoning Models — the course comes full circle

The finale, and three flags planted across ten lessons converge here. Recall them: Topic 54 — chain-of-thought works because tokens are compute (each token = one forward pass; reasoning in tokens = buying more passes). Topic 27 — RLVR: replace hackable learned rewards with verifiable ones. Topic 45 — GRPO: sample a group of answers, reward the verifiably correct ones relative to the group, no value model needed.

A reasoning model is what happens when you stop prompting for chain-of-thought and train it in with RL: o1, DeepSeek-R1, Claude's extended thinking, QwQ, Gemini's thinking modes. The recipe, at its cleanest (R1 made it public): take a capable base model; give it math and code problems with checkable answers; let it generate long solution attempts; reward correct final answers (GRPO); repeat at scale. And here is the genuinely remarkable finding — the reasoning behaviors emerge rather than being taught. Nobody wrote demonstrations of self-checking; under nothing but "correct answers score," the model discovers re-examining its work, backtracking ("wait — let me reconsider"), trying alternative approaches, and spending more tokens on harder problems. R1's training logs famously show response lengths growing on their own as the model learns that thinking longer pays. Selection pressure, not instruction.

This creates the field's new scaling axis: test-time compute. For years, capability scaled with parameters and training tokens (Topic 9). Now it also scales with thinking tokens at inference — same weights, more deliberation, better answers, with smooth measurable curves. Modern APIs expose this as a dial (thinking budgets / effort levels), and the industry has converged on hybrid models: one model, toggleable thinking — instant mode for easy queries, deliberation for hard ones.

Two more pieces complete the picture. Reasoning distills (Topic 17, one more encore): R1's release proved that SFT on a big reasoner's thinking traces transfers much of the ability into 7B–32B students — small models visibly deliberating, trained by imitation of deliberation. And the honest cost accounting, in Topic 38's terms: thinking tokens are output tokens — the expensive kind — so reasoning models multiply latency and cost, sometimes 10–50× on hard problems; they overthink trivial queries hilariously (three paragraphs deliberating "what's 2+2"); they help most exactly where their training lived (math, code, logic, planning) and much less on factual recall (weights still don't contain your docs — RAG remains undefeated) or open-ended creative work. The product answer is one you've known since Topic 38: route — deliberation is a premium tier, not a default. One epistemic caveat to carry: the visible thinking text is not guaranteed to faithfully reflect the model's actual computation — research shows the transcript and the true process can diverge. Useful, inspectable, but not a窗 into the soul — the same humility Topic 7 demanded of attention maps.

---

Module 8: complete. The zoo, mapped — and notice what the tour proved: every exotic species decomposed into course fundamentals. Vision = translation into embedding space. Small = overtraining + distillation + specialization. MoE = the FFN warehouse, subdivided. Coding = verification asymmetry, industrialized. Reasoning = CoT + RLVR + GRPO. You didn't learn six new things today; you watched everything you already knew recombine.

Next: Module 9 — Deployment. Shorter and intensely practical: local inference and on-device AI (phones, laptops, and what NPUs actually do), API serving (consolidating your Module 4–5 serving knowledge into deployment shapes), cloud GPUs (the rental market decoded — spot vs on-demand, where to actually rent, cost math), and edge AI (the constraints at the far end). Then Modules 10–11 bring evaluation and the real-world build playbooks that cap the course.