Dense Models — the honest baseline

Topic 64 of 90Module 8: Model Types2 min read

A short but load-bearing topic. A dense model is one where every parameter participates in every token's forward pass — every FFN neuron fires, every attention matrix multiplies, for every single token. Every model this course has discussed so far — Llama, Qwen's dense line, everything you've fine-tuned — is dense. It's not a variant; it's the default the next topic departs from.

Density's virtues are the quiet ones: predictability and simplicity. The Topic 34 speed formula applies directly (bandwidth ÷ total bytes); the Topic 31 memory budget is exact; quantization behaves uniformly; LoRA targets every layer cleanly (Topic 23's all-linear just works); serving engines need no special handling. When you fine-tune, deploy, and debug a dense model, the whole of Modules 3–5 applies without asterisks.

Density's problem is the scaling bill, and you can state it precisely from Topic 34: to get smarter, add parameters — but every token pays for all of them. Decode reads every weight per token, so capability scales the bandwidth bill linearly. Want 10× the stored knowledge? Pay 10× the memory traffic on every token — including the easy ones. "The" costs as much compute as a theorem. A 400B dense model isn't just expensive to train; it's expensive to whisper.

Which raises the question that defines the next topic: the FFNs — Topic 8 told you they're the knowledge warehouse and ~⅔ of all parameters — must all of them really activate for every token? Does predicting "the" need the entire warehouse walked?

Summary

Dense = all parameters active per token. Simple, predictable, friendly to every technique you've learned — and its capability scales its per-token cost linearly, which is the exact pressure that produced MoE.

Mental model

A committee where every member must review every document, however trivial. Wonderfully thorough, perfectly predictable, and increasingly absurd as the committee grows.

Mistakes to avoid

  • Treating "dense" as a criticism. Below ~30B, and for fine-tuning workflows, dense remains the right default — the asterisk-free option.
  • Forgetting the word exists. When a model card says "dense," it's telling you the Topic 31/34 math applies directly — that's information, not filler.

Exercise

A one-minute calibration: using Topic 34's formula, compute decode tok/s ceilings on an H100 (3.35 TB/s) for dense models of 8B, 70B, and 400B at FP8 (1 byte/param). Watch the ceiling collapse (~420 → ~48 → ~8). Hold that 400B number in mind — the next topic exists to rescue it.