MoE — the warehouse with a triage desk

Mixture of Experts answers Topic 64's closing question with surgical precision. The surgery site is exactly where you'd predict: the FFNs — the knowledge warehouse, two-thirds of the parameters. MoE replaces each layer's single FFN with many parallel FFNs ("experts") plus a tiny router network that, per token, picks which few experts (typically 1–2, or 8 of many in fine-grained designs) actually run. Attention stays shared and dense; only the warehouse got subdivided:

This splits one number into two, and the split is the entire concept: total parameters (all experts — the knowledge stored) versus active parameters (the chosen few — the compute spent per token). Real examples: Mixtral 8×7B — 47B total, ~13B active. DeepSeek-V3 — 671B total, 37B active. Qwen3-235B-A22B — the naming now self-decoding: 235B total, 22B Active. The frontier is overwhelmingly MoE; the labels finally tell you why.

The bargain, stated in Module 4's language: the model knows like a 671B (scaling laws reward total capacity — Topic 9) but runs like a 37B — decode's per-token weight-read (Topic 34) touches only the active subset, so speed ≈ bandwidth ÷ active-param bytes. But memory is unforgiving: the router's choice isn't known until the token arrives, so every expert must sit loaded — VRAM scales with total. Capacity of a giant, speed of a mid-size, memory bill of the giant. That asymmetry produces a delightful practical corollary for your hardware specifically: MoE is unified-memory-friendly — Macs have exactly the profile MoE wants (lots of RAM to house all experts, modest bandwidth that only the small active set must cross per token), which is why big MoE models run surprisingly well on high-memory Mac Studios that could never touch an equally-capable dense model.

Three honest complications:

1. "Expert" oversells it. Routing is learned, per-token, per-layer — and interpretability work finds experts specializing in things like punctuation, syntax patterns, or token statistics, not "the math expert" and "the French expert." The metaphor is organizational, not semantic; don't anthropomorphize the warehouse sections (the same discipline as Topic 7's attention-maps warning).
2. Load balancing is a real training battle. Left alone, routing collapses rich-get-richer — a few experts absorb all traffic while others atrophy. Training adds auxiliary balancing losses to keep the warehouse evenly used; it's a genuine source of MoE training instability.
3. The bargain shifts with batch size (advanced, but you have the tools): a single stream touches few experts per step — maximal savings. A large batch (Topic 36) scatters tokens across many experts, so most of the warehouse gets read anyway; MoE's per-token savings partially dissolve at high serving load, changing the serving economics in ways engineers must actually model. Also: fine-tuning MoE is fussier (routers + LoRA interact non-trivially) — one more reason dense stays the default at fine-tunable scales.

---