Speculative Decoding — cheating the one-token law

The one-token-per-step law looks unbeatable: token N+1 depends on token N, so generation is inherently sequential. You can't parallelize the future… can you?

Here's the loophole, and it comes from an asymmetry you already know. Verifying tokens is cheap; generating them is expensive. Remember prefill (Topic 34): given a batch of tokens that already exist, the model can process them all in parallel in one pass — and that pass yields, at every position, the model's opinion of what token should come next. Checking "would I have written these 5 tokens?" costs one forward pass. Writing those 5 tokens the normal way costs five.

So: get someone cheap to guess the future, and use the expensive model only to check the guesses.

The mechanism:

1. A small draft model (say 1B, sharing the big model's tokenizer) quickly generates K candidate tokens — say 5. Its weight-reads are cheap: 1B vs 70B.
2. The big target model runs one forward pass over all 5 candidates in parallel — prefill-style — producing its own verdict at each position.
3. Walk left to right: accept every drafted token the target agrees with. At the first disagreement, discard the rest of the draft and emit the target's own token there instead (which the verification pass already computed — free).
4. Resume drafting from the new position. Repeat.

Watch it run:

Four tokens emitted for one big-model weight read instead of four. And that yellow box is the part that sounds too good to be true but isn't: the accept/reject rule (with a small statistical correction called rejection sampling when you're using temperature) provably produces the exact same output distribution as the big model alone. The big model never signs anything it wouldn't have written. This is not a quality trade — it's pure speed, typically 2–3×.

Why does this beat the bandwidth ceiling? It doesn't — it routes around it using Topic 34's leftover resource. Decode is memory-bound: per step, the weights get read while compute idles at ~5%. The verification pass processes 5 tokens for essentially the same memory traffic as processing 1 — the extra work lands on the idle compute. Speculative decoding is literally the conversion of wasted FLOPs into tokens per second.

The speedup depends on one number: the acceptance rate — how often the junior's guesses survive review. Predictable text (code, boilerplate, structured output, formulaic prose) drafts beautifully: 70–80% acceptance, big wins. Surprising text (high-temperature creative writing) drafts poorly, and at low acceptance the overhead can even make things slower. Which sets up the variants worth knowing:

• Separate draft model — the classic; needs a small sibling with the same tokenizer (Topic 4: tokenizers don't mix).
• Self-speculation (Medusa, EAGLE) — no second model; bolt extra lightweight prediction heads onto the big model itself so it drafts its own future. EAGLE-family methods are the current state of the art in serving engines.
• Prompt-lookup / n-gram decoding — the free gem: draft by copying from the prompt. No model at all — when the output is likely to reuse input text (RAG answers quoting context, code editing, summarization), just propose the continuation from the matching text you already have. Costs nothing, wins big exactly where products like RAG chatbots live.

One honest asterisk for the advanced file: speculative decoding shines at low batch sizes — local single-user inference, latency-critical paths. On a serving GPU already saturated with a large batch (next topic), the "idle compute" is already spoken for, and the technique's advantage shrinks. Know which regime you're in.

---