Attention — the mechanism that made everything possible

I'm teaching attention before transformers, because a transformer is just attention plus plumbing. Understand this and the rest is assembly.

The problem: A word's meaning depends on other words. In "The bank of the river," what "bank" means is determined by "river" — a word 3 positions away. In "The trophy didn't fit in the suitcase because it was too big," resolving "it" requires connecting to "trophy," and real dependencies can span thousands of tokens. Old architectures (RNNs) read left to right, passing along a compressed summary — like whispering a message down a line of people. By word 500, the beginning was mush.

The solution: Let every token directly look at every other token, and choose what's relevant. That's attention.

Here's the mechanism, using an analogy that maps one-to-one to the real math. Every token creates three things from its embedding:

• a Query — "here's what I'm looking for"
• a Key — "here's what I contain / here's how to find me"
• a Value — "here's the actual information I'll hand over if you pick me"

Daily-life analogy: A conference networking hall. You (a token) walk in holding a card describing who you want to meet (Query). Everyone wears a name tag describing what they know (Key). You compare your card against every name tag — good matches get high scores. Then you collect actual knowledge (Values) from people, weighted by match score: 70% from the great match, 25% from the decent one, crumbs from everyone else. You walk out transformed — your understanding is now a blend of what the relevant people knew.

In math form (worth seeing once, then we move on):

Attention(Q, K, V) = softmax(Q · Kᵀ / √d) · V

Read it in English: multiply Queries against Keys to get match scores → scale them down (÷√d, just keeps numbers stable) → softmax turns scores into percentages summing to 100% → use those percentages to take a weighted average of Values. That's it. The most important equation in modern AI is a smart weighted average.

Concretely for "bank" in "the bank of the river": bank's Query strongly matches river's Key → bank's output vector absorbs a big helping of river's Value → the vector representing "bank" now literally contains riverness. The ambiguity is resolved geometrically.

Three upgrades that make it powerful:

1. Multi-head attention. Don't do this lookup once — do it 32–128 times in parallel, each "head" with its own learned Q/K/V lenses. One head learns to track grammar relationships, another tracks coreference (which "it" refers to what), another tracks positions in code. Like 32 specialists reading the same sentence, each highlighting different connections, then pooling notes.

2. Causal masking. During next-word prediction, a token may only attend to tokens before it — otherwise it could cheat by peeking at the answer. Implementation is brutally simple: set future match scores to negative infinity, softmax turns them into 0%. This is why these are called "causal" or "decoder-only" models.

3. The cost. Every token compares against every other token → O(n²). Double the context, quadruple the compute. 128K context means ~16 billion pairwise comparisons per layer per head. This single quadratic term is why long context is expensive, why context windows were small for years, and why half of Module 4 (Flash Attention, KV cache) exists. Plant this flag now.

---