Transformers — the full architecture

Now assemble the machine. The transformer (from the 2017 paper "Attention Is All You Need") is what happens when you stack attention with a few other components, many times.

One transformer block = two stations:

Station 1 — Attention (Topic 7): tokens exchange information. This is the communication step. "Let every word gather context from other words."

Station 2 — Feed-Forward Network (FFN, also called MLP): each token, now enriched with context, gets processed individually by a small neural network. No token-to-token communication here. This is the thinking/lookup step — research suggests FFNs act like the model's key-value memory where facts and patterns are stored. Roughly: attention figured out "we're talking about the capital of Pakistan," and the FFN is where "→ Islamabad" gets retrieved.

Daily-life analogy: A team workflow with alternating phases. Phase 1: everyone in a meeting room, sharing information (attention). Phase 2: everyone goes back to their desk and processes what they heard alone (FFN). Then another meeting, another desk session — repeated 32–100+ times.

Around these two stations there's plumbing that makes deep stacks trainable:

• Residual connections: each station's output is added to its input, not replacing it: x = x + attention(x). The token's vector is like a running document, and each layer appends edits rather than rewriting from scratch. Without this, gradients die in deep networks and training fails. Arguably the most underrated trick in deep learning.
• Normalization (LayerNorm/RMSNorm): rescales vectors between stations so numbers don't explode or vanish. Think volume control between amplifier stages.
• Positional information: attention itself is order-blind — "dog bites man" and "man bites dog" would look identical. So position gets injected into the vectors. Modern models mostly use RoPE (rotary embeddings), which encodes position as a rotation of the Q/K vectors — a clever scheme, and the thing people manipulate ("RoPE scaling") to stretch context windows beyond training length.

The full pipeline, end to end:

text → tokenizer → token IDs → embeddings 
     → [Block 1: attention + FFN]
     → [Block 2: attention + FFN]
     → ... (×32 for a 7B model, ×80+ for the biggest)
     → final vector for the last position
     → unembedding matrix → scores over the whole vocabulary
     → softmax → probabilities → sample one token
     → append it to the input, REPEAT THE WHOLE THING for the next token

That last line deserves emphasis: generation is autoregressive — one token at a time, each full forward pass producing exactly one token, which gets appended and fed back. A 500-token answer = 500 full passes through the entire stack. (Now you can already smell why the KV cache in Module 4 exists — recomputing attention for all previous tokens every pass would be insane.)

What happens across depth? Interpretability research shows a rough progression: early layers handle syntax and local structure, middle layers build semantic and factual representations, late layers get concrete about the specific next token. The vector flowing through is a thought being progressively refined.

One more term you'll encounter: the original 2017 transformer had an encoder (reads input bidirectionally) and decoder (generates output). Modern LLMs — GPT, Llama, Qwen, Claude — are decoder-only: just the generation stack with causal masking. Embedding models like BERT are encoder-only. When you see these words, that's all they mean: which half of the original design survived.

---