Lesson 2Inside the Machine

Lesson 2 keeps the plain language of Lesson 1 but goes further under the hood: actual mechanics, some math intuition, and real numbers. This is the lesson where the "magic" becomes engineering — and where attention, the most beautiful idea in the field, finally gets explained properly.

Topic 6: Embeddings — meaning as coordinates

In Lesson 1, text became token IDs: [9906, 445, ...]. But an ID is just an index — token 9906 isn't "more" than token 445 in any meaningful way. IDs carry zero meaning. Step one inside the model is fixing that.

An embedding converts each token ID into a vector — a long list of numbers, typically 1,024 to 16,000+ numbers per token. Think of it as coordinates in a space with thousands of dimensions.

Why coordinates? Because in space, you can measure distance. And the whole idea is: similar meanings should be nearby.

You live in 3D, so imagine a 3D version: "king" sits at some point, "queen" nearby, "monarch" nearby, "banana" far away. Now scale to 4,096 dimensions. Each dimension loosely captures some learned aspect of meaning — no one labels them, but if you probe them, you find directions in this space corresponding to things like gender, formality, tense, "is a place," "is code."

The famous demonstration: vector("king") - vector("man") + vector("woman") ≈ vector("queen"). Meaning became arithmetic. That should feel slightly shocking. It means relationships between concepts are directions in this space.

How are embeddings learned? They start random. During training, every time the model mispredicts, blame flows back into the embedding vectors too — tokens used in similar contexts get pushed toward each other. "Doctor" and "physician" end up neighbors purely because they appear in interchangeable sentence patterns. This is the distributional hypothesis: you shall know a word by the company it keeps.

Two embeddings you must not confuse (this trips up even experienced devs):

1. Token embeddings — the model's internal input layer, converting each token to a vector. One vector per token.
2. Sentence/document embeddings — a separate kind of model (like text-embedding-3 or BGE) that compresses an entire sentence or paragraph into a single vector. This is what powers semantic search and RAG (Module 6).

Same math, different jobs. When someone says "embed your documents," they mean type 2.

Measuring similarity between two vectors is usually cosine similarity — essentially the angle between them. Pointing the same direction = similar meaning (score near 1). Perpendicular = unrelated (near 0).

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
 
v = model.encode(["I love biryani", "This dish is delicious", "Kubernetes pod crashed"])
from sentence_transformers.util import cos_sim
print(cos_sim(v[0], v[1]))  # high (~0.6) — related meaning, zero shared words
print(cos_sim(v[0], v[2]))  # low  (~0.05)

Notice: sentence 1 and 2 share no words, yet score high. That's the leap beyond keyword search — matching meaning, not spelling.

Topic 7: Attention — the mechanism that made everything possible

I'm teaching attention before transformers, because a transformer is just attention plus plumbing. Understand this and the rest is assembly.

The problem: A word's meaning depends on other words. In "The bank of the river," what "bank" means is determined by "river" — a word 3 positions away. In "The trophy didn't fit in the suitcase because it was too big," resolving "it" requires connecting to "trophy," and real dependencies can span thousands of tokens. Old architectures (RNNs) read left to right, passing along a compressed summary — like whispering a message down a line of people. By word 500, the beginning was mush.

The solution: Let every token directly look at every other token, and choose what's relevant. That's attention.

Here's the mechanism, using an analogy that maps one-to-one to the real math. Every token creates three things from its embedding:

• a Query — "here's what I'm looking for"
• a Key — "here's what I contain / here's how to find me"
• a Value — "here's the actual information I'll hand over if you pick me"

Daily-life analogy: A conference networking hall. You (a token) walk in holding a card describing who you want to meet (Query). Everyone wears a name tag describing what they know (Key). You compare your card against every name tag — good matches get high scores. Then you collect actual knowledge (Values) from people, weighted by match score: 70% from the great match, 25% from the decent one, crumbs from everyone else. You walk out transformed — your understanding is now a blend of what the relevant people knew.

In math form (worth seeing once, then we move on):

Attention(Q, K, V) = softmax(Q · Kᵀ / √d) · V

Read it in English: multiply Queries against Keys to get match scores → scale them down (÷√d, just keeps numbers stable) → softmax turns scores into percentages summing to 100% → use those percentages to take a weighted average of Values. That's it. The most important equation in modern AI is a smart weighted average.

Concretely for "bank" in "the bank of the river": bank's Query strongly matches river's Key → bank's output vector absorbs a big helping of river's Value → the vector representing "bank" now literally contains riverness. The ambiguity is resolved geometrically.

Three upgrades that make it powerful:

1. Multi-head attention. Don't do this lookup once — do it 32–128 times in parallel, each "head" with its own learned Q/K/V lenses. One head learns to track grammar relationships, another tracks coreference (which "it" refers to what), another tracks positions in code. Like 32 specialists reading the same sentence, each highlighting different connections, then pooling notes.

2. Causal masking. During next-word prediction, a token may only attend to tokens before it — otherwise it could cheat by peeking at the answer. Implementation is brutally simple: set future match scores to negative infinity, softmax turns them into 0%. This is why these are called "causal" or "decoder-only" models.

3. The cost. Every token compares against every other token → O(n²). Double the context, quadruple the compute. 128K context means ~16 billion pairwise comparisons per layer per head. This single quadratic term is why long context is expensive, why context windows were small for years, and why half of Module 4 (Flash Attention, KV cache) exists. Plant this flag now.

Topic 8: Transformers — the full architecture

Now assemble the machine. The transformer (from the 2017 paper "Attention Is All You Need") is what happens when you stack attention with a few other components, many times.

One transformer block = two stations:

Station 1 — Attention (Topic 7): tokens exchange information. This is the communication step. "Let every word gather context from other words."

Station 2 — Feed-Forward Network (FFN, also called MLP): each token, now enriched with context, gets processed individually by a small neural network. No token-to-token communication here. This is the thinking/lookup step — research suggests FFNs act like the model's key-value memory where facts and patterns are stored. Roughly: attention figured out "we're talking about the capital of Pakistan," and the FFN is where "→ Islamabad" gets retrieved.

Daily-life analogy: A team workflow with alternating phases. Phase 1: everyone in a meeting room, sharing information (attention). Phase 2: everyone goes back to their desk and processes what they heard alone (FFN). Then another meeting, another desk session — repeated 32–100+ times.

Around these two stations there's plumbing that makes deep stacks trainable:

• Residual connections: each station's output is added to its input, not replacing it: x = x + attention(x). The token's vector is like a running document, and each layer appends edits rather than rewriting from scratch. Without this, gradients die in deep networks and training fails. Arguably the most underrated trick in deep learning.
• Normalization (LayerNorm/RMSNorm): rescales vectors between stations so numbers don't explode or vanish. Think volume control between amplifier stages.
• Positional information: attention itself is order-blind — "dog bites man" and "man bites dog" would look identical. So position gets injected into the vectors. Modern models mostly use RoPE (rotary embeddings), which encodes position as a rotation of the Q/K vectors — a clever scheme, and the thing people manipulate ("RoPE scaling") to stretch context windows beyond training length.

The full pipeline, end to end:

text → tokenizer → token IDs → embeddings 
     → [Block 1: attention + FFN]
     → [Block 2: attention + FFN]
     → ... (×32 for a 7B model, ×80+ for the biggest)
     → final vector for the last position
     → unembedding matrix → scores over the whole vocabulary
     → softmax → probabilities → sample one token
     → append it to the input, REPEAT THE WHOLE THING for the next token

That last line deserves emphasis: generation is autoregressive — one token at a time, each full forward pass producing exactly one token, which gets appended and fed back. A 500-token answer = 500 full passes through the entire stack. (Now you can already smell why the KV cache in Module 4 exists — recomputing attention for all previous tokens every pass would be insane.)

What happens across depth? Interpretability research shows a rough progression: early layers handle syntax and local structure, middle layers build semantic and factual representations, late layers get concrete about the specific next token. The vector flowing through is a thought being progressively refined.

One more term you'll encounter: the original 2017 transformer had an encoder (reads input bidirectionally) and decoder (generates output). Modern LLMs — GPT, Llama, Qwen, Claude — are decoder-only: just the generation stack with causal masking. Embedding models like BERT are encoder-only. When you see these words, that's all they mean: which half of the original design survived.

Topic 9: Parameters — the knobs, quantified

Time to make "billions of knobs" concrete, because parameter count drives everything you'll do practically: what fits on your Mac M4, what fine-tuning costs, what inference speed you get.

A parameter is one learned number. Every weight in every attention matrix, every FFN, every embedding vector — one float each. "Llama-3-8B" = 8 billion learned floats.

Where do they live in, say, a 7B model? Roughly:

• Embedding + unembedding matrices: ~0.5B (vocab size × hidden dimension)
• Attention (all Q/K/V/output matrices, all layers): ~2B
• FFNs: ~4.5B — the majority, as promised

The math you'll use constantly — memory footprint:

Each parameter stored in standard precision (FP16/BF16 — 16-bit floats) takes 2 bytes.

7B model  × 2 bytes = 14 GB   → doesn't fit consumer GPUs comfortably
7B model  × 1 byte  (8-bit)  = 7 GB    → fits a decent GPU
7B model  × 0.5 byte (4-bit) = ~4 GB   → runs on your MacBook
70B model × 2 bytes = 140 GB  → multiple datacenter GPUs

This one multiplication — params × bytes-per-param — is the first calculation of every local-AI decision you'll ever make. (Shrinking bytes-per-param is quantization, Module 3. Add ~1–2 GB overhead for the KV cache and activations on top.)

Does bigger = better? Yes, but with two huge caveats.

First, scaling laws: performance improves smoothly and predictably with more parameters + more data + more compute. This predictability is why labs confidently spend hundreds of millions on training runs — they can forecast the result before starting.

Second, the Chinchilla insight (DeepMind, 2022): for a fixed compute budget, most early models were too big and undertrained. Better to train a smaller model on more tokens. Rule of thumb from the paper: ~20 tokens of training data per parameter for compute-optimal training. But here's the modern twist — compute-optimal isn't inference-optimal. A model gets trained once but run billions of times, so today's labs deliberately "overtrain" small models far past 20:1 (Llama-3-8B saw ~15 trillion tokens — nearly 2,000 tokens per parameter). That's why a modern 8B model demolishes a 2020-era 175B model. Parameter count alone tells you almost nothing about quality anymore; training data volume and quality matter as much.

Also distinguish: parameters (learned, fixed after training) vs hyperparameters (choices made by humans before training: how many layers, learning rate, batch size — the settings of the training process itself, not the learned values). You'll set hyperparameters yourself when fine-tuning in Module 3.

Topic 10: Training vs Inference — building the brain vs using it

Two completely different modes of a model's existence. Confusing them causes more beginner misconceptions than anything else.

Training = adjusting the parameters. Inference = using frozen parameters to generate. Everything differs between them:

The memory row deserves explanation because it shocks people: training a 7B model in standard fashion needs not 14 GB but ~80–120 GB. Why? For every parameter you also store its gradient (which direction to nudge it — the blame signal from backpropagation) and optimizer states (the Adam optimizer keeps two running averages per parameter to make updates smoother — think momentum, like remembering which direction you've been nudging so you don't zigzag). That's roughly 16 bytes per parameter for training vs 2 for inference. This 8× gap is exactly the problem LoRA and QLoRA solve in Module 3 — remember this number.

The modern training pipeline — a preview map of Module 2–3, because these stages are the vocabulary of the whole field:

1. Pretraining: next-token prediction on trillions of tokens of raw internet/books/code. Produces a base model — a brilliant text-completer with zero manners. Ask it "What is Kubernetes?" and it might continue "…is a question often asked in interviews. Other common questions include…" — because that's a plausible continuation, and continuation is all it knows.
2. Supervised Fine-Tuning (SFT): train on curated (instruction → good response) pairs. Teaches the format of being an assistant: when given a question, answer it.
3. Preference tuning (RLHF/DPO): teach it which answers humans prefer — helpful over evasive, honest over confident nonsense.

Base model = raw talent. SFT = job training. Preference tuning = professional polish. When you download models, you'll see this in the names: Llama-3-8B (base) vs Llama-3-8B-Instruct (steps 2–3 applied). Grabbing the base by accident and wondering why it won't answer questions is a rite of passage — skip it.

One more critical point, repeated from Lesson 1 because it matters: your conversations are inference. The model learns nothing from them. Any "learning" during a chat is just information sitting in the context window (the whiteboard), gone when the conversation ends.

Topic 11: Open-Source vs Closed-Source Models — the ecosystem map

Last foundation: the landscape you'll build in.

Closed-source (API models): GPT, Claude, Gemini. You send a request over the internet, tokens come back. The weights — the actual parameter files — never leave the lab's servers.

Open-weight models: Llama (Meta), Qwen (Alibaba), Mistral, DeepSeek, Gemma (Google). You download the parameter files themselves and run them anywhere: your Mac, a rented GPU, an air-gapped server.

Precision matters on terminology, and sloppy usage will mark you as a beginner: most "open-source" models are really open-weight — you get the trained parameters, but usually not the training data, and often not the full training code. True open-source (data + code + weights, like OLMo from AI2) is rare. Licenses vary too: some are genuinely permissive (Apache 2.0 — Qwen, Mistral), some have restrictions (Llama's community license has clauses about very large companies and usage). Read the license before building a product — this is a real engineering-decision input, not lawyer theater.

The real decision matrix:

How to actually choose (the pattern nearly everyone converges to):

• Prototype on closed APIs. Fastest way to prove an idea works. Never start a project by setting up GPU infrastructure.
• Consider open weights when: volume makes per-token pricing painful; data can't leave the premises (healthcare, legal, government — a huge deal for selling AI services in markets with data-sovereignty concerns); you need deep customization via fine-tuning; or you need offline/edge operation.
• The hybrid is the pro move: a router sends easy, high-volume requests to a cheap small open model and hard requests to a frontier API. Best economics, best quality where it counts.

Strategic note worth internalizing: open-weight models trail the frontier by roughly 6–12 months in capability, and that lag has been shrinking. Meanwhile, for any single well-defined task, a fine-tuned 8B open model frequently beats a giant general API model — a specialist beats a generalist on the specialist's turf, at 1/50th the cost. That sentence is the entire business case for Module 3, and for a large fraction of the paid AI engineering work that exists.

Foundations: complete. You now hold the full picture: text → tokens → embeddings → attention-powered transformer blocks → probability → one token at a time, with learned parameters frozen at inference, in an ecosystem split between rented and owned models.

Two threads I deliberately planted for later: the n² attention cost and the 8× training memory multiplier. Modules 3 and 4 exist almost entirely to fight those two numbers.

Lesson 3 starts Module 2: Datasets & Training — SFT datasets, instruction tuning, preference data, synthetic data, curation, cleaning, formatting. This is where you shift from understanding models to shaping them, and dataset skill is genuinely the highest-leverage, most underrated skill in fine-tuning.