Embeddings — meaning as coordinates

In Lesson 1, text became token IDs: [9906, 445, ...]. But an ID is just an index — token 9906 isn't "more" than token 445 in any meaningful way. IDs carry zero meaning. Step one inside the model is fixing that.

An embedding converts each token ID into a vector — a long list of numbers, typically 1,024 to 16,000+ numbers per token. Think of it as coordinates in a space with thousands of dimensions.

Why coordinates? Because in space, you can measure distance. And the whole idea is: similar meanings should be nearby.

You live in 3D, so imagine a 3D version: "king" sits at some point, "queen" nearby, "monarch" nearby, "banana" far away. Now scale to 4,096 dimensions. Each dimension loosely captures some learned aspect of meaning — no one labels them, but if you probe them, you find directions in this space corresponding to things like gender, formality, tense, "is a place," "is code."

The famous demonstration: vector("king") - vector("man") + vector("woman") ≈ vector("queen"). Meaning became arithmetic. That should feel slightly shocking. It means relationships between concepts are directions in this space.

Meaning as coordinates: king, queen, and monarch cluster together while banana sits far away, and the man → woman direction matches the king → queen direction — the arithmetic king − man + woman ≈ queen falls out of the geometry.

How are embeddings learned? They start random. During training, every time the model mispredicts, blame flows back into the embedding vectors too — tokens used in similar contexts get pushed toward each other. "Doctor" and "physician" end up neighbors purely because they appear in interchangeable sentence patterns. This is the distributional hypothesis: you shall know a word by the company it keeps.

Two embeddings you must not confuse (this trips up even experienced devs):

Token embeddings — the model's internal input layer, converting each token to a vector. One vector per token.
Sentence/document embeddings — a separate kind of model (like text-embedding-3 or BGE) that compresses an entire sentence or paragraph into a single vector. This is what powers semantic search and RAG (Module 6).

Same math, different jobs. When someone says "embed your documents," they mean type 2.

Measuring similarity between two vectors is usually cosine similarity — essentially the angle between them. Pointing the same direction = similar meaning (score near 1). Perpendicular = unrelated (near 0).

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
 
v = model.encode(["I love biryani", "This dish is delicious", "Kubernetes pod crashed"])
from sentence_transformers.util import cos_sim
print(cos_sim(v[0], v[1]))  # high (~0.6) — related meaning, zero shared words
print(cos_sim(v[0], v[2]))  # low  (~0.05)

Notice: sentence 1 and 2 share no words, yet score high. That's the leap beyond keyword search — matching meaning, not spelling.

Summary

Embeddings turn tokens (or whole texts) into vectors where distance = semantic similarity. Meaning becomes geometry, learned automatically from context patterns.

Mental model

A city map of meaning. Every concept has an address; related concepts live in the same neighborhood; relationships are directions ("go north = more formal").

Mistakes to avoid

Comparing vectors from different embedding models. Each model has its own coordinate system — a point from map A means nothing on map B. In RAG, index and query with the same model, always.
Assuming embeddings capture everything. They compress meaning lossily — negation ("I do NOT like X") and exact numbers are notorious weak spots.

Exercise

Using the snippet above, embed 10 sentences from your own domain (5 about one topic, 5 about another). Compute all pairwise similarities and print a 10×10 grid. You should see two bright blocks. You've just built the core of a semantic search engine in ~20 lines.