VLMs — how images become tokens

A transformer processes a sequence of vectors. It does not care what those vectors originally were. That single fact is the whole of vision-language modelling: get an image into vector form the model already understands, and every mechanism from Lesson 2 works unchanged.

The three parts, decoded:

• Patches — "visual words." A Vision Transformer (ViT) splits the image into a grid of small tiles (commonly 14×14 or 16×16 pixels) and treats each as one unit — the visual analog of Topic 3's tokens. LEGO bricks again, cut from pixels.
• Vision encoder. A transformer over those patches (attention between image regions instead of between words), typically pretrained CLIP-style: on hundreds of millions of (image, caption) pairs, pushing each image's vector toward its caption's vector — Lesson 2's "similar meaning, nearby coordinates" built across modalities. A photo of a dog and the text "a dog" end up neighbors.
• Projector. Vision-space and LLM-embedding-space are different coordinate systems (Topic 6's incompatible-maps warning), so a small trained adapter — often just an MLP — translates one into the other. The famous LLaVA recipe proved how little glue is needed: frozen CLIP + tiny projector + Llama, and suddenly the LLM answers questions about pictures. That simplicity is the deep lesson: vision wasn't bolted on as a new faculty; images were translated into the language the model already spoke.

After the merge, there is no special vision machinery — one transformer, one sequence, text tokens attending to image tokens exactly as they attend to each other (Topic 7). "What color is the car?" works because the word "car" literally attends to the patches containing it.

Engineering realities: images are expensive context — a single image typically costs ~500–1,500+ tokens (high-resolution images get split into tiles, each encoded separately, multiplying cost), so your Topic 5/38 budget discipline applies with force. And the strengths/weaknesses follow directly from the architecture: VLMs excel at description, reading text in images, charts, and UI understanding (Topic 61's screenshots, now explained) — but are weak at precise counting and exact spatial coordinates, because the world reaches them pre-chunked into patches: fine detail below patch resolution is simply not in the sequence. That's the mechanical reason browser agents' pixel-clicks miss, completing last lesson's story. Last distinction to keep crisp: VLMs understand images; generating them is a different architecture family (diffusion models) — don't conflate.

---