Chunking — the cut decides the quality

Chunking looks like a preprocessing footnote. It is, in practice, the highest-leverage decision in most RAG systems — more than the vector database, often more than the embedding model. Here's why it can't be dodged.

You know from Lesson 2 that an embedding is a lossy compression of meaning into one vector. Compress a sentence: crisp. Compress a 40-page document: the vector becomes an average of forty pages of topics — a gray smear that matches everything weakly and nothing well. But cut too small and each fragment loses the context needed to be understood ("It increased by 40%" — what did?). Chunking is finding the size where each piece is about one thing, and understandable alone.

The film-archive analogy: you're cutting a movie into clips for a searchable library. Keep whole reels and every search returns two hours of footage for a ten-second moment. Cut mid-scene and no clip makes sense on its own. You want cuts at scene boundaries — with a little overlap so nothing important falls between two clips.

The strategy ladder, simplest to smartest:

1. Fixed-size with overlap — the baseline: ~300–800 tokens per chunk, 10–20% overlap between neighbors (the overlap exists precisely so an idea straddling a boundary survives whole in at least one chunk). Works surprisingly well; start here.
2. Recursive splitting — try to split on paragraph breaks first, fall back to sentences, then words, only when a piece is still too big. Respects natural seams instead of cutting mid-sentence.
3. Structure-aware — the professional default: split documents by headings/sections, code by functions and classes (parser-based, not line counts), PDFs with awareness of pages and tables. The document's author already marked the scene boundaries; use them.
4. Semantic chunking — embed each sentence, walk the sequence, and cut wherever similarity between neighbors drops (a topic shift, detected geometrically). Elegant, costlier, sometimes worth it for messy unstructured text.

Two modern upgrades worth knowing by name:

• Contextual retrieval (popularized by Anthropic): before embedding each chunk, prepend a short LLM-generated situating line — "This chunk is from the Q3 sales report, section on the Karachi region…" — so the fragment carries its context into the vector. Directly attacks the "understandable alone" problem; measured retrieval-failure reductions are large.
• Small-to-big (parent-document) retrieval: search over small, precise chunks, but return the larger parent section they came from. Matching precision of small pieces, reading context of big ones — decoupling the two jobs a chunk was doing.

And always attach metadata to every chunk — source, title, section, date, permissions. It costs nothing at indexing time and enables filtering, citations, and access control forever after.

---