Lesson 10Giving the Model a Library and a Past

Welcome to Module 6: RAG & Memory — the course's pivot point. Modules 1–5 were about the model; from here on, it's about the system around the model. And this module cashes a promise made all the way back in Topic 12: "fine-tuning for form, RAG for facts." Facts finally get their architecture.

One reassurance before we start: you already own the hard parts. Embeddings (Topic 6), the whiteboard (Topic 5), grounding as anti-hallucination layer 2 (Topic 21), even a working semantic search engine (your Lesson 2 exercise — that 10×10 similarity grid). This module assembles pieces you've held for weeks.

Topic 48: RAG — the open-book exam

RAG (Retrieval-Augmented Generation) answers the question every product hits in week one: how does the model know about MY data? Your docs, your tickets, yesterday's news — none of it is in the weights, and Topic 20 taught you the weight-based path (continued pretraining) needs 100M+ tokens and a GPU budget. RAG takes the other path entirely:

This works because of an asymmetry you've known since Topic 21: models are dramatically more reliable reading from context than recalling from weights. A closed-book exam invites confident fabrication; an open-book exam mostly just requires finding the right page. RAG is the machinery that finds the page.

The architecture — two phases, always:

Picking up right where we left off — here's the RAG architecture, drawn:

And here's the whole loop in code — deliberately minimal, no framework, because you should see there's no magic. It's your Lesson 2 exercise plus a prompt:

from sentence_transformers import SentenceTransformer, util
 
embedder = SentenceTransformer("all-MiniLM-L6-v2")
chunks = [...]                                   # your document pieces (Topic 49)
index  = embedder.encode(chunks)                 # indexing phase, done once
 
def rag_answer(question):
    q = embedder.encode(question)
    top = util.cos_sim(q, index)[0].topk(5)      # retrieve
    context = "\n\n".join(chunks[i] for i in top.indices)
    prompt = f"""Answer using ONLY the context below.
If the answer isn't in the context, say so.        # Topic 21, layer 2
 
Context:
{context}
 
Question: {question}"""
    return llm(prompt)                            # generate

Fifteen lines. Everything else this module teaches is about making each line good: what chunks should be (Topic 49), what replaces that brute-force cos_sim at scale (Topic 50), why encode alone isn't enough (Topic 51), and what belongs between retrieve and generate (Topic 52).

Why RAG won over the obvious alternatives:

• vs. fine-tuning facts in: you know this one cold — Topics 12, 14, 20. Below 100M tokens it barely works, and it trains hallucination.
• vs. "just use the 128K context window": four reasons, all from your own modules. Cost — Topic 5: you pay for the whole whiteboard every turn, and Topic 33: prefill on huge prompts is real quadratic compute; stuffing 100K tokens to answer one question is burning money. Quality — lost-in-the-middle: relevance dilution is real. Freshness — re-index tonight vs. nothing. Access control — the sleeper enterprise killer: retrieval can filter by who's asking ("only chunks this user's role may see"), which no amount of context stuffing or fine-tuning can do.

Topic 49: Chunking — the cut decides the quality

Chunking looks like a preprocessing footnote. It is, in practice, the highest-leverage decision in most RAG systems — more than the vector database, often more than the embedding model. Here's why it can't be dodged.

You know from Lesson 2 that an embedding is a lossy compression of meaning into one vector. Compress a sentence: crisp. Compress a 40-page document: the vector becomes an average of forty pages of topics — a gray smear that matches everything weakly and nothing well. But cut too small and each fragment loses the context needed to be understood ("It increased by 40%" — what did?). Chunking is finding the size where each piece is about one thing, and understandable alone.

The film-archive analogy: you're cutting a movie into clips for a searchable library. Keep whole reels and every search returns two hours of footage for a ten-second moment. Cut mid-scene and no clip makes sense on its own. You want cuts at scene boundaries — with a little overlap so nothing important falls between two clips.

The strategy ladder, simplest to smartest:

1. Fixed-size with overlap — the baseline: ~300–800 tokens per chunk, 10–20% overlap between neighbors (the overlap exists precisely so an idea straddling a boundary survives whole in at least one chunk). Works surprisingly well; start here.
2. Recursive splitting — try to split on paragraph breaks first, fall back to sentences, then words, only when a piece is still too big. Respects natural seams instead of cutting mid-sentence.
3. Structure-aware — the professional default: split documents by headings/sections, code by functions and classes (parser-based, not line counts), PDFs with awareness of pages and tables. The document's author already marked the scene boundaries; use them.
4. Semantic chunking — embed each sentence, walk the sequence, and cut wherever similarity between neighbors drops (a topic shift, detected geometrically). Elegant, costlier, sometimes worth it for messy unstructured text.

Two modern upgrades worth knowing by name:

• Contextual retrieval (popularized by Anthropic): before embedding each chunk, prepend a short LLM-generated situating line — "This chunk is from the Q3 sales report, section on the Karachi region…" — so the fragment carries its context into the vector. Directly attacks the "understandable alone" problem; measured retrieval-failure reductions are large.
• Small-to-big (parent-document) retrieval: search over small, precise chunks, but return the larger parent section they came from. Matching precision of small pieces, reading context of big ones — decoupling the two jobs a chunk was doing.

And always attach metadata to every chunk — source, title, section, date, permissions. It costs nothing at indexing time and enables filtering, citations, and access control forever after.

Topic 50: Vector Databases — search at scale

Your Topic 48 exercise compared a query against 30 vectors by brute force — checking every single one. Fine at 30. At 10 million chunks × 1,536 dimensions, every query means ~15 billion multiplications, and your P99 latency dies. The fix is the same move databases made 50 years ago with B-trees: build an index so you don't look at everything.

For vectors, the indexes are ANN — approximate nearest neighbor: give up the guarantee of the exact top-k for a ~95–99% chance of it, and gain 100–1000× speed. (Recall the theme from speculative decoding and reranking-to-come: cheap-approximate wins when verified or when 98% is indistinguishable from 100% in product terms.)

HNSW — the algorithm you'll actually meet everywhere — in one analogy: navigating to a house via the road network. You don't check every house on Earth; you take the highway to roughly the right region, exit to city roads toward the right district, then walk the street to the door. HNSW builds exactly that: a multi-layer graph where the top layer has few nodes with long-range links (highways), each lower layer gets denser and more local (city roads → streets). A query enters at the top, greedily hops toward the target at each level, descends, and lands among the true nearest neighbors after touching a few hundred nodes instead of millions. Fast, high recall, the default index almost everywhere. (The other family, IVF, clusters the space into cells and searches only the nearest few cells — searching only the relevant library sections. Simpler, cheaper to build, common at huge scale.)

And quantization returns for its third appearance. 10M chunks × 1,536 floats × 4 bytes = ~61 GB of vectors — RAM is now your cost center. So vectors get quantized too: scalar (float32→int8, 4× smaller), product quantization, even binary (1 bit per dimension, 32× smaller, surprisingly usable when followed by exact rescoring of the shortlist). Topic 24's crayon boxes, drawing embeddings now.

What a vector database adds beyond the index — and this list is why "just use FAISS" (a library, not a database) stops sufficing: CRUD with persistence, metadata filtering (the crucial one: "nearest neighbors WHERE team='legal' AND date > 2025" — and note the subtlety that filtering during graph traversal is genuinely tricky, a real differentiator between products), hybrid search support (next topic), replication, and scaling.

The landscape, with the guidance that actually matters:

The guidance: below a few million vectors, pgvector in the Postgres you already run is the right answer — one system, real transactions, metadata filtering is just SQL WHERE. Since your stack is Supabase, you're one migration away from production vector search right now:

create table chunks (id bigserial primary key,
                     content text, metadata jsonb,
                     embedding vector(384));
create index on chunks using hnsw (embedding vector_cosine_ops);
 
select content from chunks
where metadata->>'team' = 'legal'
order by embedding <=> $query_vec limit 5;

Dedicated vector infrastructure earns its complexity at tens of millions of vectors, extreme QPS, or when its specialized features become load-bearing — not before.

Topic 51: Semantic Search — what embeddings miss, and the hybrid fix

Semantic search — embed query, find nearest chunks — has been our quiet workhorse since Lesson 2. Time to be honest about where it fails, because production quality lives in the failure modes:

1. Exact-identifier blindness. Embeddings match meaning, and identifiers barely have any. Search "error E-4032" and the nearest neighbors include E-4031 and E-4088 — semantically they're all just "error codes." Product SKUs, function names, ticket numbers, person names: the fuzzy-matching superpower becomes a liability precisely when the user knows exactly what they want.
2. Negation and numbers — Lesson 2's warning, still true: "flights under $200" and "flights over $200" embed uncomfortably close.
3. Question–answer asymmetry. "How do I reset my password?" and the doc paragraph that answers it share little surface form. Good retrieval embedders are trained on Q→A pairs to bridge this — and here's the practical gotcha that burns everyone once: many such models require role prefixes ("query: ..." vs "passage: ...") because they embed questions and documents into deliberately different regions. Omit the prefix and quality silently craters. Read your embedder's model card (Topic 43's skill, paying off).

Now meet the 50-year-old rival that happens to be strong exactly where embeddings are weak: BM25, the classic keyword algorithm behind traditional search engines. It scores documents by exact term matches, weighted so that rare words count heavily (matching "E-4032" is a jackpot; matching "the" is nothing) and diminishing returns apply to repetition. No neural anything — and it nails identifiers, names, and exact phrases every time they appear.

Two complementary specialists suggest the obvious architecture, and it is the production default — hybrid search: run both searches, then fuse the rankings. The standard fusion is RRF (Reciprocal Rank Fusion), almost embarrassingly simple: each document scores Σ 1/(60 + rank) across the lists it appears in. No score normalization, no tuning, robust — a document ranked well by either specialist surfaces, and one ranked well by both dominates. Every serious stack (including pgvector setups — Postgres has full-text search built in) supports this pattern.

Last practical layer — choosing the embedder: the MTEB leaderboard is the standard comparison (with the usual Topic-18 caveat about benchmark contamination); dimensions trade quality against storage/speed (your Topic 50 RAM math); Matryoshka embeddings are a lovely modern trick — trained so the first N dimensions form a valid smaller embedding, letting you truncate 1536→256 dims for a cheap first pass and rescore with full vectors. And the iron rule stands, third and final repetition: one embedding model per index. Changing embedders means re-embedding everything — a real migration cost to plan for, not discover.

Topic 52: Retrieval Pipelines — from demo to production

Naive RAG — embed, top-5, stuff, pray — demos beautifully and plateaus at "mediocre with confidence." Production quality comes from treating retrieval as a pipeline with stages, each fixing a specific failure. Here's the assembly, stage by stage:

Stage 1 — Query transformation (fixing the question before searching):

• Rewriting for standalone-ness: in a conversation, "what about the pricing?" retrieves nothing — the topic lives three turns back (Topic 5's whiteboard, biting again). A cheap LLM call rewrites it into "what is the pricing of [the thing from turn 2]" before search. Non-negotiable for chat products.
• HyDE (Hypothetical Document Embeddings) — a genuinely clever move against Topic 51's Q-A asymmetry: have the LLM hallucinate a plausible answer first, and embed that to search with. The fake answer is wrong on facts but right on shape — it lives in answer-space, near the real answers. Hallucination, weaponized for good.
• Decomposition: "Compare our refund policy with our exchange policy" → two sub-queries, retrieved separately, merged.

Stage 2 — Retrieval, wide: hybrid search (Topic 51) plus metadata filters, fetching generously — top 50–150, not top 5. Because the next stage exists:

Stage 3 — Reranking. The conceptual heart of the pipeline. Embedding search is a bi-encoder: query and document were each compressed into a vector alone, never having met — fast (vectors precomputed) but lossy (Lesson 2's compression, forever). A cross-encoder reads the query and document together through a full transformer — every token of one attending to every token of the other (Topic 7, doing what it does best) — producing a far more accurate relevance score. Too slow to run against millions; perfect against a hundred. So: cheap wide net, then expensive precise sieve —

Recognize the shape? It's the course's recurring master pattern — cheap-wide then expensive-precise — the same economics as speculative decoding's draft-and-verify and routing's cascade. Rerankers are available as small local models (BGE-reranker and friends) or APIs (Cohere, Voyage); adding one is typically the single largest retrieval-quality jump available for an afternoon of work.

Stage 4 — Assembly. The survivors become a prompt, and details matter: order the chunks with the best at the start and end (Topic 5's lost-in-the-middle, now an engineering instruction), deduplicate near-identical chunks (your corpus has them; Topic 18 told you so), label each with its source for citations, respect the token budget (Topic 38's prompt dieting — more chunks ≠ better answers), and close with the grounding instructions from Topic 48.

And the stage that separates professionals: measurement. The single most important habit in RAG engineering is debugging retrieval and generation separately — because if the right chunk never made it into the context, no prompt engineering on Earth can fix the answer, and you'll waste days tuning the wrong stage. Build a golden set: 30–50 questions each mapped to the chunk(s) that answer them (you know how to build datasets — that's Module 2 muscle). Then track:

• Recall@k — is a correct chunk anywhere in the top k? (The retrieval health metric; if this is low, nothing downstream matters.)
• MRR — how high does the first correct chunk rank?
• And on the generation side, the RAG triad: was the retrieved context relevant? Was the answer grounded in it (every claim supported — an LLM-judge task with receipts, per Topic 21 layer 4)? Did it actually answer the question?

Every knob from this module — chunk size, hybrid weights, reranker on/off, k — becomes tunable the moment these numbers exist, and folklore the moment they don't.

One forward pointer: everything above is a fixed pipeline. The next evolution hands the retrieval decisions to the model itself — deciding whether to search, what to search for, reading results, and searching again if unsatisfied. That's agentic RAG, and it's three topics away (Module 7), built from tool calling.

Topic 53: AI Memory Systems — the goldfish gets a past

Close your eyes and recall Topic 5: the model is stateless — a goldfish handed a transcript each time — and "memory" in chat products is software deciding what to paste onto the whiteboard. This final topic is that software, and here's the framing that makes it click: memory is RAG where the corpus is your own past. Same embeddings, same retrieval, same pipeline discipline — pointed at conversation history instead of documents.

The layered architecture every serious memory system converges on (with names borrowed from psychology, because they map perfectly):

1. Working memory — the context window itself: recent turns, verbatim. Free, perfect fidelity, brutally finite.
2. Rolling summarization — as old turns fall off the whiteboard, compress them into a running summary that stays on it. Trades fidelity for space; details blur, the gist survives. (Every long-chat product you've used does this — now you know why it remembers that you discussed deployment but not the exact port number.)
3. Episodic memory — store past conversation chunks in a vector index; retrieve by similarity when relevant. "What happened in that session about the Kafka bug" — searchable experience. Literally Topics 49–52 applied to transcripts.
4. Semantic memory — the distilled facts: "user is vegetarian," "project targets TypeScript," "prefers direct feedback." Extracted by an LLM as conversations happen (or in background batch jobs — Topic 36's other meaning, employed), stored structured, injected when relevant. Not "what was said" but "what is true."
5. Procedural memory — learned instructions: standing corrections and preferences that effectively edit the system prompt over time ("always use pnpm, never npm").

The reference architecture that named this space is MemGPT (now the Letta project), with a metaphor this course has made you fluent in: treat the context window as RAM and external storage as disk, and let the model itself page memories in and out using tools — the OS analogy from PagedAttention, promoted from cache management to cognition. Products like mem0 and Zep package these layers as services; ChatGPT's and Claude's memory features are in-house versions of the same stack.

Why it's genuinely hard — five problems, each with a course callback:

• Salience (what to write): most conversation is not worth remembering. Store everything and retrieval drowns in noise — Topic 18's curation problem, live and continuous. Every memory must earn its place.
• Contradiction (how to update): "I moved to Karachi" must supersede "lives in Lahore," not coexist with it. Naive append-only memory accumulates confident contradictions — and you know exactly what a model does with contradictory context.
• Forgetting (deliberately): stale facts decay in usefulness; good systems age memories out or down-weight them. Forgetting is a feature, not a failure.
• Retrieval relevance: injecting the wrong memory is worse than none — confusing at best, and at worst the product feels creepy ("why is it bringing that up?"). Precision matters more here than in document RAG, because errors are personal.
• Privacy: you're building a dossier. Scrub PII where appropriate (Topic 19's tooling), give users visibility and deletion, and treat the memory store with the sensitivity its contents deserve.

And the grounding truth to end on, which collapses all the architecture back to something you learned in week one: every memory system, however elaborate, terminates in the same act — choosing which tokens go on the whiteboard. Summaries, episodes, facts, procedures: all of it is prompt assembly. Memory architecture is context-window curation with a database attached.

Module 6: complete. The full grounding stack is yours: chunking that respects meaning, indexes that search millions in milliseconds, hybrid retrieval that catches both meaning and exact terms, pipelines with rerankers and golden-set metrics, and memory as RAG over the past. The model now has a library and a history — everything except the ability to act.

Next: Module 7 — Agents & Workflows. Prompt engineering done properly (the craft, systematized), system prompts, tool calling and function calling (how text generation becomes action — the mechanism underneath everything from Topic 52's agentic RAG to browser automation), agents and agentic workflows, multi-agent systems, and browser agents. This is the module where the compounding math from Topic 38 (0.95¹⁰ ≈ 60%) becomes the central engineering constraint — and where everything you've built starts doing things.