Retrieval Pipelines — from demo to production

Naive RAG — embed, top-5, stuff, pray — demos beautifully and plateaus at "mediocre with confidence." Production quality comes from treating retrieval as a pipeline with stages, each fixing a specific failure. Here's the assembly, stage by stage:

Stage 1 — Query transformation (fixing the question before searching):

• Rewriting for standalone-ness: in a conversation, "what about the pricing?" retrieves nothing — the topic lives three turns back (Topic 5's whiteboard, biting again). A cheap LLM call rewrites it into "what is the pricing of [the thing from turn 2]" before search. Non-negotiable for chat products.
• HyDE (Hypothetical Document Embeddings) — a genuinely clever move against Topic 51's Q-A asymmetry: have the LLM hallucinate a plausible answer first, and embed that to search with. The fake answer is wrong on facts but right on shape — it lives in answer-space, near the real answers. Hallucination, weaponized for good.
• Decomposition: "Compare our refund policy with our exchange policy" → two sub-queries, retrieved separately, merged.

Stage 2 — Retrieval, wide: hybrid search (Topic 51) plus metadata filters, fetching generously — top 50–150, not top 5. Because the next stage exists:

Stage 3 — Reranking. The conceptual heart of the pipeline. Embedding search is a bi-encoder: query and document were each compressed into a vector alone, never having met — fast (vectors precomputed) but lossy (Lesson 2's compression, forever). A cross-encoder reads the query and document together through a full transformer — every token of one attending to every token of the other (Topic 7, doing what it does best) — producing a far more accurate relevance score. Too slow to run against millions; perfect against a hundred. So: cheap wide net, then expensive precise sieve —

Recognize the shape? It's the course's recurring master pattern — cheap-wide then expensive-precise — the same economics as speculative decoding's draft-and-verify and routing's cascade. Rerankers are available as small local models (BGE-reranker and friends) or APIs (Cohere, Voyage); adding one is typically the single largest retrieval-quality jump available for an afternoon of work.

Stage 4 — Assembly. The survivors become a prompt, and details matter: order the chunks with the best at the start and end (Topic 5's lost-in-the-middle, now an engineering instruction), deduplicate near-identical chunks (your corpus has them; Topic 18 told you so), label each with its source for citations, respect the token budget (Topic 38's prompt dieting — more chunks ≠ better answers), and close with the grounding instructions from Topic 48.

And the stage that separates professionals: measurement. The single most important habit in RAG engineering is debugging retrieval and generation separately — because if the right chunk never made it into the context, no prompt engineering on Earth can fix the answer, and you'll waste days tuning the wrong stage. Build a golden set: 30–50 questions each mapped to the chunk(s) that answer them (you know how to build datasets — that's Module 2 muscle). Then track:

• Recall@k — is a correct chunk anywhere in the top k? (The retrieval health metric; if this is low, nothing downstream matters.)
• MRR — how high does the first correct chunk rank?
• And on the generation side, the RAG triad: was the retrieved context relevant? Was the answer grounded in it (every claim supported — an LLM-judge task with receipts, per Topic 21 layer 4)? Did it actually answer the question?

Every knob from this module — chunk size, hybrid weights, reranker on/off, k — becomes tunable the moment these numbers exist, and folklore the moment they don't.

One forward pointer: everything above is a fixed pipeline. The next evolution hands the retrieval decisions to the model itself — deciding whether to search, what to search for, reading results, and searching again if unsatisfied. That's agentic RAG, and it's three topics away (Module 7), built from tool calling.

---