Vector Databases — search at scale

Your Topic 48 exercise compared a query against 30 vectors by brute force — checking every single one. Fine at 30. At 10 million chunks × 1,536 dimensions, every query means ~15 billion multiplications, and your P99 latency dies. The fix is the same move databases made 50 years ago with B-trees: build an index so you don't look at everything.

For vectors, the indexes are ANN — approximate nearest neighbor: give up the guarantee of the exact top-k for a ~95–99% chance of it, and gain 100–1000× speed. (Recall the theme from speculative decoding and reranking-to-come: cheap-approximate wins when verified or when 98% is indistinguishable from 100% in product terms.)

HNSW — the algorithm you'll actually meet everywhere — in one analogy: navigating to a house via the road network. You don't check every house on Earth; you take the highway to roughly the right region, exit to city roads toward the right district, then walk the street to the door. HNSW builds exactly that: a multi-layer graph where the top layer has few nodes with long-range links (highways), each lower layer gets denser and more local (city roads → streets). A query enters at the top, greedily hops toward the target at each level, descends, and lands among the true nearest neighbors after touching a few hundred nodes instead of millions. Fast, high recall, the default index almost everywhere. (The other family, IVF, clusters the space into cells and searches only the nearest few cells — searching only the relevant library sections. Simpler, cheaper to build, common at huge scale.)

And quantization returns for its third appearance. 10M chunks × 1,536 floats × 4 bytes = ~61 GB of vectors — RAM is now your cost center. So vectors get quantized too: scalar (float32→int8, 4× smaller), product quantization, even binary (1 bit per dimension, 32× smaller, surprisingly usable when followed by exact rescoring of the shortlist). Topic 24's crayon boxes, drawing embeddings now.

What a vector database adds beyond the index — and this list is why "just use FAISS" (a library, not a database) stops sufficing: CRUD with persistence, metadata filtering (the crucial one: "nearest neighbors WHERE team='legal' AND date > 2025" — and note the subtlety that filtering during graph traversal is genuinely tricky, a real differentiator between products), hybrid search support (next topic), replication, and scaling.

The landscape, with the guidance that actually matters:

The guidance: below a few million vectors, pgvector in the Postgres you already run is the right answer — one system, real transactions, metadata filtering is just SQL WHERE. Since your stack is Supabase, you're one migration away from production vector search right now:

create table chunks (id bigserial primary key,
                     content text, metadata jsonb,
                     embedding vector(384));
create index on chunks using hnsw (embedding vector_cosine_ops);
 
select content from chunks
where metadata->>'team' = 'legal'
order by embedding <=> $query_vec limit 5;

Dedicated vector infrastructure earns its complexity at tens of millions of vectors, extreme QPS, or when its specialized features become load-bearing — not before.