AI Memory Systems — the goldfish gets a past

Topic 53 of 90Module 6: RAG & Memory5 min read

Close your eyes and recall Topic 5: the model is stateless — a goldfish handed a transcript each time — and "memory" in chat products is software deciding what to paste onto the whiteboard. This final topic is that software, and here's the framing that makes it click: memory is RAG where the corpus is your own past. Same embeddings, same retrieval, same pipeline discipline — pointed at conversation history instead of documents.

The layered architecture every serious memory system converges on (with names borrowed from psychology, because they map perfectly):

  1. Working memory — the context window itself: recent turns, verbatim. Free, perfect fidelity, brutally finite.
  2. Rolling summarization — as old turns fall off the whiteboard, compress them into a running summary that stays on it. Trades fidelity for space; details blur, the gist survives. (Every long-chat product you've used does this — now you know why it remembers that you discussed deployment but not the exact port number.)
  3. Episodic memory — store past conversation chunks in a vector index; retrieve by similarity when relevant. "What happened in that session about the Kafka bug" — searchable experience. Literally Topics 49–52 applied to transcripts.
  4. Semantic memory — the distilled facts: "user is vegetarian," "project targets TypeScript," "prefers direct feedback." Extracted by an LLM as conversations happen (or in background batch jobs — Topic 36's other meaning, employed), stored structured, injected when relevant. Not "what was said" but "what is true."
  5. Procedural memory — learned instructions: standing corrections and preferences that effectively edit the system prompt over time ("always use pnpm, never npm").

The reference architecture that named this space is MemGPT (now the Letta project), with a metaphor this course has made you fluent in: treat the context window as RAM and external storage as disk, and let the model itself page memories in and out using tools — the OS analogy from PagedAttention, promoted from cache management to cognition. Products like mem0 and Zep package these layers as services; ChatGPT's and Claude's memory features are in-house versions of the same stack.

Why it's genuinely hard — five problems, each with a course callback:

  • Salience (what to write): most conversation is not worth remembering. Store everything and retrieval drowns in noise — Topic 18's curation problem, live and continuous. Every memory must earn its place.
  • Contradiction (how to update): "I moved to Karachi" must supersede "lives in Lahore," not coexist with it. Naive append-only memory accumulates confident contradictions — and you know exactly what a model does with contradictory context.
  • Forgetting (deliberately): stale facts decay in usefulness; good systems age memories out or down-weight them. Forgetting is a feature, not a failure.
  • Retrieval relevance: injecting the wrong memory is worse than none — confusing at best, and at worst the product feels creepy ("why is it bringing that up?"). Precision matters more here than in document RAG, because errors are personal.
  • Privacy: you're building a dossier. Scrub PII where appropriate (Topic 19's tooling), give users visibility and deletion, and treat the memory store with the sensitivity its contents deserve.

And the grounding truth to end on, which collapses all the architecture back to something you learned in week one: every memory system, however elaborate, terminates in the same act — choosing which tokens go on the whiteboard. Summaries, episodes, facts, procedures: all of it is prompt assembly. Memory architecture is context-window curation with a database attached.

Summary

Memory = RAG over your own past, layered: verbatim recent turns, rolling summaries, retrievable episodes, distilled facts, standing instructions. The hard parts are salience, contradiction handling, deliberate forgetting, retrieval precision, and privacy — and it all compiles down to prompt assembly.

Mental model

A great executive assistant: keeps minutes (episodic), maintains a client profile card that gets corrected, not appended (semantic), updates standing instructions (procedural) — and, crucially, knows what not to bring up in a meeting (salience and relevance).

Mistakes to avoid

  • Storing raw transcripts as "memory" and retrieving big blobs of them. Distillation is the work — undigested history is noise with a vector index.
  • Append-only facts. Without supersede-and-resolve logic, six months of usage produces a profile that disagrees with itself, and your model will confidently pick the wrong version.

Exercise · module capstone

Add memory to your Topic 48 system. After each Q&A exchange, run one extraction prompt: "List any durable facts about the user or their project from this exchange, or NONE." Store extractions with embeddings in a second pgvector table (or list); on each new question, retrieve the top-2 relevant memories and add them to the prompt under a Known context: header. Have a 6–8 turn conversation that establishes a fact early ("my API is in TypeScript") and implicitly relies on it later ("write me the client snippet"). Then break it: tell it you've migrated to Python, and check whether your system supersedes or contradicts. Congratulations — you've built, and stress-tested, every layer of this topic in miniature.

Module 6: complete. The full grounding stack is yours: chunking that respects meaning, indexes that search millions in milliseconds, hybrid retrieval that catches both meaning and exact terms, pipelines with rerankers and golden-set metrics, and memory as RAG over the past. The model now has a library and a history — everything except the ability to act.

Next: Module 7 — Agents & Workflows. Prompt engineering done properly (the craft, systematized), system prompts, tool calling and function calling (how text generation becomes action — the mechanism underneath everything from Topic 52's agentic RAG to browser automation), agents and agentic workflows, multi-agent systems, and browser agents. This is the module where the compounding math from Topic 38 (0.95¹⁰ ≈ 60%) becomes the central engineering constraint — and where everything you've built starts doing things.