Context Windows — the model's working memory

Topic 5 of 90Module 1: Foundations3 min read

The context window is the maximum number of tokens a model can look at at once — your entire conversation, system prompt, documents, and the reply being generated, all combined.

If a model has a 128K context window, everything together must fit in 128,000 tokens (~96,000 words). Go past it and the oldest content falls off or the request fails.

The context window as a whiteboard: system prompt, older messages that may fall off, recent messages, and the reply being generated all have to fit inside it. The model is stateless — a goldfish handed the full transcript every message.
The context window as a whiteboard: system prompt, older messages that may fall off, recent messages, and the reply being generated all have to fit inside it. The model is stateless — a goldfish handed the full transcript every message.

Daily-life analogy: A whiteboard. The model can only reason about what's currently written on the whiteboard. It has vast learned knowledge in its knobs (like your long-term skills), but its awareness of the current situation is only the whiteboard. Fill it up, and to write more, something must be erased.

This explains behavior you've definitely seen:

  • Long chats where the model "forgets" the beginning → those messages fell off the whiteboard.
  • Chat apps feel like memory, but there is none. Every single message, the entire conversation is re-sent to the model from scratch. The model is stateless — a goldfish handed a transcript each time. "Memory" features are just software that saves notes and pastes them onto the whiteboard.

Three engineering realities:

  1. You pay for the whole whiteboard, every turn. A long conversation re-sends everything each message, so turn 50 costs far more than turn 1. This is why context management matters for any product's economics.
  2. Bigger window ≠ equally good everywhere. Models are often sharpest with information at the start and end of the context, weaker in the middle — known as the "lost in the middle" problem. Dumping a 300-page PDF in and expecting perfect recall of page 147 is optimistic.
  3. Context is a budget to design. Real products carefully decide what goes on the whiteboard: system prompt + relevant retrieved documents + recent messages + a summary of old messages. (This is the seed of RAG and memory systems — Module 6.)

Summary

The context window is the model's short-term working memory, measured in tokens. Everything the model can currently "see" must fit in it, and you pay for all of it every turn.

Mental model

A whiteboard of fixed size. Learned knowledge lives in the knobs; situational awareness lives only on the board.

Mistakes to avoid

  • Believing the model remembers previous chats natively. It doesn't. Software around it creates that illusion.
  • Stuffing maximum context "because it fits." More irrelevant text = higher cost, slower responses, and often worse answers because the important stuff gets diluted.

Exercise

Have a long chat with any model (30+ messages). Early on, tell it a fake fact: "My cat's name is Bolt." Keep chatting about other things, then much later ask the cat's name. Then try the same in a very short chat. You're empirically probing context retention — a real evaluation skill.

That's Lesson 1 — you now understand what an LLM is, how it learns, what it reads, and what it can "see." These five concepts are load-bearing walls; everything else in the curriculum stands on them.

Lesson 2 covers: Embeddings, Transformers, Attention, Parameters, Training vs Inference, Open vs Closed models — that completes Foundations, and honestly, attention is the most beautiful idea in the whole field.

Before we continue, do the tiktoken exercise from Topic 4 if you do only one — it takes 10 minutes and turns tokens from theory into something you've touched.