Lesson 1What an LLM Actually Is

Welcome to Module 1: Foundations. Five topics, and they are load-bearing walls — what an LLM is, how it learns, what it actually reads, and what it can see at any moment. Everything in the sixteen lessons after this one stands on them, so this lesson is worth reading slowly even if parts feel familiar.

Topic 1: LLM Basics — the autocomplete that ate the internet

Forget "artificial intelligence" for a second. An LLM (Large Language Model) is, at its core, a very powerful autocomplete.

You've used autocomplete on your phone. You type "I'll call you" and it suggests "later." It learned that from patterns in text. An LLM is the same idea, scaled up insanely: instead of learning from your texting history, it learned from trillions of words — books, websites, code, Wikipedia, conversations.

The core job of an LLM is stupidly simple: given some text, predict the next word.

That's it. That's the whole trick.

"The capital of Pakistan is ___" → the model predicts "Islamabad" because in its training data, that sentence pattern almost always ended that way.

The magic is what emerges when you do this prediction extremely well. To predict the next word in "The lawyer argued that the contract was ___", the model has to implicitly understand law, grammar, logic, and context. Good prediction forces understanding. That's the deep insight of the whole field.

Daily-life analogy: Imagine someone who has read every book in the world's biggest library, millions of times. They don't have the books with them anymore — but they absorbed the patterns. Ask them anything, and they complete your sentence based on everything they've absorbed. Sometimes they're brilliant. Sometimes they confidently make things up, because they're pattern-matching, not looking things up.

Difficult word explained: "Model" just means a mathematical system that takes an input and produces an output. Like a formula, but with billions of adjustable knobs instead of two or three.

Topic 2: How AI Models Work — knobs, guesses, and corrections

How does a model learn to predict the next word? Three ideas:

Idea 1: Everything is numbers. Computers can't process words. So words get converted into numbers (we'll cover exactly how in Tokens/Embeddings). The model is math operating on numbers.

Idea 2: The model is billions of adjustable knobs. These knobs are called parameters (we'll go deeper later). Think of a giant sound mixing board with 7 billion sliders. The position of all sliders together determines what output you get for any input.

Idea 3: Training = adjusting knobs by making mistakes. Here's the loop, and it's beautifully simple:

1. Take a sentence from the training data: "The cat sat on the mat"
2. Hide the last word: "The cat sat on the ___"
3. Ask the model to guess. Early in training it guesses garbage: "banana"
4. Compare guess to truth. Measure how wrong it was (this measurement is called loss — literally a "wrongness score")
5. Nudge every knob slightly in the direction that would have made the guess less wrong
6. Repeat trillions of times

Daily-life analogy: Learning to cook biryani by taste. First attempt: too salty. You don't know exactly which of the 20 ingredients caused it, but you adjust several slightly. Next attempt: better. Repeat hundreds of times and eventually your hands "know" the recipe. The model does this, except with billions of ingredients and trillions of attempts. The math that figures out which knob to nudge and how much is called backpropagation — you don't need to know its internals yet, just that it's the "blame assignment" mechanism.

Now, why did your exercise in Topic 1 give different answers each time? Because the model doesn't output one word — it outputs a probability for every possible next word: "later" 40%, "soon" 25%, "tomorrow" 15%... Then it samples from those probabilities, like a weighted dice roll. The setting that controls how adventurous this dice roll is is called temperature: low temperature = always pick the top choice (predictable), high temperature = take risks (creative, but more errors).

Topic 3: Tokens — the model's alphabet

Here's something surprising: models don't read words. They don't read letters either. They read tokens — chunks of text that sit somewhere between letters and words.

"unbelievable" might be split into: un + believ + able — 3 tokens. "the" is common enough to be 1 token. "Lahore" might be 2 tokens; a rare word like "supercalifragilistic" might be 6.

Rough rule for English: 1 token ≈ ¾ of a word, or ~4 characters. 100 words ≈ 130 tokens.

Why not just use whole words? Two problems. There are millions of words (including typos, names, new slang) — the model's "vocabulary" would be gigantic and it could never handle a word it hasn't seen. Why not just letters? Then "understanding" becomes 13 pieces, sequences get super long, and the model wastes capacity re-learning basic spelling.

Tokens are the compromise: a fixed vocabulary of ~50,000–250,000 chunks that can build any text, common words as single chunks, rare words assembled from pieces.

Daily-life analogy: LEGO. You don't get a pre-made "castle" piece (whole words) and you don't build from individual atoms of plastic (letters). You get a fixed set of standard bricks. Common shapes are single bricks; unusual shapes you assemble from smaller ones.

Why you should deeply care as an engineer:

• APIs charge per token. Cost analysis = token analysis.
• Context limits are in tokens. "8K context" means 8,000 tokens, ~6,000 words.
• Weird model failures come from tokenization. Famous example: models struggling to count letters in "strawberry" — because the model never sees letters, it sees tokens like straw + berry. Asking it to count r's is like asking you to count atoms in a LEGO brick.
• Non-English text often uses more tokens per word — Urdu text costs you more per sentence than English. Relevant if you build for Pakistani users.

Topic 4: Tokenization — how the chunks get chosen

So who decides that "unbelievable" splits into un/believ/able? A separate small program called the tokenizer, built before the model is trained.

The most common method is BPE (Byte Pair Encoding), and the idea is lovely:

1. Start with the smallest possible pieces (individual characters/bytes)
2. Scan a huge pile of text and find the pair of pieces that appears together most often — say t + h
3. Merge them into a new single piece: th
4. Repeat: maybe th + e → the. Maybe in + g → ing
5. Stop when you've built a vocabulary of the desired size (say 100,000 pieces)

The result: frequent text becomes single tokens automatically. Statistics decides the vocabulary, not a human with a dictionary.

Daily-life analogy: Texting shortcuts evolving naturally. People typed "as soon as possible" so often it compressed to "asap." "Laughing out loud" → "lol." BPE does exactly this: whatever appears often gets its own shortcut.

Two practical consequences worth engraving in your brain:

1. The tokenizer is frozen forever. Once the model is trained, the tokenizer can't change — the model's entire understanding is built on those specific chunks. This is why every model family (Llama, GPT, Qwen) has its own tokenizer, and token counts differ between them for the same text.

2. Tokenizer quality shapes model behavior. If the tokenizer's training text had little Urdu, Urdu gets shredded into tiny pieces → longer sequences → worse and more expensive Urdu performance. Same for niche programming languages. When you hear "this model is good at code," part of the answer is: its tokenizer treats code kindly.

Tiny code taste (Python):

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(enc.encode("Hello Lahore!"))   # → [9906, 445, 1494, 461, 0]  (token IDs)
print(enc.decode([9906]))            # → "Hello"

Those numbers are token IDs — each chunk's index in the vocabulary. This list of numbers is what the model actually receives. Not text. Ever.

Topic 5: Context Windows — the model's working memory

The context window is the maximum number of tokens a model can look at at once — your entire conversation, system prompt, documents, and the reply being generated, all combined.

If a model has a 128K context window, everything together must fit in 128,000 tokens (~96,000 words). Go past it and the oldest content falls off or the request fails.

Daily-life analogy: A whiteboard. The model can only reason about what's currently written on the whiteboard. It has vast learned knowledge in its knobs (like your long-term skills), but its awareness of the current situation is only the whiteboard. Fill it up, and to write more, something must be erased.

This explains behavior you've definitely seen:

• Long chats where the model "forgets" the beginning → those messages fell off the whiteboard.
• Chat apps feel like memory, but there is none. Every single message, the entire conversation is re-sent to the model from scratch. The model is stateless — a goldfish handed a transcript each time. "Memory" features are just software that saves notes and pastes them onto the whiteboard.

Three engineering realities:

1. You pay for the whole whiteboard, every turn. A long conversation re-sends everything each message, so turn 50 costs far more than turn 1. This is why context management matters for any product's economics.
2. Bigger window ≠ equally good everywhere. Models are often sharpest with information at the start and end of the context, weaker in the middle — known as the "lost in the middle" problem. Dumping a 300-page PDF in and expecting perfect recall of page 147 is optimistic.
3. Context is a budget to design. Real products carefully decide what goes on the whiteboard: system prompt + relevant retrieved documents + recent messages + a summary of old messages. (This is the seed of RAG and memory systems — Module 6.)

That's Lesson 1 — you now understand what an LLM is, how it learns, what it reads, and what it can "see." These five concepts are load-bearing walls; everything else in the curriculum stands on them.

Lesson 2 covers: Embeddings, Transformers, Attention, Parameters, Training vs Inference, Open vs Closed models — that completes Foundations, and honestly, attention is the most beautiful idea in the whole field.

Before we continue, do the tiktoken exercise from Topic 4 if you do only one — it takes 10 minutes and turns tokens from theory into something you've touched.