Tokens — the model's alphabet

Topic 3 of 90Module 1: Foundations2 min read

Here's something surprising: models don't read words. They don't read letters either. They read tokens — chunks of text that sit somewhere between letters and words.

"unbelievable" might be split into: un + believ + able — 3 tokens. "the" is common enough to be 1 token. "Lahore" might be 2 tokens; a rare word like "supercalifragilistic" might be 6.

Rough rule for English: 1 token ≈ ¾ of a word, or ~4 characters. 100 words ≈ 130 tokens.

Why not just use whole words? Two problems. There are millions of words (including typos, names, new slang) — the model's "vocabulary" would be gigantic and it could never handle a word it hasn't seen. Why not just letters? Then "understanding" becomes 13 pieces, sequences get super long, and the model wastes capacity re-learning basic spelling.

Tokens are the compromise: a fixed vocabulary of ~50,000–250,000 chunks that can build any text, common words as single chunks, rare words assembled from pieces.

“unbelievable” splits into three tokens — un, believ, able — while a common word like “the” is a single token. Rule of thumb: 1 token ≈ 0.75 words ≈ 4 characters, and the model only ever sees token IDs, never letters.
“unbelievable” splits into three tokens — un, believ, able — while a common word like “the” is a single token. Rule of thumb: 1 token ≈ 0.75 words ≈ 4 characters, and the model only ever sees token IDs, never letters.

Daily-life analogy: LEGO. You don't get a pre-made "castle" piece (whole words) and you don't build from individual atoms of plastic (letters). You get a fixed set of standard bricks. Common shapes are single bricks; unusual shapes you assemble from smaller ones.

Why you should deeply care as an engineer:

  • APIs charge per token. Cost analysis = token analysis.
  • Context limits are in tokens. "8K context" means 8,000 tokens, ~6,000 words.
  • Weird model failures come from tokenization. Famous example: models struggling to count letters in "strawberry" — because the model never sees letters, it sees tokens like straw + berry. Asking it to count r's is like asking you to count atoms in a LEGO brick.
  • Non-English text often uses more tokens per word — Urdu text costs you more per sentence than English. Relevant if you build for Pakistani users.

Summary

Tokens are the chunks models actually read — bigger than letters, usually smaller than words, from a fixed vocabulary.

Mental model

LEGO bricks for text.

Mistakes to avoid

  • Assuming 1 word = 1 token when estimating API costs (you'll underestimate by ~30%).
  • Being surprised when models fail at spelling/counting-letter tasks. It's not stupidity, it's blindness — they literally can't see letters.

Exercise

Go to a tokenizer playground (search "OpenAI tokenizer" or use the tiktoken library in Python). Paste in: your name, an English paragraph, an Urdu paragraph, and some TypeScript code. Compare token counts. Notice how code and non-English text tokenize differently.