Tokenization — how the chunks get chosen

Topic 4 of 90Module 1: Foundations2 min read

So who decides that "unbelievable" splits into un/believ/able? A separate small program called the tokenizer, built before the model is trained.

The most common method is BPE (Byte Pair Encoding), and the idea is lovely:

  1. Start with the smallest possible pieces (individual characters/bytes)
  2. Scan a huge pile of text and find the pair of pieces that appears together most often — say t + h
  3. Merge them into a new single piece: th
  4. Repeat: maybe th + e → the. Maybe in + g → ing
  5. Stop when you've built a vocabulary of the desired size (say 100,000 pieces)
BPE merging frequent pairs step by step: characters t-h-e merge to th+e, then to the single token “the” — earned through frequency, the same way “lol” or “asap” became shortcuts. The tokenizer is trained once, then frozen forever.
BPE merging frequent pairs step by step: characters t-h-e merge to th+e, then to the single token “the” — earned through frequency, the same way “lol” or “asap” became shortcuts. The tokenizer is trained once, then frozen forever.

The result: frequent text becomes single tokens automatically. Statistics decides the vocabulary, not a human with a dictionary.

Daily-life analogy: Texting shortcuts evolving naturally. People typed "as soon as possible" so often it compressed to "asap." "Laughing out loud" → "lol." BPE does exactly this: whatever appears often gets its own shortcut.

Two practical consequences worth engraving in your brain:

1. The tokenizer is frozen forever. Once the model is trained, the tokenizer can't change — the model's entire understanding is built on those specific chunks. This is why every model family (Llama, GPT, Qwen) has its own tokenizer, and token counts differ between them for the same text.

2. Tokenizer quality shapes model behavior. If the tokenizer's training text had little Urdu, Urdu gets shredded into tiny pieces → longer sequences → worse and more expensive Urdu performance. Same for niche programming languages. When you hear "this model is good at code," part of the answer is: its tokenizer treats code kindly.

Tiny code taste (Python):

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(enc.encode("Hello Lahore!"))   # → [9906, 445, 1494, 461, 0]  (token IDs)
print(enc.decode([9906]))            # → "Hello"

Those numbers are token IDs — each chunk's index in the vocabulary. This list of numbers is what the model actually receives. Not text. Ever.

Summary

Tokenization converts text into token IDs using a vocabulary learned by merging frequent character pairs (BPE). It's fixed per model and quietly shapes cost, speed, and quality.

Mental model

A shortcut dictionary that evolved from usage frequency, like "lol" and "asap" emerging from repetition.

Mistakes to avoid

  • Comparing token counts across different model families as if they're the same currency. GPT tokens ≠ Llama tokens.
  • Ignoring tokenization when a model behaves weirdly with numbers, rare names, or non-English text. Check the tokenizer first — it's often the culprit.

Exercise

In Python, pip install tiktoken, then write a 10-line script that takes any text file and reports: word count, token count, and the token-to-word ratio. Run it on English text, code, and Urdu text. You now have a real cost-estimation tool — genuinely useful for pricing any AI product you build.