Loading lesson…
LLM Foundations / From Text to Tokens
How text becomes tokens.
Reviewed by Yuvaraj
A language model does not read letters, and it does not quite read words. It reads tokens. Getting comfortable with what a token is explains a lot of otherwise-mysterious model behavior.
The practical compromise is the subword token: common words become a single token, while rare or novel words break into a few meaningful pieces. This keeps the vocabulary a fixed, manageable size while still being able to represent text.
Answer from memory before revealing, retrieval practice is what builds durable recall.
Tokenization is the step that:
Every common English word is always exactly one token.
In modern LLMs, a token is usually:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
A tokenizer is built once, before training, by scanning a huge amount of text and finding which character sequences are common enough to deserve their own token. A widely used method is Byte-Pair Encoding (BPE), which repeatedly merges the most frequent adjacent pair.
Start with every character as its own token, then merge the most frequent adjacent pair, again and again. Suppose the words low, lower, and lowest appear often. Beginning from characters:
l + o is very frequent → merge it into lo.lo + w is frequent → merge into low.low is now a single token. lower becomes low + er; lowest becomes low + est.Common whole words collapse to one token; rarer words reuse those pieces. A brand-new word the tokenizer never saw still works, it simply falls back to smaller pieces, in the worst case single characters. That is why a subword vocabulary can represent any text with a fixed vocabulary size.
The result is a fixed vocabulary, often tens of thousands of tokens. Every token has an integer id, and the model only ever sees the integers:
the, learning.tokenization might become token + ization."dog" and " dog" (with a leading space) are often different tokens.A token is not a word
Token counts drive context limits and cost, and they are not word counts. Code, math, emoji, and non-English text often use many more tokens per word, sometimes several tokens per character, which is one reason the same idea can be "cheaper" in one language than another.
A quick worked estimate: a 1,000-word English article is roughly 1,300 tokens. At an example price of 0.50 USD per million input tokens, reading it costs about 0.00065 USD, trivial. But the same meaning written in a language the tokenizer handles poorly can take three to four times as many tokens, and therefore three to four times the cost and context space, for identical content. Tokenization is not a formatting detail; it is a budget.