Loading lesson…
Natural Language Processing / Foundations
Turning raw text into model input.
Reviewed by Yuvaraj
A language model never sees your text. It sees numbers, sequences of integer IDs that index a learned table of vectors. Text preprocessing is the pipeline that turns a raw string such as "Don't tokenize naïvely!" into that sequence, and every step shapes what the model can represent. This lesson walks through the modern tokenization pipeline and the classic preprocessing it replaced, so you know which steps still earn their place and which are historical baggage.
Turning text into model input is a short, ordered pipeline. Each stage is deterministic, and the whole thing is reversible enough that you can decode IDs back into readable text.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Normalization makes superficially different strings comparable. Unicode normalization (NFC or NFKC) is the important one: the character é can be stored as a single code point or as e plus a combining accent, and only after normalizing do those hash to the same token. Classic pipelines also lowercased aggressively. Modern subword tokenizers usually keep case, because capitalization carries meaning the model can use.
Tokenization splits normalized text into units. Word-level tokenizers split on whitespace and punctuation, so the vocabulary is the set of observed words, intuitive, but it explodes in size, and any word unseen during training is out-of-vocabulary (OOV).
Subword tokenizers (BPE, WordPiece, Unigram) instead start from characters or raw bytes and merge frequent adjacent pairs into larger units. A fixed number of merges produces a fixed vocabulary of size (commonly to tokens). Frequent words become single tokens; rare words split into reusable pieces.
The vocabulary is a fixed lookup table mapping each token string to an integer ID, plus a few reserved special tokens. Word-level models handle unknown input with a single <unk> token, which throws away information. Byte-level BPE has no OOV at all: because every byte is in the base vocabulary, any string, an emoji, a code snippet, a new brand name, is representable.
Take the sentence Tokenization isn't magic. and run it through both approaches. The IDs below are illustrative, not from any specific model.
text = "Tokenization isn't magic."
# Word-level: split on whitespace/punctuation, then map through a fixed vocab
word_tokens = ["tokenization", "isn", "'", "t", "magic", "."]
word_ids = ["<unk>", 512, 8, 40, 733, 5] # "tokenization" unseen -> <unk>
# Subword (byte-level BPE): learned merges, leading spaces kept, no OOV
sub_tokens = ["Token", "ization", " isn", "'t", " magic", "."]
sub_ids = [30642, 1634, 2762, 470, 5536, 13] # illustrative IDs
| Stage | Word-level | Subword (byte-level BPE) |
|---|---|---|
| Tokens | tokenization, isn, ', t, magic, . | Token, ization, " isn", 't, " magic", . |
| Handles "tokenization"? | Miss → <unk> | Split into known pieces |
| True OOV possible? | Yes | No |
The word-level tokenizer never saw tokenization, so it emits <unk> and the model is blind to the entire word. The subword tokenizer decomposes it into Token + ization, pieces it already knows, and loses nothing.
Three classic steps still appear in older tutorials:
the, is, of) to cut noise.running → run, studies → studi), fast but lossy, and it produces non-words.better → good, running → run).Modern LLMs skip all three. Removing stop words destroys the syntax a transformer depends on, and subword vocabularies already share pieces across related word forms. These techniques still shine in classical NLP: TF-IDF search, topic modeling, and lightweight text classifiers.
Common mistakes