Lab
Tokenizer Playground
See how text becomes tokens and token IDs.
Language models never see raw text, they see tokens, discrete pieces each mapped to an integer id. Type below and watch the split change. Switch between an approximate word-level view and a raw character view.
Each coloured chip is one token (4 non-space here); hover to see its illustrative id. Notice that common words become a single token while spaces and punctuation are separate.
Note: ids are stable illustrative hashes, not a specific model's vocabulary. Real tokenizers learn subword merges (BPE) from data, so a rare word may split into several tokens and the id maps into a trained vocabulary.
Challenge
Find a long or unusual word. In a real BPE tokenizer it would split into several subword tokens, how might that change the token count compared with this word-level approximation?