All termsTry it in the Tokenizer Playground labExplore in the AI Universe
Glossary · Term
Token
L2 · IntermediateEvolving
In plain terms
The unit of text a model actually reads.
Technical definition
A subword unit produced by a tokenizer and mapped to an integer id before embedding.