Loading lesson…
Transformers / Inputs
From text to vectors a model can process.
Reviewed by Yuvaraj
A transformer never reads your text. It reads numbers. Before a single attention head runs, a short and fully deterministic pipeline converts a human string into a grid of floating-point vectors, one vector per token, and those vectors are the only thing the rest of the network ever touches. Understanding this front door matters: almost every "why did the model do that?" question about context length, rare words, or token cost traces straight back to how text is tokenized and embedded.
Modern language models do not split on whole words or single characters. They use subword tokenization (algorithms such as Byte-Pair Encoding or WordPiece) that break text into frequently occurring chunks. Common words map to a single token; rare words fragment into pieces. This keeps the vocabulary at a fixed, manageable size (typically 30k–130k entries) while still being able to represent any string, including typos and names it has never seen.
For example, "tokenization" might split into ["token", "ization"], while "ai" stays whole.
The vocabulary is just an ordered list, so every token has a fixed integer index, its token ID. Tokenizing a string produces a list of these IDs. The IDs are arbitrary labels: ID 12 is not "bigger" or "later" than ID 5 in any meaningful sense; it is only an address.
Here is the core mechanism. The model stores a learned embedding matrix , where is the vocabulary size and is the model's hidden dimension (e.g. 768 or 4096). Turning an ID into a vector is a pure row lookup:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
That lookup is mathematically identical to multiplying a one-hot vector by , which is why an embedding layer is sometimes drawn as a matrix multiply.
Because is trained by gradient descent alongside everything else, the model is free to place tokens that behave similarly close together in the -dimensional space. "learns", "studies", and "trains" drift toward nearby vectors; "learns" and "granite" do not. That geometric closeness is the meaning the network gets to exploit.
Take a toy model with hidden dimension and the string "ai learns".
With a tiny vocabulary and embedding matrix, the lookup is concrete:
| token | id | embedding (d = 4) |
|---|---|---|
ai | 5 | [ 0.21, -0.83, 0.05, 0.44] |
learns | 12 | [-0.11, 0.67, -0.52, 0.30] |
In code the whole pipeline is two lines:
ids = tokenizer.encode("ai learns") # -> [5, 12]
vectors = embedding_matrix[ids] # -> shape (2, 4)
The output is a matrix: two tokens, four dimensions each. A real model produces an matrix for tokens, with in the hundreds or thousands.
Those per-token embeddings, once a positional signal is added so word order is not lost, are exactly the input to the first transformer block. The query, key, and value vectors inside self-attention are all linear projections of these embeddings. Attention never operates on text or on IDs; it operates entirely on this learned vector space. Get the embedding step right and everything downstream has meaningful material to work with.
Common mistakes