Loading lesson…
Transformers / Inputs
Giving order to a set-blind mechanism.
Reviewed by Yuvaraj
Self-attention has a surprising blind spot: on its own, it cannot tell where a token sits in a sequence. Each output is a weighted sum over all other tokens, and those weights depend only on content, how similar the query and key vectors are, never on order. Permute the input tokens and the outputs permute identically (attention is permutation-equivariant); the mechanism genuinely treats its input as a set, not a sequence. That means "the cat sat" and "sat cat the" are indistinguishable to raw attention. Positional encoding fixes this by injecting a signal that tells the model which slot each token occupies.
Because attention scores come from dot products of content vectors alone, nothing in the formula references the index 0, 1, 2, and so on. Word order carries most of the meaning in language, so we add position information before attention ever runs. The only open design questions are how to represent position and where to inject it.
The original Transformer used a fixed, non-learned scheme. For model width , position , and dimension index :
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Each pair of dimensions is a sinusoid, and the divisor stretches the wavelength geometrically, fast-oscillating in low dimensions, slow in high ones. Every position gets a unique fingerprint, and because sines and cosines relate positions through fixed rotations, relative offsets stay linearly recoverable.
Take a tiny model, . The two frequency divisors are and .
| Position | ||||
|---|---|---|---|---|
| 0 | 0.000 | 1.000 | 0.000 | 1.000 |
| 1 | 0.841 | 0.540 | 0.010 | 1.000 |
Now add these to a token embedding for "cat", say :
pos 0: x + PE(0) = [0.200, 0.900, 0.400, 1.500]
pos 1: x + PE(1) = [1.041, 0.440, 0.410, 1.500]
Same word, two different input vectors, attention can now tell "cat" at the start from "cat" later.
Instead of fixed sinusoids, many models (BERT, GPT-2) learn a position table: one trainable vector per slot 0 ... max_len - 1, added just like the sinusoidal signal. It is simple and often slightly better in-distribution, but it has a hard ceiling, there is no vector for any position at or beyond max_len, so the model cannot process sequences longer than it was trained on.
Rather than adding a position vector, RoPE rotates the query and key vectors by an angle proportional to their position, applied to 2D pairs of dimensions. The key property: the dot product between a query at position and a key at position depends only on the relative offset . So RoPE encodes relative position directly into the attention score, inside every layer, with no separate additive term. It is the dominant choice in modern open LLMs (LLaMA, Mistral, Qwen, and others), current common practice, not hype.
Why keep inventing schemes? Because context windows keep growing, and models must serve inputs longer than anything seen in training. Learned absolute positions cannot exceed max_len at all. Sinusoidal encoding was hoped to extrapolate but empirically degrades on longer sequences. Relative schemes generalize better, and RoPE in particular can be stretched by scaling its rotation frequencies (position interpolation, NTK-aware scaling, YaRN), which is how many models extend from a few thousand to hundreds of thousands of tokens. This extrapolation gap is the central reason the field moved past simple absolute encodings.
Common mistakes
max_len. There is simply no learned vector for those slots.