Self-attention lets every token in a sequence pull information from every other token, but to make that work a single token has to play three roles at once. When the word it scans a sentence, it must announce what kind of thing it is searching for, advertise what it itself has to offer other words, and carry the actual content it will hand over once it is chosen. Transformers give a token these three roles by projecting its embedding into three separate vectors, a query, a key, and a value, using three learned weight matrices. This projection is the foundation everything else in attention is built on.
The three projections
Let xi∈Rd be the embedding of token (after positional encoding). An attention layer holds three learned matrices , and it projects every token independently:
Ask the tutor
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
model
i
WQ,WK,WV∈Rdmodel×dk
qi=xiWQ,ki=xiWK,vi=xiWV
Two things matter here. First, all three vectors come from the same embedding xi, they are three different views of one token, not three different inputs. Second, the matrices are shared across all positions: the same WQ turns every token into a query. Because they are learned by gradient descent, the model discovers on its own what makes a useful question, a useful advertisement, and useful content. In practice dk is typically smaller than dmodel (a single head's slice).
What each vector means
Vector
Made from
Plain-language role
Analogy
Query q
xWQ
"What am I looking for?"
a search box
Key k
xWK
"What do I offer?"
a label on a shelf
Value v
xWV
"The content I pass along if chosen"
the payload
A token compares its query against every other token's key. Where a query and a key align, that token's value is the content that flows through.
Scoring: query dot key
The raw compatibility between token i (asking) and token j (offering) is the dot product of the query with the key:
score(i,j)=qi⋅kj=∑dqi,dkj,d
A large positive score means the two vectors point in similar directions, so token j is relevant to what token i wants. A score near zero means little relevance. The next lesson scales these scores, passes them through softmax, and uses the results to average the value vectors; here we only produce one score.
Worked example (2-dimensional)
Take querying token xA=[1,2] and candidate token xB=[0,1], with
WQ=[1101],WK=[0110]
Project A into a query, using the row-vector convention q=xW:
qA=[1,2][1101]=[1⋅1+2⋅1,1⋅0+2⋅1]=[3,2]
Project B into a key:
kB=[0,1][0110]=[0⋅0+1⋅1,0⋅1+1⋅0]=[1,0]
Now the single score we care about:
score(A,B)=qA⋅kB=3⋅1+2⋅0=3
For contrast, a third token xC=[1,0] projects to kC=[0,1], giving qA⋅kC=3⋅0+2⋅1=. Token B scores 3 and token C scores 2, so from A's point of view, B is the more relevant match.
1EmbedStart from a token embedding, e.g. token A is xA = [1, 2] after positional encoding.
2ProjectMultiply by the shared matrices to get the query qA = [3, 2] and, for token B, the key kB = [1, 0] and a value vB that would be carried if B is chosen.
3ScoreDot product of A's query with B's key: 3*1 + 2*0 = 3. A larger score means B is more relevant to what A is looking for.
Why three matrices, not one?
Separating "what I want" (query) from "what I offer" (key) lets a token be
searched for differently than it searches. That asymmetry is exactly what
a dot product between two distinct projections of two distinct tokens
captures, and it is why we cannot collapse WQ and WK into one matrix.
Common mistakes
Thinking Q, K, and V come from three different inputs. They are three
projections of the same token embedding. - Believing WQ,WK,WV are
hand-designed or fixed. They are learned parameters, trained end-to-end. -
Letting the value vector enter the score. Only the query and key form the
score; the value is the content carried once scores become weights. - Assuming
each token owns its own matrices. The three matrices are shared across every
position in the layer.