Loading lesson…
LLM Foundations / How LLMs Work
Deciding what matters.
Reviewed by Yuvaraj
Attention is the idea that made modern language models possible. It sounds abstract, but the core is intuitive: when interpreting a word, look at the other words that matter, and mostly ignore the rest.
Take the sentence: "The trophy did not fit in the suitcase because it was too big."
What does "it" refer to, the trophy or the suitcase? You resolved it instantly by relating "it" to the rest of the sentence. Attention is how a model does the same: for each word, it decides how much every other word should influence its interpretation.
For a curious beginner
Picture reading a sentence with a highlighter. For the word "it," you brighten the words that help you understand it, here, "trophy", and dim the rest. Attention is the model learning where to shine the highlighter for every word at once.
How it is actually used
Each token produces three vectors: a query (what am I looking for?), a key (what do I offer?), and a value (what I contribute if chosen). A token's query is compared against every token's key to get scores; the scores become weights (via softmax); the output is the weighted sum of the values. Do this with several independent "heads" in parallel and you get multi-head attention, letting the model attend to different relationships at once.
The underlying mechanism
Scaled dot-product attention, over queries , keys , and values :
Answer from memory before revealing, retrieval practice is what builds durable recall.
Self-attention updates a token's representation using:
In attention, the dot product between a query and a key determines:
In causal (masked) self-attention, a token is allowed to attend to tokens that come after it.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
scores every query against every key; dividing by keeps the scores from growing too large as the key dimension grows; softmax turns them into weights that sum to 1; multiplying by returns a weighted blend of the value vectors.
scores every query against every key; dividing by keeps the scores from growing too large as the key dimension grows; softmax turns them into weights that sum to 1; multiplying by returns a weighted blend of the value vectors.
Watch attention resolve "it" with tiny numbers. Suppose the query for "it," compared against three candidate keys and scaled by , gives these scores:
trophy: 2.0suitcase: 0.5big: 0.5Softmax turns scores into weights that sum to 1:
So the representation of "it" is built as 0.69 × value(trophy) + 0.15 × value(suitcase) + 0.15 × value(big), it draws most of its meaning from "trophy." Change one score and the blend shifts. That weighted blend, computed for every token at once, is the whole mechanism.
Two flavors
When the queries, keys, and values all come from the same sequence, it is self-attention, words in a sentence relating to each other. When queries come from one sequence and keys/values from another (say, relating a translation to its source), it is cross-attention. Language models rely heavily on self-attention.
Earlier sequence models processed words one after another, which was slow and struggled to connect distant words. Attention relates every token to every other token directly and in parallel, better at long-range links and far friendlier to modern hardware. That combination is what unlocked large-scale transformers.
Attention weights are not explanations
It is tempting to read a high attention weight as "this is why the model answered that way." Resist it. Attention shows where information flowed, not a faithful, causal explanation of the output, a model has many layers and heads, and interpretability research treats attention maps as a hint, not proof.