Lab

Attention Visualizer

Inspect which tokens attend to which, with scaled dot-product.

Attention lets each token pull information from the others. For a chosen query token the model scores every key token, turns those scores into weights with softmax, and blends the value vectors accordingly. Pick a query token and see what it attends to.

	key →
	The	cat	sat	on	mat
The
cat
sat
on
mat

The
19%
cat
24%
sat
16%
on
19%
mat
24%

Scaled dot-product attention computes:

$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\dfrac{QK^{\top}}{\sqrt{d}}\right)V$

Each row of the grid is one query token; the shade shows how much of its attention lands on each key token. Content words (“cat”, “mat”) attend to each other while the function words cluster separately. The output vector is the softmax-weighted blend of the value vectors.

Note: these vectors are hand-picked to make the pattern clear, and are used directly as Q, K and V. A real model learns separate Q/K/V projections from token embeddings over training data.

Challenge

Select “sat”. Why does it attend so differently from “cat”? Then select “The” and “on”, the two function words point at each other. What linguistic role is the attention grouping by here?

From the lesson: Attention, Intuitively