Loading lesson…
Transformers / The Transformer Block
Assembling attention into a trainable stack.
Reviewed by Yuvaraj
A transformer block is the single unit of computation that, repeated dozens of times, forms a modern language model. Once you understand one block you understand the whole stack: every layer is structurally identical, differing only in its learned weights. Each block does two things in sequence, it lets every position gather information from other positions (self-attention), then it processes each position independently through a small neural network (the feed-forward network). Both operations are wrapped in the same protective scaffolding: a residual connection and a normalization step. That scaffolding is what makes it possible to stack the block deep enough to be useful.
Every block contains exactly two sub-layers, applied one after the other:
Attention handles communication across the sequence; the FFN handles per-token computation. Depth comes from alternating the two.
Each sub-layer is not used raw. Instead of returning Sublayer(x), the block adds the original input back and then normalizes the result. The residual (skip) connection is the add:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
This matters for gradients. During backpropagation the addition gives the gradient a direct path back to earlier layers, the derivative of with respect to contains an identity term that never vanishes, so signal still reaches the bottom of a very deep stack. Without residuals, gradients shrink multiplicatively through each layer and training stalls.
Layer normalization rescales each token's vector to zero mean and unit variance, then applies a learned gain and bias. This keeps activations in a stable range regardless of depth or input, so the optimizer sees a consistent loss surface and can use a higher, steadier learning rate.
Where you place the normalization changes training stability dramatically.
Pre-LN wins in practice because the residual highway stays unnormalized end to end, preserving the clean gradient path that makes deep stacks trainable.
Take a sequence of tokens with model dimension , using heads and an FFN hidden size of . The block is shape-preserving, watch the dimensions:
Because the output shape equals the input shape, you can feed it straight into another identical block. Stacking of them (12, 24, 96, and beyond) is what builds depth, and each block refines the representation a little more.
The feed-forward network deliberately widens before it narrows: . The first linear layer projects up to a higher-dimensional space (the ratio is a common default), a non-linearity such as GELU or ReLU is applied, and the second linear layer projects back down to . The wide middle gives the network room to compute richer non-linear features per token; projecting back to keeps the block shape-preserving so it can stack. This FFN typically holds roughly two-thirds of a block's parameters.
Why the block is shape-preserving
Every sub-layer maps back to . That invariant is the whole trick behind stacking: identical blocks compose because each one hands the next exactly the tensor shape it expects.
Common mistakes
x + Sublayer(...), not just Sublayer(...). Dropping the skip connection breaks gradient flow and deep models stop training.