Loading lesson…
Computer Vision / Modern Vision
Reframing an image as a sequence of patches.
Reviewed by Yuvaraj
For most of the 2010s, "computer vision" meant "convolution." The idea that the Transformer, the architecture built for sequences of words, could match or beat convolutional networks on images seemed unlikely, because images are grids, not sentences. Then a 2020 paper, An Image is Worth 16x16 Words, showed it works with almost no vision-specific machinery: cut the image into fixed square patches, treat each patch as a token, and run the exact same Transformer encoder used for language. The trick is entirely in how the image becomes a sequence.
A Transformer consumes a sequence of vectors (tokens). A sentence supplies these naturally; an image does not, so the Vision Transformer (ViT) manufactures them. It divides the image into a grid of non-overlapping patches and turns each patch into one token.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
The number of patch tokens is fixed by the image and patch sizes. For a image with patches:
Add the single prepended class token and the encoder processes tokens. Each token starts as raw pixel values, which the patch-embedding projection maps to a -dimensional vector. That linear projection is the only image-specific component, and it is equivalent to a single convolution whose kernel size and stride both equal the patch size.
Model names encode these choices directly: ViT-B/16 is the Base configuration with patches, while ViT-L/14 is the larger Large variant with patches, a smaller patch means more tokens and more compute, as the next section makes precise.
The patch-embedding projection is one learned weight matrix applied identically to every patch. After it, the model is architecturally identical to a language Transformer operating on 197 vectors. The extra class token is a learnable vector that carries no image content at the start; its job is to aggregate information from all patches through the attention layers, so that its final-layer representation summarizes the whole image for the classifier. (Later ViTs sometimes replace it with a global average over patch tokens, but the class token is the original design.)
Self-attention is permutation-invariant: shuffle the input tokens and the output is shuffled identically, but nothing about the computation changes. That is fatal for images, where patch in the top-left and patch in the bottom-right must not be interchangeable. ViT restores order by adding a learnable positional embedding to each token, a per-position vector the model trains from scratch. Without it, the ViT would treat the image as an unordered bag of patches and lose all spatial structure.
Self-attention compares every token to every other token, so its cost grows with the square of the sequence length . Halving the patch size doubles the patches per side and quadruples :
Smaller patches see finer detail and usually raise accuracy, but they make the model quadratically more expensive. Patch size is therefore a direct dial between resolution of detail and cost.
This is the concept that defines ViTs, and the reason they behaved so differently from CNNs at first.
For a curious beginner
A CNN arrives already believing two things about images: nearby pixels belong together, and an object is the same object wherever it appears. A ViT believes almost nothing, it must learn even that neighboring patches are related. Fewer built-in assumptions means more to learn from data, but also fewer wrong assumptions to unlearn once data is abundant.
How it is actually used
Convolution hard-codes locality (small kernels) and translation-equivariant weight sharing. ViT drops both in favor of global self-attention plus learned positional embeddings. The consequence is empirical: on a medium dataset like ImageNet-1k a good CNN beats a plain ViT, but pretrained on a much larger corpus (ImageNet-21k or the 300-million-image JFT), the ViT catches up and passes it. In practice you close the gap with heavy augmentation, distillation (DeiT), or self-supervised pretraining (DINO, MAE).
The underlying mechanism
An inductive bias restricts the hypothesis space. Convolution constrains the weight matrix to a shared local stencil: a strong prior that lowers variance at the cost of some bias. Self-attention imposes almost none of that structure, so its hypothesis class is far larger, higher capacity and lower bias, but higher variance that only pays off when the training set is large enough to constrain it. It is the bias-variance trade-off expressed through architecture.
The two architectures therefore occupy different regimes:
| Property | CNN | Vision Transformer |
|---|---|---|
| Built-in spatial prior | Strong (locality, equivariance) | Weak (only patch grid + learned positions) |
| Receptive field | Grows with depth | Global from the first attention layer |
| Data efficiency | High, works on modest datasets | Low, needs large data or strong pretraining |
| Scaling with data + compute | Good, but saturates sooner | Excels; keeps improving at large scale |
Weak priors are not a defect
A ViT's lack of built-in vision assumptions looks like a weakness on small datasets and becomes a strength at scale. With enough data the model learns the right spatial structure directly, unconstrained by hand-designed priors, which is precisely why Transformers now dominate at the largest scales across vision, language, and multimodal models alike.
Common mistakes