Loading lesson…
Computer Vision / Modern Vision
Connecting images to language.
Reviewed by Yuvaraj
A classifier trained on ImageNet knows exactly 1,000 categories and not one more; to add "capybara" you would relabel data and retrain. CLIP broke that constraint. You can hand it an image and a list of category names written in plain English, including names it never saw a labeled example of, and it will pick the best match. The mechanism behind this, and behind image search by sentence and the vision encoders inside modern chat assistants, is a single idea: train images and text into one shared space where a picture and its description land close together.
A vision-language model like CLIP (Contrastive Language–Image Pretraining) runs two separate networks. An image encoder, a CNN or, more often now, a Vision Transformer, turns a picture into a vector. A text encoder, a Transformer over tokens, turns a caption into a vector. Both project to the same dimensionality and their outputs are L2-normalized to unit length. The two encoders are trained together on hundreds of millions of image–text pairs scraped from the web, which is the only reason their outputs become comparable; bolting two independently trained models together would produce two unrelated spaces.
Training is contrastive. For a batch of image–text pairs, the model embeds all images and all texts and forms the grid of similarities between every image and every text. The correct pairs lie on the diagonal; the off-diagonal entries are mismatches. The objective, a softmax cross-entropy applied across each row and each column, pushes the diagonal similarities up and the rest down:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
where is an image embedding, a text embedding, is cosine similarity, and a learned temperature; a symmetric text-to-image term is added. Over billions of comparisons the encoders learn to agree on meaning across modalities.
Once both encoders output unit vectors in the same space, comparing an image to a text is just a dot product, which, for unit vectors, equals the cosine of the angle between them:
Higher cosine means "more related." This one number links pixels to words and powers everything that follows.
The headline capability. To classify an image without any task-specific training, turn each candidate class name into a prompt such as a photo of a {label}, embed all the prompts with the text encoder, embed the image with the image encoder, and pick the class whose text embedding has the highest cosine similarity to the image.
For a curious beginner
Instead of training a model to emit a label from a fixed menu, you ask a different question: "which of these written descriptions best matches this picture?" Because the model already understands how pictures and words relate, you can rewrite the menu at any moment, no retraining. The set of labels becomes an input to the model rather than a baked-in part of it.
How it is actually used
Precompute one text embedding per class prompt once. At inference, embed the
image, L2-normalize, take its cosine similarity to every class embedding,
multiply by the learned temperature, and softmax to get class probabilities;
the argmax is the prediction. Adding a brand-new class is a single extra
text embedding, not a training run, and prompt wording (a photo of a{" "} {label} versus just {label}) measurably shifts accuracy.
The underlying mechanism
A linear classifier scores class as row of a learned weight matrix times the feature vector. Zero-shot CLIP replaces each learned with the text embedding of the class name: the score for class is (cosine, since vectors are unit length), scaled by the temperature before the softmax. The text encoder synthesizes the classifier weights from language, so the decision boundary is defined by prompts rather than fit by gradient descent on labeled images.
An image of a dog is compared against three class prompts. Suppose the raw cosine similarities come out:
| Class prompt | Cosine similarity | Scaled logit () |
|---|---|---|
| a photo of a dog | 0.28 | 28 |
| a photo of a cat | 0.14 | 14 |
| a photo of a car | 0.05 | 5 |
In raw cosine terms the differences look modest, a symptom of the modality gap, the tendency of image and text vectors to occupy somewhat separate regions even in a shared space. The learned temperature fixes this by scaling similarities by roughly 100 before the softmax. The scaled logits produce probabilities dominated by the top class: against and , so the model assigns "dog" a probability of about . Temperature scaling is what converts small cosine gaps into a decisive prediction.
This is a different kind of classifier from the one you train on a fixed label set, and the contrast is the whole point:
Zero-shot classification is one use of many. The same image–text alignment is a general building block:
Common mistakes
a photo of a {label}
routinely beats a bare label, and prompt ensembles help further. - Expecting
reliable fine-grained compositional reasoning; CLIP-style models encode a bag
of concepts more than precise relations, so "a red cube on a blue sphere"
often confuses the two objects. - Trusting a VLM's fluent description as
ground truth; these models still hallucinate objects and misread text, small
details, and spatial relationships. - Forgetting that web-scraped training
data carries biases and unsafe content into the shared space, which propagate
into every downstream use.