Loading lesson…
Embeddings / In Practice
Text, images, and audio in one shared space.
Reviewed by Yuvaraj
A search engine that finds photos from a typed sentence does something that sounds impossible: it compares words to pixels. Multimodal embeddings make this work by mapping different kinds of data, here, images and text, into one shared vector space, where a caption and the photo it describes land close together and unrelated pairs land far apart. Once both modalities live in the same space, comparing them is just arithmetic: the same cosine similarity you would use to compare two sentences now compares a sentence directly to an image.
A model like CLIP (Contrastive Language–Image Pretraining) uses two separate networks: an image encoder (often a vision transformer) and a text encoder (a transformer over tokens). Each turns its input into a fixed-length vector, and both project into the same dimensionality. The vectors are only comparable because the two encoders are trained together, not because images and text are inherently alike.
The training data is paired, an image and its caption, scraped at scale (hundreds of millions of pairs). For a batch of pairs, the model computes all cosine similarities between every image and every text. The contrastive objective (InfoNCE) pushes the correct pairs (the diagonal of that matrix) toward high similarity and the mismatched pairs toward low similarity:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
where is an image embedding, a text embedding, is cosine similarity, and is a learned temperature. A symmetric text-to-image term is added. Over billions of such comparisons, the two encoders learn to agree on what "a dog on a beach" means, whether it arrives as pixels or as tokens.
For a curious beginner
Two describers, one who only sees pictures, one who only reads words, are trained until they always point to the same spot on a shared map for the same idea. After training, a phrase and its matching photo naturally land near each other.
How it is actually used
Run the image through the vision encoder, the caption through the text encoder, L2-normalize both outputs into the same -dimensional space, and compare with a dot product (equals cosine on unit vectors). Store image vectors in your vector DB; embed the query text at search time.
The underlying mechanism
The symmetric InfoNCE loss maximizes mutual information between paired modalities. Gradients pull matched together and push every mismatched apart, shaping a joint manifold where cross-modal cosine similarity is meaningful.
| Candidate image | Cosine similarity to "a dog on a beach" |
|---|---|
| Dog on a sandy beach | 0.31 |
| Dog in a city park | 0.22 |
| Cat on a sofa | 0.08 |
The beach photo wins at 0.31 because its embedding aligns on both the subject (dog) and the setting (beach). The park dog matches the subject but not the setting, so it scores lower; the cat matches neither and scores near zero. Zero-shot classification flips the same trick: embed one image, embed a set of label prompts like "a photo of a dog" and "a photo of a cat", and assign the class whose text vector is nearest the image vector, no task-specific training required.
Common mistakes