Loading lesson…
Deep Learning & the Frontier / Learning Machines
One shared space for images, text, and more.
Reviewed by Yuvaraj
Most of the world's information is not text. It is images, audio, video, and the messy combinations of all three. Multimodal AI is the effort to build models that perceive and relate more than one of these at once, describing a photo, answering a question about a chart, transcribing speech, or generating an image from a sentence.
A text encoder turns a sentence into a vector. An image encoder turns a picture into a vector. On their own these live in unrelated spaces. The central idea of modern multimodal models is to train the two encoders so that matching pairs land near each other in one shared vector space, the caption "a dog on a beach" ends up close to a photo of exactly that.
Why a shared space is powerful
Once text and images share a geometry, tasks become distance queries. Image search by text prompt, zero-shot classification ("is this closer to cat or dog?"), and retrieval all reduce to "which vectors are nearby?", the same cosine-similarity idea you met with embeddings, now spanning modalities.
The dominant recipe is contrastive learning. Take a big batch of image–caption pairs. Push each image's vector toward its own caption's vector and away from all the other captions in the batch. With a similarity score and a temperature, the loss for one image against captions is
Answer from memory before revealing, retrieval practice is what builds durable recall.
What is the goal of contrastive training in a CLIP-style model?
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
This is the InfoNCE loss, the numerator rewards the true pair, the denominator is a softmax over all candidates, so minimizing it makes the right match stand out from the crowd. CLIP trained this way over hundreds of millions of pairs and produced encoders general enough to classify images it was never explicitly trained on.
Take one image and three candidate captions in a batch, its true match first. Suppose cosine similarities come out and the temperature is , so the scaled logits are . The softmax puts probability on the true pair, giving a loss of , tiny, because the right caption already dominates. If the true pair scored no higher than the distractors, its probability would fall toward and the loss would jump to about , pushing the encoders to pull that pair together. The temperature controls how sharply near-misses are punished.
Generation vs. understanding
Understanding an image (captioning, question answering) and generating an image (from a prompt) are different jobs. Generation typically uses a diffusion model steered by a text encoder; understanding routes image features into a language model. Many products chain both, which is why one assistant can see a picture and also make one.
Document understanding (invoices, forms, charts), accessibility (describing images for screen readers), medical and scientific imaging assistance, video summarization, and voice interfaces that combine speech recognition, language, and speech synthesis. In each case the win comes from not forcing a rich signal through a text bottleneck first.
Honest limits
Multimodal models inherit every failure mode of their parts and add new ones: they hallucinate details not present in an image, miscount objects, are fooled by text embedded in pictures, and reflect biases in their training pairs. A confident caption is not evidence the model actually parsed the image correctly, verify when it matters.
Multimodal AI does not introduce a new kind of intelligence; it reuses the machinery you already know, encoders, embeddings, attention, softmax, gradient descent, and points it at more than one sense at a time. The unifying bet is that meaning can be represented in a common space regardless of whether it arrived as pixels, waveforms, or words.