Learning Transferable Visual Models From Natural Language Supervision

TL;DR

Trains image and text encoders together on hundreds of millions of image-caption pairs so that matching pictures and descriptions land near each other in one shared space.

Why it matters

CLIP showed that natural-language supervision at scale yields visual models that classify new categories zero-shot. Its shared image-text space powers modern multimodal systems and text-to-image generators.

Key ideas

• Contrastive training pulls true image-caption pairs together, pushes mismatches apart.
• One shared embedding space for images and text.
• Zero-shot classification by comparing an image to text label embeddings.
• Learns from noisy web-scale data instead of curated labels.

Related concepts