Loading lesson…
Embeddings / Representation
Meaning as a point in space.
Reviewed by Yuvaraj
An embedding is a list of numbers that captures the meaning of something, a word, a sentence, an image, as a single point in a shared space. The trick is geometric: items with similar meaning are placed close together, and unrelated items land far apart. This is what lets a computer, which only manipulates numbers, reason about similarity, search, and relatedness without ever being handed explicit rules for what "similar" means.
Before embeddings, the naive way to turn a word into numbers was a one-hot vector: one dimension per vocabulary item, all zeros except a single 1 marking which word it is. It works, but it is a poor representation of meaning.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Crucially, an embedding is learned, not designed. A model, word2vec, a transformer text encoder, an image encoder, is trained on some task (predict a masked word, predict the next token, match captions to images), and the vectors are a byproduct of doing that task well. Guided by millions of examples, the model discovers that things appearing in similar contexts should get similar vectors. Nobody decides what each dimension "means," and in general the individual dimensions are not interpretable on their own.
Real embeddings live in hundreds of dimensions, but the idea is visible in just two. Below, the two axes are labeled for illustration only, treat "royalty" and "gender" as made-up directions, since real dimensions carry no such clean labels.
| Word | Vector (toy, 2-D) |
|---|---|
| king | [0.90, 0.60] |
| queen | [0.90, 0.40] |
| man | [0.20, 0.60] |
| woman | [0.20, 0.40] |
| banana | [0.10, 0.05] |
Measure closeness with straight-line (Euclidean) distance, . Then king and queen are close because they share a high first coordinate, while banana is off in its own corner:
The geometry even encodes relationships as directions. The classic word2vec analogy falls straight out of the numbers:
The one idea to keep
An embedding turns "how similar are these two things?" into "how close are these two points?" Every downstream use, semantic search, clustering, recommendations, retrieval for RAG, is built on that single substitution.
For a curious beginner
Think of a map. Towns in the same region sit near each other; distant cities are far apart. An embedding space is that map for meaning: "puppy" and "dog" are neighbors, "invoice" is in another part of town, and closeness on the map is similarity of meaning.
How it is actually used
You compute a single number between two vectors. Two common choices: Euclidean distance (smaller means more alike) and cosine similarity (larger means more alike). To find related items you embed a query, then fetch its nearest neighbors from a store of precomputed vectors, the core of vector search.
The underlying mechanism
Cosine similarity measures the angle between two vectors, ignoring their length:
It ranges from (opposite) through (unrelated) to (same direction). Because it normalizes by magnitude, it compares orientation in the space rather than raw scale.
Dimensionality is a model property: a given model always outputs vectors of one fixed size (384, 768, 1536, and so on). More dimensions can capture finer distinctions but cost more memory and compute, bigger is not automatically better. Most importantly, the numbers are only comparable within one model's space. You must embed both sides of any comparison, every query and every document, with the same model, or the distances are meaningless.
Common mistakes