Loading lesson…
LLM Foundations / From Text to Tokens
Meaning as geometry.
Reviewed by Yuvaraj
Once text is split into tokens and mapped to integer ids, the model faces a problem: an id like 4127 has no meaning. The number is just a label. The fix is one of the most important ideas in modern AI: the embedding.
An embedding turns each token (and, with more work, whole sentences) into a vector, a list of numbers, often hundreds or thousands of them. The point is not the individual numbers, but the arrangement:
Things with similar meaning are placed close together in the space; things with different meaning are placed far apart.
Meaning becomes geometry. "Distance" now stands for "difference in meaning." This is what lets a search system find a document about "car maintenance" when you asked about "auto repair," even with no shared words.
Because embeddings are vectors, we can measure how alike two of them are. The most common measure is cosine similarity, the cosine of the angle between two vectors:
Answer from memory before revealing, retrieval practice is what builds durable recall.
An embedding represents a word or item as:
Two sentences with similar meaning tend to have embeddings that are:
Cosine similarity is a common way to compare two embedding vectors.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Cosine similarity looks at direction, not length, which is usually what we want for meaning.
Real embeddings have hundreds of dimensions, but the arithmetic is identical in three. Take toy vectors for three words:
cat = [2, 1, 0]kitten = [3, 1, 0]airplane = [0, 1, 3]Compare cat with kitten:
Now cat with airplane:
Nearly 1 for the related pair, near 0 for the unrelated one, the number is the semantic closeness. A retrieval system does exactly this, at scale, to rank which stored vectors are nearest your query.
In practice, strong systems often combine both, a theme you will meet again in the retrieval lessons.
Embeddings are learned, not assigned by hand. During training, the model adjusts each vector so that arrangements which help it do its job (like predicting the next token) get reinforced. Meaning falls out of the training objective.
The famous analogy, with a caveat
Early word embeddings showed striking patterns like king − man + woman ≈ queen, where relationships appear as consistent directions in the space. It
is a genuine and useful intuition, but treat it as illustrative, not exact.
Real embedding spaces are messier than the clean analogy suggests.
Same model on both sides, and 'similar' is not 'true'
Vectors are only comparable if they came from the same embedding model: never compare a query embedded by model A against documents embedded by model B. And a high cosine score means "close in this model's learned sense," not "factually correct" or "the right answer", it is a similarity signal, not a verdict.
Embeddings are the foundation for:
You will meet all of these later; each rests on the idea introduced here.