Loading lesson…
Embeddings / In Practice
How they are trained and how to choose one.
Reviewed by Yuvaraj
An embedding model is a neural network that maps a piece of text to a single fixed-length vector, positioned so that semantically similar inputs land close together in the space. That property is what makes vector search, clustering, deduplication, and retrieval-augmented generation possible. But "an embedding model" is not one thing, dozens of them differ in how they are trained, how they pool tokens into a vector, what languages and lengths they handle, and how expensive they are to run. Choosing well means understanding the mechanism, then matching it to your corpus, latency budget, and quality bar.
Most modern text embedders are bi-encoders: an encoder (usually a Transformer) reads the input once and emits a per-token hidden state, then a pooling step collapses those token vectors into one vector per input. Two pooling strategies dominate:
Neither is universally better, use whatever the model card specifies, because the pooling was chosen during training and the two are not interchangeable at inference.
A raw language model does not naturally place paraphrases near each other. Embedders acquire that geometry through contrastive training on positive and negative pairs. A positive pair is two texts that should be close (a question and its answer, a query and a relevant passage); negatives are unrelated texts. The model is trained to pull positives together and push negatives apart, typically with an InfoNCE-style objective:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Here is usually cosine similarity, is the positive, the are in-batch negatives, and is a temperature. The takeaway: the geometry is only meaningful within the space the model learned, which is why you must embed your corpus and your queries with the same model.
For a curious beginner
A bi-encoder hands every document a fixed name tag you can sort in advance. A cross-encoder reads the query and one document side by side, sharper judgment, but it cannot be done ahead of time.
How it is actually used
Embed the corpus once with a bi-encoder and store the vectors in a vector database. At query time you embed only the query and run ANN search. A cross-encoder runs the model once per query-document pair, so you use it only to rerank the top 20-100 candidates the bi-encoder returned.
The underlying mechanism
A bi-encoder scores with where the encoder is applied independently, so document vectors are reusable and indexable. A cross-encoder computes jointly over the concatenated tokens; the score does not decompose, so every pair costs a full forward pass.
Weigh dimensionality (storage and ANN cost scale with it), max sequence length (inputs longer than this are silently truncated), domain match, multilingual support, retrieval quality on a benchmark like MTEB, cost/latency, and whether outputs are L2-normalized (so cosine similarity equals a dot product).
| Model | Dims | Max tokens | MTEB-ish | Latency / 1k texts | Multilingual |
|---|---|---|---|---|---|
| Compact-384 | 384 | 512 | 62 | ~40 ms | English only |
| Balanced-768 | 768 | 8192 | 66 | ~120 ms | 100+ languages |
| Large-1024 | 1024 | 8192 | 68 | ~300 ms | 100+ languages |
For multilingual support-docs search, documents are long and mixed-language, so Compact-384 is disqualified: 512 tokens would truncate real articles, and it is English-only. Pick Balanced-768, its 8192-token context ingests full docs, it is multilingual, and its score is close to Large-1024 at roughly a third of the cost. Reserve Large-1024 for cases where the extra recall demonstrably pays for the storage and latency.
For low-latency autocomplete, inputs are short English strings and speed dominates. Pick Compact-384: 512 tokens is ample, 384 dimensions keep the index cheap, and 40 ms keeps suggestions instant.
# One model, used for BOTH corpus and queries
model = load_embedder("balanced-768")
doc_vecs = model.encode(docs, normalize=True) # offline: index once
index.add(doc_vecs)
q = model.encode(["reset my password"], normalize=True)
hits = index.search(q, k=20) # bi-encoder retrieval; rerank if needed
Common mistakes