Loading lesson…
Vector Databases / Retrieval Quality
Sharpening results beyond raw vector similarity.
Reviewed by Yuvaraj
A raw approximate-nearest-neighbor (ANN) lookup returns the passages whose embeddings sit closest to your query embedding. That is a strong starting signal, but on its own it is rarely good enough for production retrieval: it ignores who is allowed to see a document, it fails on exact identifiers and rare terms, and its ordering is only as good as a single similarity number. Three layers turn that raw lookup into reliable retrieval, metadata filtering, hybrid search, and reranking. Each fixes a different failure mode, and they compose into one pipeline.
Store structured fields alongside every vector, tenant or user ID, language, publication date, document type, access level, and constrain the search to rows that satisfy a predicate. This is both a correctness requirement (never return another tenant's data) and a quality lever (restrict to recent, in-language documents).
Where the filter runs matters. Post-filtering retrieves the vector top-k first and then discards rows that fail the predicate, which can leave you with far fewer than k results, or none, because the matching documents may never have entered the top-k in the first place. Pre-filtering restricts the candidate set before or during the search, so recall is measured over the allowed set.
Pre-filtering and graph indexes
Pre-filtering is easy for a flat brute-force index but tricky for graph indexes like HNSW: the graph's connectivity assumes every node is present, so masking out non-matching nodes can strand the traversal in a disconnected region. Engines handle this with filtered graph search, or by falling back to brute force over the (small) matching subset when a filter is very selective.
Dense (semantic) retrieval and sparse (keyword) retrieval fail in opposite ways. The fix is to run both and fuse their ranked lists. A robust, model-free fusion method is Reciprocal Rank Fusion (RRF), which combines lists by rank position rather than by raw score.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
For a curious beginner
How it is actually used
The underlying mechanism
RRF scores each document as
where is the position of in list (best rank is 1) and is a smoothing constant, conventionally . Suppose the two retrievers return the same four documents in different orders, dense ranks them A, B, C, D; BM25 ranks them C, A, D, B:
| Document | Dense rank | BM25 rank | RRF terms | Fused score |
|---|---|---|---|---|
| A | 1 | 2 | 0.03252 | |
| C | 3 | 1 | 0.03227 | |
| B | 2 | 4 | 0.03175 | |
| D | 4 | 3 | 0.03150 |
The fused order is A, C, B, D, which matches neither input list. A wins because it sits near the top of both channels (ranks 1 and 2); C is strongest in keyword search but held back by its middling dense rank; B, second in dense, collapses because it was last in BM25. Fusing by rank is what makes this robust: the tiny absolute differences (all scores cluster near ) never let one channel's unnormalized scale dominate the other.
Fusion gives a good candidate set, but the first stage still scored every passage without ever reading the query and passage together. A cross-encoder does exactly that: it feeds the concatenated query and passage through one transformer and outputs a single relevance score, letting every query token attend to every passage token.
This is the retrieve-then-rerank pattern: a cheap, recall-oriented first stage pulls N candidates; an expensive, precision-oriented reranker reorders them. The tradeoff is direct, a larger N raises the recall ceiling but adds one forward pass per candidate, so latency grows roughly linearly with N.
Common mistakes