Loading lesson…
Vector Databases / Retrieval Quality
Matching the index to your recall, latency, and memory budget.
Reviewed by Yuvaraj
An index is not a feature you switch on, it is a bet about your workload. Every vector index trades away one of recall, latency, or memory to buy the other two, and the "best" index is simply the one whose trade matches your data and your service-level objectives (SLOs). The right choice falls out of a handful of measurable properties, not from whichever database has the slickest landing page. This lesson gives you a framework for reasoning from those properties to a concrete index and its parameters.
Before naming any index, characterize the workload along six axes. Each one pushes you toward a different family.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Flat (brute-force) search sits below both: exact by definition, trivially correct, and the right answer for small sets or when recall must be perfect. pgvector deserves its own mention, when you already run Postgres and scale is modest, its HNSW and IVFFlat indexes let you keep vectors beside your relational data and avoid operating a second system.
For a curious beginner
You can pick any two of high recall, low latency, and low memory, never all three at full strength. Compressing vectors saves memory but blurs distances, so recall drops. Searching harder restores recall but costs time.
How it is actually used
Raising efSearch (HNSW) or nprobe (IVF) visits more candidates: recall
climbs toward the exact baseline while latency rises roughly linearly.
Product quantization shrinks the memory footprint but adds
distance-estimation error that no amount of search fully removes.
The underlying mechanism
Brute force is exact at per query. HNSW reaches high recall at roughly by trading a memory-heavy graph. PQ replaces a -dimensional float vector with subquantizer codes, cutting storage from bytes to about bytes at the cost of a quantization error that lower-bounds achievable recall.
| Scenario | Workload profile | Recommended index + params | Why |
|---|---|---|---|
| 50k-doc internal wiki | Small set, low QPS, exact answers valued, already on Postgres | pgvector: flat scan, or HNSW (m=16, ef_search=64) | At 50k, brute force answers in milliseconds with exact recall and no extra service. Add HNSW only if QPS grows. |
| 20M-vector product catalog | Large set, high QPS, tight RAM, mostly static with nightly rebuilds | IVF + PQ (nlist≈8192, nprobe=32, PQ m=32) | PQ compresses vectors so the index fits in RAM; static data suits IVF's batch-trained centroids. Recover lost recall by raising nprobe. |
| 2M chunks, heavy tenant filtering | Medium set, fits in RAM, every query filtered by tenant_id, frequent upserts | HNSW with a payload index on tenant_id (m=32, ef_search=128) | 2M fits comfortably in memory and HNSW handles incremental upserts. Filtered HNSW keeps recall high on selective filters; raise ef_search when a filter is broad. |
| 800M-vector log archive | Billions-scale, batch static, relaxed recall, cost-sensitive | IVF + PQ or DiskANN, sharded (nprobe tuned per shard) | RAM-only indexes are uneconomical here; on-disk and heavily quantized indexes trade latency and recall for feasibility. |
Product quantization (PQ) splits each vector into subvectors and replaces each with the nearest of a small learned codebook, shrinking storage roughly 8–16x. Scalar quantization is gentler, store each dimension as int8 instead of float32 for a 4x saving with little recall loss. Reach for quantization when memory is the binding axis, and always re-measure recall afterward: it is the lever that trades your recall budget for VRAM.
Common mistakes
efSearch or nprobe from a blog post was tuned on different embeddings and a different distribution. Measure recall on your own data or you are flying blind.