Semantic Search & QA

In dense retrieval you embed the query and every passage with the same model into one shared vector space, then rank passages by their similarity to the query and keep the nearest neighbors. Cosine similarity is the usual metric because it measures the angle between vectors and ignores their length. At scale you do not compare against every vector; an approximate nearest-neighbor (ANN) index (HNSW, IVF, FAISS) finds the closest ones in sub-linear time.

Worked example: ranking two passages

Take toy 3-dimensional embeddings (real models use hundreds to thousands of dimensions):

• Query "how do I reset my password?" → $q = [0.8,\ 0.5,\ 0.3]$
• Passage A "change your password in Settings, then Reset Credentials" → $a = [0.7,\ 0.6,\ 0.2]$
• Passage B "our office hours are 9am to 5pm" → $b = [0.1,\ 0.2,\ 0.9]$

Dot products: $q\cdot a = 0.8(0.7)+0.5(0.6)+0.3(0.2) = 0.92$ and $q\cdot b = 0.08+0.10+0.27 = 0.45$.

Norms: $\lVert q\rVert=\sqrt{0.98}\approx 0.990$, $\lVert a\rVert=\sqrt{0.89}\approx 0.943$, $\lVert b\rVert=\sqrt{0.86}\approx 0.927$.

Cosine scores: $\cos(q,a)=\frac{0.92}{0.990\times 0.943}\approx 0.985$ and $\cos(q,b)=\frac{0.45}{0.990\times 0.927}\approx 0.490$.

Because $0.985 > 0.490$, Passage A ranks first and is the context handed to the QA step, even though it never repeats the word "reset."

Steps: a semantic-search / QA pipeline

1. 1Ingest and chunkSplit source documents into passage-sized chunks so each unit is small enough to embed and retrieve independently.
2. 2Embed passagesEncode every chunk into a dense vector with an embedding model; store the vector next to the original text.
3. 3Build the indexLoad vectors into an approximate nearest-neighbor index (HNSW or FAISS) for fast similarity search at scale.
4. 4Embed the queryEncode the user question with the same model so query and passages share one vector space.
5. 5Retrieve top-kRank passages by cosine similarity to the query vector and keep the k highest-scoring chunks.
6. 6Rerank (optional)Rescore the shortlist with a cross-encoder that reads query and passage together for sharper ordering.
7. 7AnswerReturn an extractive span or generate a grounded answer with an LLM using the retrieved passages.

Extractive vs generative QA

Extractive QA selects a literal span from a retrieved passage, a reader model predicts start and end token positions, so the answer is always verbatim source text (great for auditability, but it cannot synthesize across passages). Generative QA feeds the passages to an LLM that writes a fluent answer in its own words, optionally citing sources. Generation is more natural and can combine evidence, but it can also hallucinate if the context is weak.

RAG is the pattern that makes generation trustworthy: retrieve top-k passages, inject them into the prompt as context, and instruct the model to answer only from that context and cite it. The retrieved text grounds the answer, shrinks hallucination, and lets you update knowledge by re-indexing documents rather than retraining the model.

Further reading

• Sentence-Transformers (SBERT) documentation, practical embedding and semantic-search models.
• Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering", the DPR paper.
• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", the original RAG paper.
• Jurafsky & Martin, Speech and Language Processing (3rd ed.), chapters on vector semantics and QA.