A RAG system can be wrong in two unrelated ways: it can fetch the wrong passages, or it can fetch the right ones and still write a bad answer. A single end-to-end "was the answer good?" score blends these together, so when it drops you cannot tell what to fix. Evaluating RAG well means scoring the retriever and the generator on separate scoreboards, then reading the two together to localize the fault. This lesson defines the metrics for each stage and shows how their combination points you at the one thing to change.
Two stages, two scoreboards
A RAG pipeline factors cleanly into two components. Retrieval maps a query to a ranked list of top-k passages. Generation maps those passages plus the question to an answer. Each stage fails in its own way and demands its own fix, a mis-ranked retriever is a data and indexing problem, while an answer that ignores good context is a prompting and model problem. An aggregate metric such as end-to-end answer correctness is necessary but not sufficient: it is a lagging indicator that tells you something regressed without telling you where.
You can only improve the stage you can measure
If your only number is "final answer quality," every regression triggers the
same undirected scramble: swap the model, rewrite the prompt, re-chunk the
corpus, all at once. Separate retrieval and generation scores turn that
scramble into a decision, the green stage is exonerated, the red stage gets
the work.
Metric
Stage
Ground truth it needs
A low score means
Ask the tutor
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Recall@k
Retrieval
Labeled relevant docs
Gold passages are not in the top-k at all, widen search or raise k
Precision@k
Retrieval
Labeled relevant docs
The top-k is padded with irrelevant chunks, tighten or rerank
MRR
Retrieval
Labeled relevant docs
The first relevant hit sits too low in the list, improve ranking
nDCG@k
Retrieval
Labeled (graded) relevant docs
Relevant docs are present but poorly ordered, rerank
Faithfulness
Generation
None (answer + context)
The answer asserts claims the context does not support, hallucination
Answer relevance
Generation
None (question + answer)
The answer dodges, pads, or only partly addresses the question
Context precision
Retrieval (bridge)
Reference answer / labels
Useful chunks are buried among noise in the retrieved set
Context recall
Retrieval (bridge)
Reference answer
The context is missing facts the answer needs, a retrieval gap
The RAG evaluation scoreboard. Retrieval metrics need relevance labels on documents; the generation and bridge metrics work from the answer and (sometimes) a reference answer. A low score in each row points at a specific fix.
Scoring the retriever
Treat retrieval as classic ranked information retrieval. For each evaluation query you need relevance judgments, which passages in the corpus count as relevant (the "gold" set). With those in hand, four metrics cover the ground. Let Retrievedk be the top-k returned and Relevant the gold set.
Recall@k, of the passages that should have been found, what fraction landed in the top-k? This is the ceiling on everything downstream: a fact the retriever never surfaces cannot be used by the generator.
Recall@k=∣Relevant∣∣Relevant∩Retrievedk∣
Precision@k, of the k passages you returned, what fraction were relevant? Low precision means the generator must read around noise, and long, noisy context degrades answers.
Precision@k=k∣Relevant∩Retrievedk∣
MRR (mean reciprocal rank), how high is the first relevant passage, averaged over queries? For query q, let rankq be the position of its first relevant hit (the term is 0 if none is retrieved).
MRR=∣Q∣1q=1∑∣Q∣rankq1
nDCG@k (normalized discounted cumulative gain), the rank-aware metric. It discounts each relevant hit by a logarithm of its position and normalizes against the ideal ordering (IDCG), giving a score in [0,1]. With relevance grades reli at position i:
DCG@k=i=1∑klog2(i+1)reli,nDCG@k=IDCG@kDCG@k
Intuition
For a curious beginner
Recall@k and precision@k ask a set question, did the right cards come back at all, and how much junk came with them? MRR and nDCG ask an ordering question, did the right cards land near the top? RAG cares intensely about order because the generator reads the front of the list most reliably and your context budget is finite. A relevant passage buried at rank 20 might as well not exist.
Engineering
How it is actually used
Track recall@k to choose k (how many chunks you actually paste into the prompt) and to confirm the answer is even reachable. Track nDCG to judge ordering and to justify a reranker: if recall@k is high but nDCG is low, the passages are there but mis-ordered, and a cross-encoder reranker is the targeted fix. MRR is the cheap first-hit summary for single-answer lookups (FAQ, "what is X"), where only the top passage matters.
Mathematical
The underlying mechanism
MRR is the special case of a rank-aware metric that scores only the first relevant item and ignores the rest. nDCG generalizes it: the log2(i+1)1 term keeps discounting every relevant hit as it moves down the list, and grades reli let a "perfect" passage outweigh a "partly relevant" one. Normalizing by the ideal DCG makes scores comparable across queries with different numbers of relevant documents.
For a curious beginner
Recall@k and precision@k ask a set question, did the right cards come back at all, and how much junk came with them? MRR and nDCG ask an ordering question, did the right cards land near the top? RAG cares intensely about order because the generator reads the front of the list most reliably and your context budget is finite. A relevant passage buried at rank 20 might as well not exist.
How it is actually used
Track recall@k to choose k (how many chunks you actually paste into the prompt) and to confirm the answer is even reachable. Track nDCG to judge ordering and to justify a reranker: if recall@k is high but nDCG is low, the passages are there but mis-ordered, and a cross-encoder reranker is the targeted fix. MRR is the cheap first-hit summary for single-answer lookups (FAQ, "what is X"), where only the top passage matters.
The underlying mechanism
MRR is the special case of a rank-aware metric that scores only the first relevant item and ignores the rest. nDCG generalizes it: the log2(i+1)1 term keeps discounting every relevant hit as it moves down the list, and grades reli let a "perfect" passage outweigh a "partly relevant" one. Normalizing by the ideal DCG makes scores comparable across queries with different numbers of relevant documents.
A worked retrieval example
Take one query with a known gold set of three relevant passages, Relevant={D2,D5,D9}. The retriever returns this ranked top-5: [D8,D5,D1,D2,D7]. Label each returned passage 1 if it is in the gold set, else 0.
The IDCG assumes the ideal ranking [1,1,1,0,0], all three relevant passages at the top, so nDCG@5 is penalized both for the missed D9 and for placing the two hits at ranks 2 and 4 instead of 1 and 2. Now roll MRR over three queries, using the miss rule (0 when no relevant passage is retrieved):
Query
First relevant at rank
Reciprocal rank
q1
2
1 / 2 = 0.50
q2
1
1 / 1 = 1.00
q3
not in top-5
0.00
MRR=30.50+1.00+0.00=0.50
Read q1 as a diagnosis, not just a number. Recall@5 is a decent 0.667, but precision@5 is only 0.40 and nDCG@5 is 0.498: the relevant passages are present yet mis-ordered and diluted by noise. If the final answer is weak, these numbers say do not blame the generator first, add a reranker and tighten retrieval, because the context the model receives is itself second-rate.
Scoring the generator
Once you trust the context, judge what the model did with it. These metrics are typically computed by an LLM-as-judge, and frameworks such as RAGAS and ARES package them.
Faithfulness (groundedness), is every claim in the answer supported by the retrieved context? Decompose the answer into atomic claims and count how many are entailed by the context. If an answer makes 4 claims and 3 are supported, faithfulness is 3/4=0.75. This is your hallucination detector relative to the context, and it needs no reference answer.
Faithfulness=total claims in the answerclaims in the answer supported by the context
Answer relevance, does the answer actually address the question, without evasion or padding? RAGAS estimates it by prompting a model to generate candidate questions the answer would satisfy, then measuring their similarity to the real question. Note the scope: it measures on-topic-ness, not factual correctness.
Context precision and context recall, the bridge metrics. Context precision is rank-aware: are the genuinely useful chunks ranked ahead of the noise in the retrieved set? Context recall asks whether the retrieved context contains everything the reference answer needs. Context recall requires a ground-truth answer; context precision needs a reference answer or relevance labels. Where classic recall@k needs corpus-wide relevance judgments, these lean on a reference answer plus a judge, cheaper to assemble, but they inherit the judge's error.
Faithfulness is agreement with context, not truth
An answer can be perfectly faithful and still wrong, if the context it
faithfully summarizes is wrong. Faithfulness measures whether the generator
respected the passages it was given; correctness measures whether the final
claim matches reality. Keep them as separate columns, conflating them hides
the most dangerous failure in the next section.
Localizing the failure
Cross the two stages and four situations appear. Which cell a failing case lands in is the whole reason you measured them separately, it names the fix.
Localizing a RAG failure. The row is whether retrieval surfaced the needed context; the column is whether the answer is faithful to that context. Each cell names the fix, and the bottom-right warns that a 'correct' answer from missed context is only the model's memory, a latent retrieval bug an end-to-end score would reward.
The two off-diagonal cells are the failures the whole exercise exists to separate. They look similar end-to-end and need opposite fixes.
Right answer from wrong context
Retrieval missed the gold passages, so context recall is low
The answer came from the model's parametric memory, not your data
Faithfulness (attribution to the given context) is therefore low
End-to-end correctness looks fine, so one blended score hides the bug
It will not generalize to private or fresh facts and cannot cite, fix retrieval
Hallucination despite good context
Retrieval surfaced the right passages, so context recall is high
The answer contradicts or ignores them, so faithfulness is low
Retrieval metrics are green, the fault is squarely in generation
Correctness may be low, but the diagnosis is unambiguous
Fix the prompt, grounding instructions, or model, not the retriever
The dangerous one is the first. Because the model answered correctly from memory, an end-to-end grader gives it full marks while the retriever silently failed, and the moment the question turns to a private document or a fact newer than the model's training, the same pipeline breaks with equal confidence. This is exactly why you check attribution (faithfulness against the retrieved context), not only the final answer.
Common mistakes
Reporting only end-to-end correctness. It is a lagging score that cannot say whether to fix retrieval or generation. Always decompose into per-stage metrics before acting.
Judging retrieval by recall@k alone. High recall@k with low precision@k or nDCG means the right passage is present but buried in noise the generator may skip, a well-documented "lost in the middle" effect. Watch ordering, not just presence.
Treating faithfulness as correctness. Faithfulness measures agreement with the retrieved context, not with reality. A faithful summary of a wrong passage is confidently wrong.
Rewarding a correct answer that came from missed context. It is the model's memory, not your system. Verify the answer is attributable to the retrieved passages, or it will not survive private or fresh data.
Trusting an unvalidated LLM judge. Faithfulness and answer-relevance scores come from a model with its own biases (it can favor longer, more confident text). Calibrate against human labels on your own data before you gate releases on it.
Building the eval set with the same model and prompts you are testing. Self-preference and contamination inflate every score. Author gold answers and relevance labels independently.