RAG Evaluation: Faithfulness, Answer Relevancy, Context Precision, and Context Recall

For a RAG pipeline, evaluate retrieval and generation separately. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) The four metrics that make this separation concrete each answer one specific question:

Faithfulness asks: is the answer grounded in the retrieved context? [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) It measures the generator's honesty about its own evidence — whether every claim in the answer can be traced back to something the retriever actually fetched, independent of whether that fetched evidence is itself true.

Answer relevancy asks: does the answer address the question? [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) A perfectly grounded answer that wanders off-topic, or pads itself with tangentially related but unrequested content, scores poorly here even while scoring well on faithfulness.

Context precision asks: are the retrieved chunks relevant, with few distractors? [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) This is a retrieval-side question entirely — it never looks at the generated answer at all, only at what the retriever handed to the generator.

Context recall asks: did retrieval fetch all the needed information? [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) Also a pure retrieval-side question, and the more expensive of the two retrieval metrics to compute, because it requires knowing in advance what should have been retrieved.

[GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) Ragas provides these component-level metrics, and NVIDIA's RAG evaluation approach uses Ragas to measure how well retrieved contexts support generated answers — the tooling behind these four numbers is not this cert's own invention, it is a widely adopted open framework NVIDIA's stack builds on rather than replaces.

The column worth staring at is the last one. Two of the four metrics never even examine the generated answer — they can be computed the instant retrieval finishes, before generation has run at all, which is exactly why a well-built RAG evaluation pipeline can flag a retrieval problem before it ever produces a wrong answer to diagnose.

L1 — Intuition: an evidence locker and a witness, inspected separately

Picture a courtroom analogy instead of a pipeline diagram. Before a witness ever speaks, an evidence clerk assembles a box of documents relevant to the case — that box is what retrieval hands the generator. Two separate questions can be asked of the clerk's box before anyone hears testimony: did the box contain everything relevant to the case (context recall), and is the box mostly relevant material or is it padded with unrelated paperwork (context precision)? Only after the box is assessed does the witness speak, and two different questions apply to the testimony itself: did the witness stick strictly to what was in the box (faithfulness), or did the witness wander in and answer a question nobody asked (answer relevancy)? A jury told only "the testimony was unconvincing" learns nothing about whether the clerk's box was incomplete or the witness embellished — exactly the ambiguity a single end-to-end RAG score leaves unresolved, and exactly what these four separate questions exist to remove.

L2 — Mechanism: the formula behind each metric

Context recall, at the claim level, is the fraction of ground-truth claims that the retrieved context actually supports:

context recall = (number of ground-truth claims supported by retrieved context)
                  ────────────────────────────────────────────────────────────
                  (total number of claims in the ground-truth answer)

Computing this requires a human or a curated process to have already decided what the ground-truth answer's claims are and which passages support them — this is the single most expensive label in RAG evaluation, because it requires knowing in advance what should have been retrieved, not just judging what was.

Context precision is the fraction of retrieved chunks that are actually relevant:

context precision = (number of relevant retrieved chunks)
                     ───────────────────────────────────
                     (total number of retrieved chunks)

A position-aware variant additionally rewards placing relevant chunks near the top of the retrieved set rather than anywhere in it, which matters because context assembled for a generator is read in order, and evidence buried deep in a long context window is used less reliably than evidence near the top or bottom.

Faithfulness decomposes the generated answer into individual atomic claims, then checks each claim against the retrieved context:

faithfulness = (number of answer claims entailed by the retrieved context)
               ─────────────────────────────────────────────────────────
               (total number of claims in the generated answer)

The entailment check — does this specific passage support this specific claim — is typically delegated to an LLM judge (the mechanism M6-03 covers), presented with the claim and the retrieved passage and asked for an entailment verdict, though a dedicated natural-language-inference model or a human reviewer can perform the identical check at higher cost or higher reliability respectively.

Answer relevancy measures whether the answer addresses the question asked, independent of whether it is grounded:

answer relevancy ≈ a judge's rating of how directly the answer addresses the question,
                    or the similarity between the question asked and question(s) a judge
                    reconstructs from reading only the answer

One common implementation has a judge model generate the question(s) the answer would be a good response to, then measures how closely those reconstructed questions match the question actually asked — an answer that drifted off-topic reconstructs into a different question than the one posed, which is exactly the signal this metric is built to catch.

L3 — Why context precision and context recall must stay two separate numbers, not one

⚠️ UNVERIFIED the specific reasoning below extends the source material's stated distinction rather than quoting it directly, but the shape of the trap it guards against is directly named: context precision and context recall assess retrieval, not the final answer text, and collapsing them into one blended "retrieval quality" score destroys exactly the information that makes the four-metric decomposition useful. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) A retriever can have perfect recall (everything needed was fetched) and terrible precision (it also fetched nine irrelevant chunks for every relevant one) — a noisy-but-complete retriever. A different retriever can have perfect precision (every chunk it fetched was relevant) and terrible recall (it fetched three of the ten passages actually needed) — a clean-but-incomplete retriever. These are different engineering problems with different fixes: a noisy-but-complete retriever needs reranking or tighter chunk filtering to raise precision without losing recall; a clean-but-incomplete retriever needs a larger retrieval budget, better recall-oriented retrieval (a wider k, hybrid search closing a lexical gap), or better chunking so the missing content becomes findable at all. A single blended number showing "retrieval is at 60%" in either scenario points a team at the wrong fix roughly half the time, which is precisely why the source material insists on keeping the two numbers apart rather than averaging them into a single retrieval score.

The row worth re-reading twice is faithfulness's. It is the only metric in this table that needs no external ground-truth answer at all, because the retrieved context supplies its own reference — a structural advantage specific to RAG that unaided generation evaluation does not have, and it is also the only metric here built to check whether a claim is actually supported by evidence rather than merely similar in wording to some reference text.

Constructed example, built purely to walk the arithmetic step by step — the passages, claims, and judgments below describe no real product or real corpus.

Question: "What is the warranty period for the industrial sensor, and does it cover water damage?"

Ground truth, established by a human reviewer reading the full policy corpus: two facts are needed to answer completely.

• GT-1: The sensor carries a 24-month warranty. → lives in passage WARRANTY-DOC §3.1
• GT-2: Water damage is explicitly excluded from coverage. → lives in passage WARRANTY-DOC §5.4

Retrieved context, top-5 chunks the retriever returned:

Note that WARRANTY-DOC §5.4 — the passage stating water damage is excluded — was not retrieved.

Generated answer: "The industrial sensor carries a 24-month warranty. Water damage is covered under this warranty at no additional cost. Warranty claims must be filed within 30 days of discovering the defect."

Step 1 — context recall

GT claims supported by the retrieved context:
  GT-1 (24-month warranty)              → supported by rank-1 chunk    ✓
  GT-2 (water damage excluded)          → not present in any chunk     ✗

context recall = 1 / 2 = 0.500

Context recall = 0.500. Half of the evidence needed to answer completely was never retrieved — and this is visible before the generated answer is even read.

Step 2 — context precision

relevant retrieved chunks = 1 (rank 1 only)
total retrieved chunks    = 5

context precision (plain) = 1 / 5 = 0.200

The lone relevant chunk sits at rank 1, so a position-aware variant would credit good ordering (precision@1 = 1.000) even while the plain score of 0.200 flags that four of five context slots were wasted on irrelevant material — the same "ordering fine, density poor" pattern worth checking whenever plain and position-aware precision diverge sharply.

Step 3 — faithfulness

Decompose the generated answer into atomic claims:

faithfulness = supported claims / total claims = 2 / 3 = 0.667

Faithfulness = 0.667. One of three claims is unsupported, and it is the highest-stakes one: the model asserted water-damage coverage that no retrieved passage backs up, and — worse than an omission — the true answer is the opposite of what was asserted. This is not a case of the model inventing a number from nothing; it is a case of the model filling a gap in its retrieved evidence with a plausible-sounding guess, which faithfulness catches precisely because it checks the claim against what was actually retrieved, not against whether the claim merely sounds reasonable.

Step 4 — answer relevancy

The question asked two things: the warranty length, and whether water damage is covered. The answer addresses both (the second one incorrectly, but it is on-topic) and adds one additional claim about the filing deadline that was not asked about.

claims addressing the question asked = C1, C2   → 2
total claims in the answer            = 3
answer relevancy = 2 / 3 = 0.667

Answer relevancy = 0.667. The filing-deadline sentence is faithful (it is genuinely supported by rank-4's chunk) and simultaneously irrelevant to what was asked — a real, common pattern where a model pads an answer with whatever else happened to be in its retrieved context. This is worth noting explicitly: padding can raise or hold faithfulness steady while lowering relevancy, because the two metrics are checking different things and can move independently, sometimes in opposite directions.

Step 5 — read all four numbers as one diagnosis

A single "is this answer correct" verdict says no, and stops there. The four-number decomposition names two independent engineering tickets. Retrieval ticket: WARRANTY-DOC §5.4 was never fetched for a question that explicitly asked about water damage — investigate whether chunking separated the exclusion clause from language that would make it retrievable under this query's wording, whether the embedding model captures the semantic link between "water damage" and "excluded," or whether k=5 is simply too small for a two-part question needing evidence from two different sections. Generation ticket: even with the water-damage passage absent, the model should not have asserted coverage — the correct behavior was to answer the warranty-length part confidently and state that the retrieved context does not address water damage specifically, rather than filling the gap with an assertion. Both tickets are real, both are independently fixable, and a single wrong-answer verdict would have surfaced neither.

Step 6 — what changes after the retrieval fix

Suppose the team improves chunking so WARRANTY-DOC §5.4 is retrievable, and reruns the same question. The exclusion passage now appears at rank 2, and the two off-topic chunks (ranks 3 and 5) are replaced by reranking:

context recall    = 2 / 2 = 1.000
context precision = 2 / 5 = 0.400   (two relevant of five retrieved, still room to tighten k)

If the generator, now given the exclusion passage, produces "The sensor carries a 24-month warranty; water damage is explicitly excluded from coverage," then:

faithfulness      = 2 / 2 = 1.000
answer relevancy  = 2 / 2 = 1.000

Every number moved, and each movement is attributable to the specific change that caused it: context recall and precision moved because retrieval changed; faithfulness and relevancy moved because the generator received different evidence. If faithfulness had stayed low even after the retrieval fix — if the model still asserted coverage with the exclusion passage sitting directly in front of it — that would isolate the remaining defect to generation, not retrieval, with no ambiguity. That attributability, movement by movement, is exactly what a single end-to-end score can never provide.

Constructed scenario, illustrating a failure mode single-passage questions do not surface. All figures are invented for the arithmetic.

Question: "Which of the two data centers, East or West, has the lower average PUE, and by how much?"

Answering this correctly requires combining two separate facts from two separate documents — a genuinely multi-hop question, unlike the single-document lookups in section 4's example.

Ground truth, two facts needed:

• GT-1: East data center's average PUE is 1.35. → lives in FACILITIES-RPT-EAST §2.2
• GT-2: West data center's average PUE is 1.48. → lives in FACILITIES-RPT-WEST §2.2

Retrieved context, top-4 chunks:

FACILITIES-RPT-WEST §2.2, the passage with West's actual PUE figure, was never retrieved — the retriever found West-related content but not the specific West fact this question needs, a distinct failure from the single-document recall gap in section 4.

Generated answer: "East data center has a lower PUE at 1.35. West data center's PUE is approximately 1.42, so East is more efficient by about 0.07."

Step 1 — context recall on a multi-hop question

GT claims supported by retrieved context:
  GT-1 (East PUE = 1.35)  → supported by rank-1 chunk    ✓
  GT-2 (West PUE = 1.48)  → not present in any chunk      ✗

context recall = 1 / 2 = 0.500

Context recall = 0.500 — identical to section 4's single-hop example's score, but for a structurally different reason. In section 4, the missing fact simply was not surfaced by the retriever at all for any related query. Here, the retriever did surface content about the right entity (West's cooling retrofit is genuinely about West), just not the right fact about that entity. A recall score alone cannot distinguish "wrong entity entirely" from "right entity, wrong fact" — both register as a claim that went unsupported — which is exactly why the diagnostic step that follows recall's number matters as much as the number itself.

Step 2 — faithfulness on a fabricated-but-plausible number

Claim              Text                                    Supported by retrieved context?
C1  East PUE is 1.35                                        Yes — rank-1 chunk states it
C2  West PUE is approximately 1.42                           No — no retrieved chunk states any PUE figure for West
C3  East is more efficient by about 0.07                     No — this is arithmetic performed on an unsupported number (C2)

faithfulness = 1 / 3 = 0.333

Faithfulness = 0.333, sharply lower than section 4's 0.667. This is a more dangerous failure than the water-damage case: the model did not just fill a gap with a plausible assertion, it fabricated a specific, precise-sounding number (1.42) with no supporting evidence whatsoever, and then performed correct-looking arithmetic on top of that fabricated number, producing a claim (C3) that inherits the fabrication while sounding like an independently verified calculation. A claim built from correct arithmetic on a fabricated input is still a fabrication — faithfulness's claim-by-claim check catches this because it traces C3 back to C2, not because it re-does the arithmetic itself.

Step 3 — what this pair of examples together demonstrates

Section 4 and this section both produced a context recall of 0.500, and both point at a genuine retrieval gap — but the faithfulness scores diverge sharply (0.667 versus 0.333) because the generation-side failure differed in kind. This is the practical argument for computing all four numbers rather than stopping once one looks informative: two RAG failures can share an identical retrieval-side score while representing very different levels of downstream risk, and only the generation-side metrics reveal that difference. A monitoring dashboard that alerts only on context recall dropping below some threshold would have treated these two incidents as equally severe; reading faithfulness alongside recall shows they are not.

Two rules complete the practice. Always compute faithfulness first among the generation-side metrics, since it needs no ground-truth answer and catches the failure that damages user trust most directly. Never report context precision and context recall as one blended number — section 2's L3 discussion is the reason why, and it is this lesson's single most exam-relevant point.

Faithfulness asks whether the answer's claims are supported by the retrieved context; correctness asks whether the answer matches the actual truth. The two come apart whenever the retrieved evidence itself is wrong, stale, or incomplete — a fully faithful answer built entirely on an outdated policy document is a wrong answer that would still score perfectly on faithfulness, because faithfulness never checks the retrieved context against anything outside the pipeline. This is exactly the pattern in section 4's worked example after the retrieval fix in step 6: once the correct passage was retrieved and the generator used it faithfully, faithfulness and correctness aligned — but that alignment depended on the corpus itself being right, which is a separate concern from anything faithfulness alone can verify.

Check whether the required evidence was present in the retrieved context before reading the generated answer at all. If the evidence is absent, no prompt change or model swap can fix the answer, because the information was never available to generate from — the fix lives entirely on the retrieval side: chunking, the embedding model, a larger retrieval budget, or reranking. If the evidence is present and the answer is still wrong, the fix lives on the generation side: prompt grounding, an explicit citation requirement, better context ordering, or a stronger model. The worked example in section 4 demonstrates this ordering rule directly — context recall's 0.500 score identified a retrieval-side gap before the faithfulness score even needed to be computed to know something had gone wrong.

Partially, and further than most teams expect starting out. Faithfulness and answer relevancy need no ground-truth answer whatsoever — the retrieved context is faithfulness's own reference, and the question itself is answer relevancy's reference — so a meaningful evaluation harness measuring the generation side can be stood up before any labeling project begins. What cannot be measured without labels is context recall, which requires knowing in advance which passages should have been retrieved, and any correctness metric requiring a gold answer. If a team can only afford to label one thing, retrieval ground truth is the highest-value purchase, because it is what unlocks distinguishing a retrieval failure from a generation failure — the central capability this whole lesson is built around.

• Four numbers, two stages. Context recall and context precision measure retrieval; faithfulness and answer relevancy measure generation.
• Faithfulness needs no ground-truth answer — the retrieved context is its own reference, a structural advantage specific to RAG.
• Faithfulness is not correctness. A faithful answer built on wrong or stale retrieved evidence is still a wrong answer, and that pattern points at the corpus, not the model.
• Context precision and context recall must stay two separate numbers, never averaged into one retrieval score — a noisy-but-complete retriever and a clean-but-incomplete retriever need opposite fixes and look identical once blended.
• Check whether the evidence was retrieved before judging the generated answer. In the worked example, context recall's 0.500 flagged a retrieval gap before faithfulness needed to be computed at all.
• In the worked example, one query yielded context recall 0.500, context precision 0.200, faithfulness 0.667, and answer relevancy 0.667 — and named two independent, differently-fixed engineering tickets a single "wrong" verdict would have hidden entirely.
• Padding can raise or hold faithfulness steady while lowering relevancy — the two generation-side metrics can move in opposite directions, which is exactly why both are reported.

⭐ THE EARNED INSIGHT

A single "is this answer right" verdict treats a RAG pipeline as one opaque box; the four-metric decomposition treats it as exactly what it is — two separate systems, retrieval and generation, handing work to each other across a boundary — and every trap in this lesson is a variant of the same mistake: collapsing that boundary back into one number and losing the one piece of information, which stage actually broke, that made measuring the two systems separately worth doing in the first place.

This module has now covered every layer this cert's Evaluation domain names: automatic metrics with hard boundaries (M6-01, M6-02), the judge and human-review layer that catches what those metrics cannot (M6-03), the scalable framework that runs all of it consistently across models and platforms (M6-04), and now the stage-specific decomposition that applies the exact same "don't collapse a diagnosis into one number" discipline to RAG specifically. What comes next moves from measuring a model's quality to putting a validated model in front of real traffic — the deployment concerns this cert's next domain covers, where the questions shift from "is this good" to "is this actually running correctly, reliably, and efficiently once it is live."