RAG Evaluation Metrics: Faithfulness, Answer Relevance, Context Precision and Context Recall

RAG evaluation metrics are per-item scores computed from the four objects a RAG system produces for each query: the question, the retrieved context, the generated answer, and — when you have one — a ground-truth answer or a list of the passages that should have been retrieved.

The row that makes RAG evaluation unusually tractable is the third one. Faithfulness needs no ground-truth answer, because the retrieved context serves as the reference. That is a genuine structural advantage of RAG over unaided generation: the system hands you the evidence it claims to have used, so you can check the answer against it without a human writing a gold answer first. Every other lesson in this module has been fighting the cost of references; here the pipeline supplies one for free.

The four core metrics, placed on the pipeline:

question
   │
   ├──► RETRIEVAL ─────► retrieved context      ◄── context recall, context precision
   │                          │                     (did we fetch the right evidence,
   │                          │                      and not too much junk?)
   └──► GENERATION ◄──────────┘
             │
             ▼
          answer                                ◄── faithfulness, answer relevance
                                                    (is every claim supported,
                                                     and does it answer the question?)

07-09 walks the full nine-stage pipeline these metrics instrument, and 07-10 teaches the same retrieval-versus-generation split as a debugging discipline before any metric exists. This lesson is where that split becomes four numbers you can put on a dashboard.

L1 — Intuition: two gates, two questions each

Think of RAG as two gates in series. The retrieval gate either does or does not put the needed evidence in front of the model. The generation gate either does or does not use that evidence correctly. An answer can fail at either gate, and the failures look identical from the outside — a wrong answer.

• Retrieval gate, question 1: was the evidence there? → context recall
• Retrieval gate, question 2: was it buried in junk? → context precision
• Generation gate, question 1: did the answer stick to the evidence? → faithfulness
• Generation gate, question 2: did the answer address the question? → answer relevance

Those four cover the space. An answer that scores well on all four is, by construction, grounded, on-topic, and built from evidence that was correctly fetched.

L2 — Mechanism: how each number is actually produced

Context recall. Requires knowing which passages (or which ground-truth claims) should have been retrieved.

context recall = (number of ground-truth claims supported by the retrieved context)
                 ─────────────────────────────────────────────────────────────────
                 (total number of claims in the ground-truth answer)

An equivalent passage-level form is recall@k from 09-05: of the relevant documents that exist, what fraction appear in the top k. The claim-level form is often more useful because a multi-part question may need evidence from two documents, and passage-level recall@k treats "found one of two" as partial credit without telling you the answer is now impossible.

Context precision. Of the retrieved chunks, what fraction are relevant — and, in the position-aware variants, are the relevant ones near the top?

context precision (plain)        = relevant retrieved chunks / total retrieved chunks
context precision (position-aware) = mean of precision@i over the ranks i where a relevant chunk appears

The position-aware form matters because of the lost-in-the-middle effect covered in 07-08: relevant evidence placed in the middle of a long context is used less reliably than the same evidence placed at the top. A retriever with good plain precision but bad ordering will underperform.

Faithfulness (groundedness). Decompose the answer into atomic claims, then check each claim against the retrieved context.

faithfulness = (number of answer claims entailed by the retrieved context)
               ──────────────────────────────────────────────────────────
               (total number of claims in the answer)

Three ways to do the entailment check, in increasing cost and reliability: a natural-language-inference model classifying each claim as entailed / contradicted / neutral against each passage; an LLM judge given the claim and the context and asked whether the context supports it (09-10); or a human. A faithfulness score of 1.0 does not mean the answer is correct — it means every claim traces to the retrieved context. If the retrieved context is wrong, a perfectly faithful answer is a perfectly wrong answer. Faithfulness measures the generator's honesty about its evidence, not the evidence's truth.

Answer relevance. Does the answer address the question, without padding or drifting?

answer relevance ≈ similarity between the question asked and question(s) reconstructed
                   from the answer, or a judge's rating of responsiveness

One common implementation asks a model to generate the questions the answer would be a good answer to, then measures embedding similarity between those and the actual question. Another simply asks a judge to rate responsiveness against a rubric. Either way it catches the specific failure where a system produces a fluent, faithful essay about the general topic without answering what was asked.

Answer correctness, when you do have a ground-truth answer, is the fifth number and combines factual agreement with the reference and semantic similarity to it. It is the closest thing to an end-to-end score, and it is exactly the number that cannot tell you where a failure happened — which is why it supplements the four rather than replacing them.

L3 — Depth: what the decomposition buys and what it still misses

The diagnostic matrix. The value of four numbers instead of one is that their pattern names the failure.

That table is the most operationally valuable thing in this lesson. It converts "the answers are bad" into a specific engineering ticket.

What the decomposition still misses. Four gaps to be honest about:

1. Faithfulness ≠ correctness. Already stated, and worth repeating because it is the most common misreading. A faithful answer to bad evidence is wrong.
2. Claim decomposition is itself a judgement. How you split an answer into "atomic claims" changes the denominator and therefore the score. Two implementations of faithfulness will disagree, and neither is wrong. Pin your implementation and record it, exactly as 09-04 requires for BERTScore's encoder.
3. Most implementations use an LLM to do the checking, which imports all of the judge biases in 09-10 and the non-determinism in 09-11. A faithfulness score is not a deterministic measurement unless the entailment step is a fixed NLI model with a fixed seed.
4. Context recall needs ground truth about retrieval, which is the most expensive label in the set — someone must decide which passages should have been fetched. Many teams skip it and then cannot distinguish retrieval failures at all, which defeats the purpose of the decomposition. If you can only afford one labelled column, make it this one.

A fifth, subtler point: these metrics are per-item and should be reported per slice. A RAG system typically has excellent context recall on single-document factual questions and poor recall on multi-hop questions requiring two sources. Aggregated, that looks like a mediocre retriever; per slice, it names a specific capability gap. The stratification discipline from 09-01 applies directly.

Two columns carry the exam-relevant content. The "localises the failure" column is why RAG-specific metrics exist: every generic text metric answers "how good was the answer?" and none answers "which stage broke?". The "detects a fabricated claim" column has exactly one unambiguous yes, faithfulness — because it is the only metric in the table that compares the answer against the evidence rather than against a reference answer or nothing at all. That single property is why grounding-plus-faithfulness-measurement sits at the top of the hallucination mitigation ladder in 09-12.

Note also what is not in the table: perplexity. A reference-free model-confidence number tells you nothing about groundedness, for the reason 09-02 gives — a fabrication is high-probability text.

Constructed example. All passages, claims and judgements below are invented for the arithmetic; nothing here is a measurement of a real system.

Question: "What is the notice period for terminating a Standard plan, and is there an early-termination fee?"

Ground truth (from the corpus, established by a human): two facts are needed.

• GT-1: Standard plan requires 30 days' written notice. → lives in passage POLICY-A §4.2
• GT-2: There is no early-termination fee on Standard. → lives in passage POLICY-B §7.1

Retrieved context, top-5 chunks:

Note POLICY-B §7.1 — the passage stating that Standard has no early-termination fee — was not retrieved.

Generated answer: "Terminating a Standard plan requires thirty days' written notice. An early-termination fee of 15% applies. Plans also renew automatically on their anniversary date."

Step 1 — context recall

GT claims supported by the retrieved context:
  GT-1 (30 days' notice)          → supported by rank-1 chunk    ✓
  GT-2 (no early-termination fee) → not present in any chunk     ✗

context recall = 1 / 2 = 0.500

Context recall = 0.500. Half the evidence needed was never fetched. This is a retrieval failure and it is already visible before we look at the answer at all.

Step 2 — context precision

relevant retrieved chunks = 1 (rank 1)
total retrieved chunks    = 5

context precision (plain) = 1 / 5 = 0.200

Position-aware form: the only relevant chunk is at rank 1, so precision@1 = 1/1 = 1.000, and averaging over the ranks at which relevant chunks appear gives:

context precision (position-aware) = 1.000

Context precision = 0.200 plain, 1.000 position-aware. Read together: the retriever put the one good chunk at the top (good ordering) but filled the remaining four slots with irrelevant material (poor density). Four of five slots of context budget were wasted, and two of those wasted chunks — the Enterprise notice period and the Enterprise fee — are actively misleading, because they discuss the same concepts for a different plan. That is the worst kind of retrieval noise: plausible, adjacent, wrong.

Step 3 — faithfulness

Decompose the answer into atomic claims:

faithfulness = supported claims / total claims = 2 / 3 = 0.667

Faithfulness = 0.667. One of three claims is unsupported — and it is the one that will cost a customer money. Note the mechanism precisely: the model did not invent "15%" from nothing. It took a real number from a real retrieved passage and misattributed it to the wrong plan. This is the most common realistic faithfulness failure and it is invisible to any metric that compares the answer to a reference summary of the corpus, because the words are all corpus words.

Step 4 — answer relevance

The question had two parts: notice period, and whether a fee exists. The answer addresses both parts (part two incorrectly, but it does address it) and adds one unrequested claim about auto-renewal.

Scoring responsiveness as the fraction of the answer's claims that address the question asked:

claims addressing the question = C1, C2  → 2
total claims                   = 3
answer relevance = 2 / 3 = 0.667

Answer relevance = 0.667. The auto-renewal sentence is faithful and irrelevant — a real pattern, where a model pads an answer with whatever else was in the context. Padding degrades relevance while improving faithfulness (the padding is grounded), which is a good illustration of why you need both numbers: they can move in opposite directions.

Step 5 — read the four numbers as a diagnosis

Now compare with what a single end-to-end score would have said. A human grader asked "is this answer correct?" says no, and stops. A ROUGE score against a reference answer would be moderate, because the words overlap heavily. Neither tells the team what to do.

The four-number decomposition names two independent tickets:

1. Retrieval ticket (primary). POLICY-B §7.1 was not retrieved for a query that explicitly asked about fees. Likely causes: chunking split the fee clause from its plan heading so the chunk lost the word "Standard" (06-02); or the query's fee terminology does not match the passage's wording, which is a lexical gap that hybrid search would close (07-06); or k=5 is too small for a two-part question. Also, four of five slots were noise, so reranking would raise density (07-07).
2. Generation ticket (secondary). Even with the evidence absent, the model should not have asserted a fee. The correct behaviour was to answer the notice-period part and state that the context does not address fees for Standard — the abstention behaviour from 07-11. Adding a citation requirement would have surfaced the problem, because the model would have had to cite the Enterprise clause for a Standard claim.

One query, two independent fixes, correctly attributed. That is the entire argument for the decomposition, and it is why 07-10 teaches the same split as a debugging habit before the metrics arrive.

Step 6 — what happens after the retrieval fix

Suppose reranking and hybrid search are added, and POLICY-B §7.1 now appears at rank 2 while the two Enterprise chunks drop out. Re-run:

context recall            = 2 / 2 = 1.000
context precision (plain) = 2 / 5 = 0.400

If the generator now says "Standard requires thirty days' written notice and there is no early-termination fee," then:

faithfulness     = 2 / 2 = 1.000
answer relevance = 2 / 2 = 1.000

Every number moved, and you can attribute each movement to the change that caused it: context recall and precision moved because retrieval changed; faithfulness and relevance moved because the generator was given different evidence. Had faithfulness not moved — had the model still asserted a fee with the correct passage in front of it — you would know instantly that the remaining defect is in generation, not retrieval. That attributability is the property a single score cannot have.

Two rules complete the procedure. Always compute faithfulness, because it is the cheapest of the four — no ground-truth answer needed — and it detects the failure that damages users most. And always report the four together with per-slice breakdowns, because a system's retrieval is usually strong on one question type and weak on another, and the aggregate hides that.

RAG is the highest-weight application topic in this certification. It appears in the Core ML domain's objectives 1.3 ("Build LLM use cases such as retrieval-augmented generation (RAG), chatbots, and summarizers") and 1.4 ("Curate and embed content datasets for RAGs"), in the Software Development domain's 4.2, and in the Experimentation domain's suggested-reading list, which names evaluating RAG applications explicitly. Published candidate reports also give RAG a strategic role: in scenario questions, when one option proposes building a RAG solution, it is frequently the keyed answer — a heuristic taught in 07-12 alongside its counter-cases.

The objective-numbering defect, stated where the objectives are cited. The official study guide prints the Experimentation domain's objectives as 3.1–3.5, and those five lines are a verbatim duplicate of the Data Analysis domain's 2.1–2.5: data-mining awareness, comparing models using statistical performance metrics, conducting data analysis under supervision, creating graphs and charts, identifying relationships and trends. Read literally, 22% of the exam has objectives that describe charts and data mining rather than model evaluation and RLHF. The Experimentation section's own scope statement — "the study of how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)" — and its suggested-reading list, which names evaluating RAG applications, hallucinations in LLMs, GLUE and cross-validation, both describe the real content. The derived scope governs. For this lesson the applicable objectives are the duplicated pair 2.2 / 3.2 ("compare models using statistical performance metrics"), 1.3 and 1.4 for RAG itself, 4.5 ("monitor functioning of data collection, experiments, and other software processes"), and from Trustworthy AI 5.1, since grounding and citation are transparency instruments.

Question phrasings to expect:

• "A RAG system produces an answer containing information not present in the retrieved documents. Which metric captures this?" → Faithfulness (groundedness).
• "How do you determine whether a RAG failure is a retrieval problem or a generation problem?" → Measure retrieval and generation separately: check whether the needed evidence was in the retrieved context (context recall) before judging the answer.
• "Which RAG metric requires no ground-truth answer?" → Faithfulness — the retrieved context serves as the reference. Answer relevance also needs no ground truth.
• "A RAG answer is fully grounded in the retrieved context but still wrong. What is the most likely cause?" → The retrieved source is itself wrong, stale, or about a different entity; faithfulness measures grounding, not truth.
• "What does context recall measure?" → The fraction of the evidence needed to answer that was actually retrieved.
• "Retrieval returns the relevant document at rank 1 plus four irrelevant ones. Which metric is low?" → Context precision; recall and MRR are fine.
• "Which mitigation most directly reduces hallucination in a generative system?" → Grounding the answer in retrieved sources and requiring citations. This is the RAG-as-mitigation answer, and 09-12 gives the whole ladder.
• "Why is BLEU or ROUGE insufficient for RAG evaluation?" → They compare the answer to a reference string and cannot tell whether the answer is supported by the retrieved evidence or which stage failed.

Distractor families. (1) A single end-to-end score offered as sufficient — plausible and wrong, because it cannot localise. (2) Faithfulness conflated with correctness — the most conceptually interesting distractor; a faithful answer to bad evidence is wrong. (3) A surface-overlap metric offered for groundedness — ROUGE or BERTScore against a reference answer, which cannot see the evidence at all. (4) Perplexity or model confidence offered as a hallucination detector — ruled out by 09-02. (5) Context precision and context recall swapped — precision is about the retrieved set's purity, recall about its completeness. (6) Fine-tuning offered as the fix for a retrieval failure — a category error the decomposition exists to prevent, and the counterpart of the RAG-versus-fine-tuning decision in 11-08.

Faithfulness asks whether the answer's claims are supported by the retrieved context; answer correctness asks whether the answer matches the truth. They come apart in both directions, and both directions are diagnostic.

Faithful but incorrect happens when the retrieved evidence is itself wrong, stale, or about a different entity. In the worked example a fully faithful answer would still have been incomplete, because the passage stating the true fee position was never retrieved. If your corpus contains a superseded policy version, a perfectly faithful answer will quote it. The fix is corpus curation and freshness (12-12), authority signals (07-03), and de-duplication so an obsolete copy does not win retrieval (06-04) — none of which is a model problem.

Correct but unfaithful happens when the model answers from its pretraining knowledge rather than from the context. The answer is right, which feels fine, and it is a genuine defect: the system is not doing what it claims, so its provenance story is false, its citations are decorative, and the moment the parametric knowledge is stale or wrong the same mechanism produces a confident error with no warning. Low faithfulness with high correctness on a sample is a signal that the grounding instruction is not binding — usually fixed by requiring citations for every claim and rejecting uncited claims.

Measure both. Faithfulness is cheap and needs no ground truth; correctness is expensive and needs it. A practical split is faithfulness on every item in CI, correctness on a labelled subset quarterly.

Check whether the required evidence was present in the retrieved context, before you look at the answer. That single ordering rule resolves most RAG debugging.

• Evidence absent → retrieval failure. Nothing you do to the prompt or the model can fix it; the information was not in the room. Investigate chunking (did the chunk lose the heading that made it findable?), the embedding model (does it handle your domain vocabulary?), lexical gaps (would BM25 have found it?), k (is the budget too small for a multi-part question?), and metadata filters (did permissions or a filter exclude it?). 07-01 through 07-07 are the fix menu.
• Evidence present, answer wrong → generation failure. Now prompt and model changes are the right lever: an explicit instruction to answer only from the context, a citation requirement per claim, moving the key passage to the top or bottom rather than the middle (07-08), and an explicit permission to say the context does not contain the answer (07-11).
• Evidence present, answer right, user still unhappy → look at relevance and formatting, or at the corpus's own correctness.

The metric version of this rule is the diagnostic matrix in §2. The manual version is 07-10. They are the same discipline; the metrics just let a build server apply it without a human in the loop.

Partly, and much further than people expect. Faithfulness and answer relevance need no ground truth at all — the retrieved context is the reference for one, and the question is the reference for the other. That means you can stand up a meaningful RAG evaluation harness on day one, before any labelling project, and it will catch the two failures that most damage user trust: fabricated claims and non-answers.

What you cannot get without labels: context recall, which requires knowing which passages should have been retrieved, and answer correctness, which requires a gold answer. Both are worth buying, and if you can only afford one, buy retrieval ground truth — it is what unlocks the retrieval-versus-generation split, and the split is what makes every subsequent debugging session cheap.

A pragmatic staged plan: week one, faithfulness and answer relevance on a frozen 100-item set (09-01); week two, add retrieval ground truth for the same items to unlock context recall and precision; quarterly, add ground-truth answers for a labelled subset to measure correctness and to validate that your judge-based faithfulness scores still agree with human judgement (09-03). Note the last clause: judge-based metrics need periodic human calibration or they drift, and the agreement statistic that calibrates them is the one from 09-03.

Faithfulness, answer relevance, context recall and context precision — all four, on the frozen evaluation set, with thresholds set from the current baseline rather than from an aspiration. Three design rules make the gate usable rather than annoying.

Gate on the stage-specific metric that the change could plausibly affect. A chunking change should be gated hard on context recall and precision and loosely on faithfulness; a prompt change is the reverse. Gating everything equally means every change fights every threshold.

Set thresholds as a no-regression band, not an absolute target. "Context recall must not drop more than 3 points below the recorded baseline" is enforceable; "context recall must exceed 0.9" fails the build on day one and gets disabled by the second week.

Budget the runtime. Judge-based faithfulness costs an LLM call per claim per item, so a 100-item set can be hundreds of calls, which is minutes and real money. Options: run the cheap deterministic retrieval metrics on every commit and the judge-based generation metrics nightly; or use a small NLI model for the entailment step, which is deterministic and fast. 10-04 covers the CI mechanics; the point here is that the choice of which metric runs at which cadence is a design decision driven by cost and by which stage the change touched.

1. Four numbers, two stages. Context recall and context precision measure retrieval; faithfulness and answer relevance measure generation.
2. The decomposition exists to localise the failure. A single end-to-end score cannot tell a retrieval bug from a generation bug, and the two have opposite fixes.
3. Faithfulness needs no ground-truth answer — the retrieved context is the reference. This is RAG's structural evaluation advantage.
4. Faithfulness is not correctness. A faithful answer built on wrong or stale evidence is a wrong answer, and that pattern points at the corpus, not the model.
5. Check whether the evidence was present before judging the answer. That ordering rule resolves most RAG debugging.
6. Context recall is the most valuable label you can buy, because it is what unlocks the retrieval-versus-generation split.
7. In the worked example, one query yielded context recall 0.500, plain context precision 0.200, faithfulness 0.667 and answer relevance 0.667 — and named two independent engineering tickets a single "wrong" verdict would have hidden.
8. Cross-entity misattribution is the realistic faithfulness failure: real numbers from real passages, applied to the wrong plan. Claim-level checking and citation requirements catch it; surface metrics never will.
9. Padding raises faithfulness and lowers relevance. Report both; they can move in opposite directions.
10. Judge-based RAG metrics inherit judge bias and non-determinism (09-10, 09-11). Pin the model, version the prompts, and calibrate against human labels periodically (09-03).
11. Report per slice. Single-hop and multi-hop questions have very different retrieval profiles, and the aggregate hides the gap.

Everything so far has assumed a single frozen evaluation set and a single number per run. That is the right design for measuring a system, but it wastes data when you are comparing models or configurations on a limited labelled set — and it gives you exactly one estimate, with no sense of how much that estimate would wobble if you had drawn a different sample. There is a resampling discipline that addresses both problems, and it is also the technique the official objectives name by name.

Next: 09-08 covers cross-validation — k-fold, stratified k-fold, leave-one-out, grouped and time-series variants — how it produces a variance estimate as well as a mean, and the specific situations in which applying it to an LLM evaluation is either impossible or actively misleading.