How to Choose an Evaluation Metric: Loss Functions, R², Precision vs Recall, and Retrieval Metrics

Choosing an evaluation metric means selecting the single number, or small set of numbers, that changes when your system's most costly failure occurs — and that does not change much otherwise. A metric is a proxy for a harm. If you cannot state the harm, you cannot check the proxy, and the proxy will drift away from the harm without telling you.

The selection has four inputs:

And the output is a metric contract: a named metric, its exact configuration, the frozen dataset it runs on, and the threshold at which it gates a decision. Everything vaguer than that is a dashboard.

The four metric families you must be able to place instantly:

• Loss functions — cross-entropy for classification and language modelling, MSE/MAE for regression. These are what the model is trained to minimise, and they double as evaluation metrics.
• Classification metrics — accuracy, precision, recall, F1, ROC-AUC, computed from a confusion matrix.
• Regression metrics — MSE, RMSE, MAE, and R², the proportion of explained variance.
• Ranking / retrieval metrics — recall@k, precision@k, MRR, nDCG, for systems that return an ordered list.

Generation metrics — BLEU, ROUGE, BERTScore, perplexity, judges — sit alongside these and are covered in 09-06, 09-04, 09-02 and 09-10 respectively. This lesson gives the selection logic that spans all of them.

L1 — Intuition: the metric is the definition of the job

If you tell a team "maximise accuracy", you have told them that every error costs the same. If that is false — and for fraud detection, medical triage, content moderation, or retrieval it is always false — you have just instructed them to optimise the wrong thing, politely. The metric is not a measurement of the job; the metric is the operational definition of the job. Choose it as carefully as you would write the job description.

The corollary: when someone shows you a metric improvement, the first question is not "is it statistically significant?" (that is 09-09), it is "does that metric move when the thing we actually care about goes wrong?"

L2 — Mechanism: the decision procedure

Step 1 — Classify the task by output type.

Step 2 — Name the unacceptable error. Write one sentence: "The failure we cannot ship is ___." Then map it:

Step 3 — Check the cost and the cadence. A metric that costs a human two minutes per item cannot gate a build. Design a two-tier system: a cheap automatic metric on every commit, and an expensive human or judge metric on a schedule. Both run against the frozen evaluation set from 09-01.

Step 4 — Fix a threshold and a guardrail. A headline metric plus at least one guardrail metric that must not degrade. Optimising recall without a precision guardrail produces a system that flags everything; optimising quality without a latency guardrail produces a system nobody waits for. 12-10 covers latency as a first-class metric.

L3 — Depth: why single metrics fail and what to do instead

Every scalar metric is a projection of a multi-dimensional quality onto one axis, and projections lose information. Three structural consequences:

Goodhart's problem. Any metric you optimise hard enough stops measuring what it measured. Push ROUGE and you get summaries that copy the source. Push recall and you get a classifier that says yes. Push a reward model and you get reward hacking (11-07). The defence is a metric set — a headline plus guardrails — plus periodic human review to detect when the proxy has decoupled from the goal.

Aggregation hides asymmetry. A single number over a mixed population averages over subgroups, and a metric can improve overall while degrading for a subgroup. This is why per-slice reporting from 09-01 is not optional, and why fairness auditing in 13-04 is a per-slice activity by construction.

Thresholded metrics and rankings measure different things. Accuracy, precision, recall and F1 all depend on a decision threshold. ROC-AUC and PR-AUC do not — they summarise performance across all thresholds. So a question like "which model is better?" has two different answers depending on whether the threshold is fixed by the product or free to be tuned. Report both when you can: AUC for model comparison, thresholded precision/recall for the shipped configuration.

These three are constantly conflated, and the distinction is genuinely testable.

The key asymmetry: a loss must be differentiable, an evaluation metric need not be, and that is exactly why they differ. You cannot backpropagate through "F1" or "exact match" — they are step functions of the prediction — so you train on cross-entropy and evaluate on F1. When someone asks "why not just train on the metric we care about?", the answer is usually "because it has no useful gradient", and the workarounds (surrogate losses, reinforcement learning against a reward, differentiable relaxations) are more complex than the substitution suggests. RLHF is precisely the case where the thing you care about is non-differentiable — human preference — so you train a differentiable model of it and optimise that instead (11-06).

A moderation classifier flags policy-violating messages. All numbers below are constructed for the arithmetic. In a sample of 1,000 messages, 40 are genuine violations (4% positive rate — a realistic imbalance). The model flags 50 messages, of which 30 are true violations.

Step 1 — build the confusion matrix.

Step 2 — accuracy, and why it lies here.

accuracy = (TP + TN) / N = (30 + 940) / 1000 = 970 / 1000 = 0.970

97.0%. Now compute the accuracy of a model that flags nothing at all:

trivial accuracy = (0 + 960) / 1000 = 0.960

96.0%. Our model beats "do nothing" by one percentage point of accuracy while catching 30 of 40 violations. Accuracy on a 4%-positive problem is almost entirely a measurement of the negative class. This is the class-imbalance trap, and it is the single most common metric-selection error in applied ML.

Step 3 — precision and recall.

precision = TP / (TP + FP) = 30 / 50 = 0.600
recall    = TP / (TP + FN) = 30 / 40 = 0.750

Read them in words. Precision 0.600: of the messages we flagged, 60% really were violations — so 40% of flags waste a moderator's time or wrongly penalise a user. Recall 0.750: of the violations that existed, we caught 75% — so a quarter got through. Those are two entirely different business conversations, and accuracy told you neither.

The memory hooks worth carrying into the exam: precision is about the predictions (of what I flagged, how much was right — denominator is the predicted-positive column), and recall is about the reality (of what was really there, how much did I find — denominator is the actual-positive row). Recall is also called sensitivity or the true positive rate.

Step 4 — F1.

F1 = 2 × (precision × recall) / (precision + recall)
   = 2 × (0.600 × 0.750) / (0.600 + 0.750)
   = 2 × 0.450 / 1.350
   = 0.900 / 1.350
   = 0.6667

F1 = 0.667. The harmonic mean, not the arithmetic mean — the arithmetic mean would be 0.675, barely different here, but the harmonic mean's property is that it collapses when either component collapses. A model with precision 1.0 and recall 0.01 has an arithmetic mean of 0.505 and an F1 of 0.0198. That is the whole reason F1 uses the harmonic mean: you cannot buy a good F1 by sacrificing one side entirely.

Step 5 — Fβ when the errors are not equally costly. F1 weights precision and recall equally, which is a choice, and usually the wrong one.

Fβ = (1 + β²) × precision × recall / (β² × precision + recall)

β > 1 weights recall more; β < 1 weights precision more. With β = 2 (recall twice as important — appropriate if a missed violation is worse than a false alarm):

F2 = (1 + 4) × 0.600 × 0.750 / (4 × 0.600 + 0.750)
   = 5 × 0.450 / (2.400 + 0.750)
   = 2.250 / 3.150
   = 0.7143

With β = 0.5 (precision twice as important):

F0.5 = (1 + 0.25) × 0.600 × 0.750 / (0.25 × 0.600 + 0.750)
     = 1.25 × 0.450 / (0.150 + 0.750)
     = 0.5625 / 0.900
     = 0.6250

Same model, same confusion matrix: F2 = 0.714, F1 = 0.667, F0.5 = 0.625. The metric you choose changes the ranking of candidate models, which is why "which kind of wrongness is unacceptable" must be answered before you pick.

Step 6 — move the threshold and watch the trade-off. Lowering the decision threshold flags more messages. Suppose at a lower threshold the model flags 120 messages, catching 36 of the 40 violations:

TP = 36, FP = 84, FN = 4, TN = 876
precision = 36 / 120 = 0.300
recall    = 36 / 40  = 0.900
F1        = 2 × 0.300 × 0.900 / (0.300 + 0.900) = 0.540 / 1.200 = 0.450
accuracy  = (36 + 876) / 1000 = 0.912

Recall rose from 0.750 to 0.900; precision fell from 0.600 to 0.300; F1 fell from 0.667 to 0.450; accuracy fell from 0.970 to 0.912. This is the precision–recall trade-off in concrete numbers. No model change occurred — only the threshold. Which configuration is "better" is a product decision about the relative cost of a missed violation versus a wrongly flagged user, and there is no purely technical answer.

Step 7 — ROC-AUC versus PR-AUC on imbalanced data. ROC-AUC plots true positive rate against false positive rate across all thresholds and equals the probability that a randomly chosen positive is scored above a randomly chosen negative. Its weakness on rare positives: the false positive rate has the huge negative class in its denominator.

FPR at the strict threshold = FP / (FP + TN) = 20 / 960  = 0.0208
FPR at the loose threshold  = 84 / 960                    = 0.0875

Both FPRs look tiny — under 9% — even though the loose threshold's precision is a dismal 0.300. ROC-AUC can therefore look reassuring on a heavily imbalanced problem where precision is unusable. PR-AUC (precision–recall AUC) is the better summary when positives are rare, because both its axes have the positive class in the numerator and neither is dominated by the vast negative class. Rule to memorise: balanced classes → ROC-AUC is fine; rare positives → prefer PR-AUC and report thresholded precision/recall as well.

The exam objectives name "proportion of explained variance" verbatim, so this section teaches R² in exactly those words.

A regression model predicts monthly support-ticket volume. Constructed data: five months of actuals and predictions.

Step 1 — MSE, RMSE, MAE.

MSE  = Σe² / n = 1150 / 5 = 230.0
RMSE = sqrt(230.0)         = 15.166
MAE  = Σ|e| / n = 60 / 5   = 12.0

Note RMSE (15.17) > MAE (12.0). That gap is the signature of unequal errors: squaring makes the month-4 error of 30 contribute 900 of the 1,150 total — 78.3% of all squared error from one of five months. MAE gives that month 30 of 60, or 50%. So RMSE is the metric to choose when a single large miss is disproportionately harmful; MAE is the metric to choose when you care about typical error and do not want outliers to dominate. MSE is also the standard training loss for regression precisely because its gradient penalises large errors strongly.

Step 2 — the baseline: predict the mean. R² needs a baseline, and the baseline is the mean of the actuals.

ȳ = 700 / 5 = 140.0

Total sum of squares (variance in y, unnormalised):
TSS = Σ(y − ȳ)²
    = (120−140)² + (150−140)² + (130−140)² + (200−140)² + (100−140)²
    = 400 + 100 + 100 + 3600 + 1600
    = 5800

Step 3 — residual sum of squares. That is what we already computed: RSS = Σe² = 1150.

Step 4 — R², the proportion of explained variance.

R² = 1 − RSS / TSS
   = 1 − 1150 / 5800
   = 1 − 0.19828
   = 0.80172

R² ≈ 0.802. Read it in the objective's own words: the model explains about 80.2% of the variance in ticket volume. The remaining 19.8% of the variation is unexplained residual. Equivalently: the model's squared error is 19.8% of the squared error you would incur by simply always predicting the mean.

Step 5 — R² can be negative, and here is the arithmetic. R² is not a correlation coefficient squared in the general case and it is not bounded below by zero. Suppose a different model predicts a flat 180 every month:

errors: 120−180 = −60; 150−180 = −30; 130−180 = −50; 200−180 = 20; 100−180 = −80
RSS = 3600 + 900 + 2500 + 400 + 6400 = 13800
R²  = 1 − 13800 / 5800 = 1 − 2.3793 = −1.3793

R² = −1.379. A negative R² means the model is worse than predicting the mean of the target. This happens routinely on a held-out test set — a model can overfit the training data so badly that on new data it underperforms the trivial mean baseline — and it is a genuinely useful alarm. Anyone who tells you R² ranges from 0 to 1 is describing R² on the training set for an ordinary least-squares fit, which is the one case where it cannot go below zero. On evaluation data, R² is bounded above by 1 and unbounded below. This is a favourite exam detail.

Step 6 — the two R² traps. First, R² always rises when you add a predictor, even a random one, because extra freedom can only reduce training RSS. Adjusted R² penalises the number of predictors to counteract this. Second, R² is relative to the variance of your particular test set: the same model scores a higher R² on a high-variance sample than on a low-variance one, because TSS is bigger. So R² is not comparable across datasets, only within one. Report R² alongside MAE or RMSE — R² tells you how much better than trivial you are, and MAE tells you how wrong you are in real units, and neither substitutes for the other.

Step 7 — the metric-selection summary for this task. If the product question is "how many agents do I roster?", MAE in tickets is the number the operations manager needs. If the question is "does this model add anything over a naive average?", R² is the answer. If the question is "will a bad month break us?", RMSE or the max error is the metric. One dataset, three legitimate metrics, chosen by the question.

Retrieval systems return an ordered list, so their metrics must be sensitive to position. Four you must be able to compute and distinguish.

recall@k — of the relevant documents that exist for this query, what fraction appear in the top k? This is the metric that matters most for a RAG system's retrieval stage, because a document that is not retrieved cannot be used by the generator no matter how good the generator is.

precision@k — of the k documents returned, what fraction are relevant? This matters when context-window budget is scarce, since irrelevant retrieved chunks crowd out useful ones (04-06).

MRR (Mean Reciprocal Rank) — the mean, over queries, of 1 / rank of the first relevant result. It cares only about the first hit's position. Appropriate when the user needs one good answer and will not scroll.

nDCG (normalised Discounted Cumulative Gain) — sums a relevance gain per position, discounted logarithmically by position, then normalises by the best achievable ordering. It is the metric to use when relevance is graded (highly relevant / somewhat relevant / irrelevant) rather than binary, and when the whole ordering matters.

Worked arithmetic. A query has three relevant documents in the corpus. The system returns ten, and positions 1, 4 and 8 are the relevant ones. Constructed example.

recall@5    = relevant in top 5 / total relevant = 2 / 3 = 0.667
recall@10   = 3 / 3                                       = 1.000
precision@5 = relevant in top 5 / 5              = 2 / 5 = 0.400
precision@10= 3 / 10                                      = 0.300
MRR (this query) = 1 / rank of first relevant    = 1 / 1 = 1.000

Now compute nDCG@5 with binary relevance (gain 1 for relevant, 0 otherwise) and the standard log2(i+1) discount:

DCG@5  = Σ rel_i / log2(i + 1)
       = 1/log2(2) + 0/log2(3) + 0/log2(4) + 1/log2(5) + 0/log2(6)
       = 1/1.0000 + 0 + 0 + 1/2.3219 + 0
       = 1.0000 + 0.4307
       = 1.4307

Ideal ordering puts all three relevant docs first:
IDCG@5 = 1/log2(2) + 1/log2(3) + 1/log2(4)
       = 1.0000 + 0.6309 + 0.5000
       = 2.1309

nDCG@5 = DCG@5 / IDCG@5 = 1.4307 / 2.1309 = 0.6714

nDCG@5 = 0.671. Notice how the four metrics disagree about this same result list: MRR says 1.000 (perfect — the first hit was at rank 1), recall@10 says 1.000 (perfect — everything was found), precision@10 says 0.300, and nDCG@5 says 0.671. Every one is correct about a different question. A retrieval system is not summarised by one number, and quoting only recall@10 hides a bad ordering while quoting only MRR hides missing documents.

The selection rule for retrieval:

03-04 covers testing retrieval quality by hand before any of these metrics exist, and 09-07 puts recall@k and precision@k into the RAG evaluation decomposition.

Two cross-cutting rules complete the procedure. Always pair a headline with a guardrail, or optimisation will find the degenerate solution. And always report per-slice as well as aggregate, because an aggregate improvement that hides a subgroup regression is the standard shape of a fairness failure (13-04).

This lesson is the direct delivery point for the objective the exam names verbatim. The duplicated pair 2.2 / 3.2 reads: "Compare models using statistical performance metrics, such as loss functions or proportion of explained variance." Those two named items are cross-entropy/MSE and R². If you learn only one section of this module by heart, learn the R² arithmetic in §5 and the precision/recall/F1 arithmetic in §4 — the objective text points at them explicitly.

The objective-numbering defect, in full. The official study guide prints the Experimentation domain's objectives as 3.1–3.5. Those five lines are a verbatim duplicate of the Data Analysis domain's 2.1–2.5: awareness of data mining and visualization, comparing models using statistical performance metrics, conducting data analysis under supervision, creating graphs and charts, and identifying relationships and trends. Data Analysis is 14% of the exam; Experimentation is 22%. Read literally, the printed 3.x text means 22% of the exam has objectives that describe charts and data mining rather than model evaluation and RLHF — which contradicts the Experimentation section's own scope statement, "the study of how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)". The section's suggested-reading list agrees with the scope statement, naming A/B testing, inference optimization, zero-shot testing, machine translation evaluation, hallucinations in LLMs, GLUE, evaluating RAG applications, cross-validation, and benchmarking elementary language tasks. Published candidate reports independently confirm BLEU, hallucination mitigation and RLHF on the exam. Teach and answer from the derived scope. The one place the printed text does land squarely is 2.2/3.2's "loss functions or proportion of explained variance" — which is this lesson.

Additional objectives this lesson serves: 1.5 (familiarity with ML fundamentals including model comparison and cross-validation), 2.5 / 3.5 (identify factors that could affect research results — class imbalance is exactly such a factor), and 4.5 (monitor functioning of data collection, experiments, and other software processes).

Question phrasings to expect:

• "Which metric represents the proportion of variance in the dependent variable explained by the model?" → R² (coefficient of determination). The phrase "proportion of explained variance" is the official wording, so recognise it instantly.
• "Can R² be negative?" → Yes, on evaluation data, when the model performs worse than predicting the target's mean.
• "A classifier achieves 97% accuracy on a dataset where 96% of examples are negative. What should you conclude?" → Accuracy is uninformative here; examine precision, recall, F1 or PR-AUC.
• "A model must not miss any positive cases. Which metric should be prioritised?" → Recall.
• "Which metric balances precision and recall?" → F1, the harmonic mean.
• "Which loss function is appropriate for a classification task?" → Cross-entropy. For regression → MSE or MAE. Expect the reversed pairing as a distractor.
• "Why is MSE preferred over MAE when large errors are especially costly?" → Squaring makes large errors dominate the total, so the optimiser is pushed harder to eliminate them.
• "Which retrieval metric tells you whether the relevant document was retrieved at all?" → recall@k.
• "Which retrieval metric accounts for graded relevance and position?" → nDCG.
• "What is the difference between a loss function and an evaluation metric?" → A loss must be differentiable to train against; an evaluation metric need not be, and is chosen for interpretability.

Distractor families. (1) Accuracy for imbalanced data — the most common single wrong answer in the whole domain. (2) Swapped loss-to-task pairing — MSE offered for classification, cross-entropy for regression. (3) R² described as bounded 0–1 — true only for an OLS fit on its own training data. (4) R² conflated with correlation — related but not identical, and R² can be negative while a squared correlation cannot. (5) F1 offered where the errors are explicitly asymmetric — F1 weights precision and recall equally, so it is wrong precisely when the question says one error is worse. (6) A generation metric for a classification task or vice versa — BLEU for sentiment analysis, F1 for translation. (7) ROC-AUC presented as always preferable to PR-AUC — on rare positives the false-positive-rate axis is dominated by the negative class.

A loss function is what the optimiser minimises during training and therefore must be differentiable with a useful gradient; an evaluation metric is what a human reads to decide whether the system is good enough, and it has no such constraint. Cross-entropy is a loss and also a serviceable metric (perplexity is just its readable form). F1, exact match, nDCG and BLEU are metrics but not losses — they are piecewise-constant in the model's parameters, so their gradient is zero almost everywhere and there is nothing for gradient descent to follow. This gap is why so much of applied ML consists of training on a surrogate and evaluating on the real thing, and why the surrogate sometimes optimises in a direction the metric does not reward. It is also the structural reason RLHF exists: human preference is the thing you care about and it is not differentiable, so you fit a differentiable reward model to preference labels and optimise against that instead — with all the reward-hacking risk that substitution implies (11-07).

The two pairings to have automatic: cross-entropy ↔ classification and language modelling; MSE/MAE ↔ regression. Swapping them is a standard distractor, and MSE on a classification problem is not merely stylistically wrong — it produces weaker gradients when the model is confidently wrong, which is exactly when you most want a strong gradient.

It means R², computed as 1 − RSS/TSS: one minus the ratio of the model's squared error to the squared error of always predicting the target's mean. In the worked example, RSS = 1150 and TSS = 5800, so R² = 1 − 0.198 = 0.802 — the model explains 80.2% of the variance in the target, and 19.8% remains unexplained.

Three clarifications that the official phrasing invites and does not supply. First, "explained" is a statistical term of art, not a causal claim — R² measures variance accounted for, not causation, and a high R² is fully compatible with a spurious relationship (08-02 covers the correlation-versus-causation discipline). Second, R² can be negative on held-out data, meaning the model is worse than the mean baseline. Third, R² is not comparable across datasets, because TSS depends on the particular sample's variance: the same model looks better on a high-variance test set. Always pair R² with an error metric in real units so the number is actionable rather than merely flattering.

Neither, until you have priced the two errors. Write the two sentences out: "If we raise a false alarm, the cost is ___" and "If we miss a real positive, the cost is ___". Whichever cost is larger names the metric to prioritise, and the other becomes the guardrail.

Three concrete shapes this takes. When a missed positive is dangerous and a false alarm is merely expensive — disease screening, safety filters, fraud detection — prioritise recall with a precision floor that keeps the reviewer workload survivable. When a false positive directly harms a user — wrongly banning an account, blocking legitimate content, rejecting a valid claim — prioritise precision with a recall floor so the system still does something. When the costs genuinely are similar, F1 is the right summary, and that is the only situation in which it is. The §4 worked example shows the mechanics: moving one threshold took recall from 0.750 to 0.900 and precision from 0.600 to 0.300, with no model change. The threshold is a product decision expressed as a number, and choosing it is a business conversation informed by that curve.

1. Three questions, in order: what task, which wrongness is unacceptable, what can I afford per item. The metric follows.
2. Accuracy is the default wrong answer on imbalanced data. In §4, 97% accuracy beat the do-nothing baseline by one point while missing a quarter of violations.
3. Precision is about your predictions; recall is about reality. Memorise the denominators: FP joins precision, FN joins recall.
4. F1 is the harmonic mean and it asserts the two errors cost the same. When they do not, use Fβ or a floor-plus-headline design. Same matrix gave F2 = 0.714, F1 = 0.667, F0.5 = 0.625.
5. R² is the proportion of explained variance, 1 − RSS/TSS — the objective's own wording. The worked example gave 0.802.
6. R² can be negative. A negative R² means worse than predicting the mean, and it is common on held-out data.
7. RMSE punishes big misses, MAE describes typical error. One outlier month supplied 78.3% of the squared error and 50% of the absolute error in §5.
8. Cross-entropy for classification and language modelling; MSE/MAE for regression. The reversed pairing is a standard distractor.
9. A loss must be differentiable; a metric need not be. That gap is why you train on a surrogate and evaluate on the real thing — and why RLHF fits a reward model at all.
10. Retrieval needs several numbers. The same result list scored MRR 1.000, recall@10 1.000, precision@10 0.300 and nDCG@5 0.671.
11. Always pair a headline metric with a guardrail, and always report per slice. Single-objective optimisation finds the degenerate solution, and aggregates hide subgroup harm.
12. On the exam, "loss functions or proportion of explained variance" is the one phrase where the printed objective text genuinely describes this domain — everywhere else, answer from the Experimentation scope statement rather than the duplicated 3.x lines.

The procedure above sends any generation task to a reference-based metric and then stops, because generation metrics need a lesson of their own. There are three of them you will be asked to distinguish, and the pair at the centre — one built on precision and aimed at translation, the other built on recall and aimed at summarisation — is the most-reported confusable in the whole measurement domain. Swapping them is the kind of mistake that costs a question on the exam and a quarter of misdirected effort at work.

Next: 09-06 takes BLEU, ROUGE and exact match apart with hand-computed arithmetic on the same example, so you can state which one rewards precision, which rewards recall, which task each belongs to, and exactly where each one misleads.