BLEU vs ROUGE vs Exact Match: Which Metric for Which Task

All three are reference-based automatic metrics for text output: each compares a candidate string against one or more human-written references, and none of them needs a model at evaluation time. That makes them cheap, deterministic, and reproducible — which is exactly why they remain in use despite well-known weaknesses.

The acronyms themselves encode the answer to the exam's favourite question about them. ROUGE's "R" is literally "Recall-Oriented." BLEU has no such mnemonic built in, but "understudy" in both names tells you the honest intent: these metrics were designed as cheap stand-ins for human judgement, to be used while human evaluation happens more slowly, not instead of it.

Two structural facts about all three:

• They score surface form, not meaning. A correct paraphrase scores badly; a fluent falsehood that reuses the reference's words scores well. 09-04 covers the embedding-based family that partially fixes the first problem, and nothing in this family fixes the second.
• They are reference-bounded. A candidate better than the reference scores worse than one that mimics it. Multiple references help and multiply the labelling cost from 09-03.

L1 — Intuition: three different questions about the same two strings

Put the candidate and the reference side by side.

• BLEU asks: "Of the phrases the system produced, how many appear in the reference?" Producing extra material hurts. Producing too little would help — so a penalty is bolted on to stop that.
• ROUGE asks: "Of the phrases in the reference, how many did the system produce?" Omitting material hurts. Producing extra material barely hurts at all, which is ROUGE's characteristic weakness.
• Exact match asks: "Are these the same string?" One bit of information. Brutal and, for the right task, exactly right.

L2 — Mechanism: the formulas

BLEU. Three components.

Modified (clipped) n-gram precision. For each n from 1 to 4, count how many of the candidate's n-grams appear in the reference — but clip each n-gram's count at the maximum number of times it appears in any single reference. Clipping is what stops a candidate of the the the the the from scoring perfect unigram precision.

p_n = Σ over n-grams g in candidate  min(count_cand(g), max_count_ref(g))
      ─────────────────────────────────────────────────────────────────
      Σ over n-grams g in candidate  count_cand(g)

Brevity penalty. Because precision rewards saying less, BLEU multiplies by a penalty when the candidate is shorter than the reference. With c = candidate length and r = effective reference length:

BP = 1                        if c > r
BP = exp(1 - r/c)             if c <= r

Geometric mean and combination. With uniform weights w_n = 1/4 over n = 1…4:

BLEU = BP × exp( Σ_n w_n × ln p_n )    which equals   BP × (p_1 × p_2 × p_3 × p_4)^(1/4)

The geometric mean is a design decision with a sharp consequence: if any single p_n is zero, BLEU is zero. A candidate with no 4-gram in common with the reference scores 0 regardless of how good its unigrams and bigrams are. That is why BLEU is a corpus-level metric in practice — on a single short sentence, a missing 4-gram zeroes the score, so implementations either smooth the counts or aggregate the numerators and denominators across the whole test set before dividing.

ROUGE-N.

ROUGE-N recall    = Σ matched n-grams / Σ n-grams in the reference
ROUGE-N precision = Σ matched n-grams / Σ n-grams in the candidate
ROUGE-N F1        = harmonic mean of the two

ROUGE-1 (unigrams) and ROUGE-2 (bigrams) are the two most reported. Note that ROUGE can be reported as precision or F1 — the metric family supports all three — but its name and its default reporting are recall-first, and that is the association the exam tests.

ROUGE-L. Based on the longest common subsequence (LCS) between candidate and reference. A subsequence need not be contiguous, so LCS rewards correct ordering without demanding fixed-length n-gram matches:

R_lcs = LCS(cand, ref) / len(ref)
P_lcs = LCS(cand, ref) / len(cand)
F_lcs = ((1 + β²) × R_lcs × P_lcs) / (R_lcs + β² × P_lcs)

ROUGE-Lsum is the variant that applies LCS sentence-by-sentence and aggregates, which is standard for multi-sentence summaries.

Exact match. Normalise, then compare.

EM = 1 if normalize(candidate) == normalize(reference) else 0

The normalisation is the entire engineering content of the metric, and it is where implementations diverge. A typical normaliser lowercases, strips punctuation, removes leading articles, collapses whitespace, and may normalise numbers or dates. Two teams reporting "exact match" with different normalisers are reporting different metrics, and the gap can be ten points or more. Record the normaliser as part of the metric's definition, the same way 09-04 requires recording BERTScore's encoder.

L3 — Depth: the design decisions and what they cost

Why BLEU clips. Without clipping, repeating a high-frequency reference word inflates precision arbitrarily. Clipping caps each n-gram's credit at its reference frequency, so the the the earns credit for at most as many thes as the reference contains.

Why BLEU uses a geometric mean over four n-gram orders. Unigram precision measures adequacy (are the right words there?) and 4-gram precision measures fluency (are they in fluent local order?). The geometric mean forces a system to satisfy both; an arithmetic mean would let strong unigram precision cover for word salad. The cost is the zero-collapse property above.

Why BLEU needs a brevity penalty and ROUGE does not. Precision alone is maximised by outputting the single most-confident word. BP punishes that. ROUGE, being recall-oriented, is maximised by outputting everything, so ROUGE's structural weakness is the opposite: a candidate that copies the entire source document scores excellent ROUGE recall. Anyone gaming ROUGE builds an extractive copier. Report ROUGE F1 rather than bare recall, or add a length constraint, to close that hole.

Why exact match is harsher than it looks and sometimes exactly right. EM is a step function: "Paris" and "paris, France" both fail against "Paris, France" depending on the normaliser. That harshness is a feature when the answer space is genuinely canonical — an order ID, a JSON field value, a yes/no, a chosen option — because any tolerance introduces the risk of accepting a wrong answer. It is a defect the moment paraphrase is legitimate. The standard middle ground for QA is token-level F1 against the reference answer, which gives partial credit for overlapping answer spans; you will see EM and token-F1 reported as a pair for this reason.

The paraphrase problem, quantified. Both BLEU and ROUGE compare surface strings, so the sentence "the meeting was delayed" against reference "the meeting was postponed" loses all credit for the final token. 09-04 computed this exact pair: ROUGE-1 F1 was 0.667 while BERTScore was 0.953. The overlap metrics were not wrong about the strings; they were wrong about the quality.

The factuality blind spot. Neither BLEU nor ROUGE nor EM-against-a-reference can tell you whether a claim is true — only whether it resembles the reference. A summary that inverts a number while reusing the reference's phrasing scores well. This is why summarization pipelines need a separate faithfulness measurement, which is 09-07's subject, and why hallucination detection (09-12) is not a metric-selection problem at all.

The comparison table is the highest-value asset in this lesson. Learn the first three columns cold.

Three cross-cutting readings of that table:

• The orientation column is the exam answer. BLEU → precision → translation. ROUGE → recall → summarization. If you remember nothing else, remember that ROUGE's "R" stands for Recall-Oriented.
• Every metric in the table is reference-based, so none of them can score an open-ended generation with no target. That gap belongs to rubric-based methods (09-03, 09-10).
• No metric in the table detects factual error. Every one answers "does this look like the target?"

We score the same candidate against the same reference with all three metrics. Constructed example; every number below is computed from these two strings.

• Reference (10 tokens): the cat sat quietly on the warm mat by the window

Let me count precisely. Tokens: the(1) cat(2) sat(3) quietly(4) on(5) the(6) warm(7) mat(8) by(9) the(10) window(11). The reference is 11 tokens.

• Candidate (9 tokens): the cat sat on the mat by the window

Tokens: the(1) cat(2) sat(3) on(4) the(5) mat(6) by(7) the(8) window(9). The candidate is 9 tokens. The candidate dropped quietly and warm — it is a faithful compression.

Step 1 — ROUGE-1 (unigram recall)

Reference unigram counts: the ×3, cat ×1, sat ×1, quietly ×1, on ×1, warm ×1, mat ×1, by ×1, window ×1 → 11 total. Candidate unigram counts: the ×3, cat ×1, sat ×1, on ×1, mat ×1, by ×1, window ×1 → 9 total.

Matched unigrams (clipped at reference counts): the min(3,3)=3, cat 1, sat 1, on 1, mat 1, by 1, window 1 → 9 matches.

ROUGE-1 recall    = 9 / 11 = 0.8182
ROUGE-1 precision = 9 / 9  = 1.0000
ROUGE-1 F1        = 2 × 0.8182 × 1.0000 / (0.8182 + 1.0000) = 1.6364 / 1.8182 = 0.9000

ROUGE-1 recall = 0.818. The summary covered 81.8% of the reference's unigrams; the two it missed are quietly and warm — exactly the two modifiers it dropped.

Step 2 — ROUGE-2 (bigram recall)

Reference bigrams (10): the cat, cat sat, sat quietly, quietly on, on the, the warm, warm mat, mat by, by the, the window. Candidate bigrams (8): the cat, cat sat, sat on, on the, the mat, mat by, by the, the window.

Matches: the cat, cat sat, on the, mat by, by the, the window → 6 matches. (sat on and the mat are candidate bigrams absent from the reference.)

ROUGE-2 recall    = 6 / 10 = 0.6000
ROUGE-2 precision = 6 / 8  = 0.7500
ROUGE-2 F1        = 2 × 0.6000 × 0.7500 / (0.6000 + 0.7500) = 0.9000 / 1.3500 = 0.6667

ROUGE-2 recall = 0.600. Bigram recall falls faster than unigram recall, because dropping one word destroys two bigrams. This is why ROUGE-2 is the stricter and more informative of the pair.

Step 3 — ROUGE-L (longest common subsequence)

The LCS of the two token sequences: the cat sat on the mat by the window — every candidate token appears in the reference in the same relative order, so LCS = 9.

R_lcs = 9 / 11 = 0.8182
P_lcs = 9 / 9  = 1.0000
F_lcs (β = 1) = 2 × 0.8182 × 1.0000 / (0.8182 + 1.0000) = 0.9000

ROUGE-L F1 = 0.900. Note it exceeds ROUGE-2 F1 (0.667) because LCS does not require contiguity — deleting an interior word costs LCS one token, whereas it costs bigram matching two bigrams.

Step 4 — BLEU

Now score the same pair as if this were a translation, where dropping quietly and warm is an omission rather than a compression.

Clipped n-gram precisions.

• 1-grams: 9 candidate unigrams, 9 clipped matches → p_1 = 9/9 = 1.0000
• 2-grams: 8 candidate bigrams, 6 matches → p_2 = 6/8 = 0.7500
• 3-grams: candidate trigrams (7): the cat sat, cat sat on, sat on the, on the mat, the mat by, mat by the, by the window. Reference trigrams (9): the cat sat, cat sat quietly, sat quietly on, quietly on the, on the warm, the warm mat, warm mat by, mat by the, by the window. Matches: the cat sat, mat by the, by the window → 3. So p_3 = 3/7 = 0.4286
• 4-grams: candidate 4-grams (6): the cat sat on, cat sat on the, sat on the mat, on the mat by, the mat by the, mat by the window. Reference 4-grams (8): the cat sat quietly, cat sat quietly on, sat quietly on the, quietly on the warm, on the warm mat, the warm mat by, warm mat by the, mat by the window. Matches: mat by the window → 1. So p_4 = 1/6 = 0.1667

Brevity penalty. c = 9, r = 11, and c <= r, so:

BP = exp(1 - r/c) = exp(1 - 11/9) = exp(1 - 1.2222) = exp(-0.2222) = 0.8007

Geometric mean.

ln p_1 = ln 1.0000 =  0.00000
ln p_2 = ln 0.7500 = -0.28768
ln p_3 = ln 0.4286 = -0.84730
ln p_4 = ln 0.1667 = -1.79176

mean = (0.00000 - 0.28768 - 0.84730 - 1.79176) / 4 = -2.92674 / 4 = -0.73169
exp(-0.73169) = 0.48104

BLEU.

BLEU = BP × 0.48104 = 0.8007 × 0.48104 = 0.3852

BLEU ≈ 0.385 (38.5 when reported on the conventional 0–100 scale).

Step 5 — exact match

normalize(candidate) = "the cat sat on the mat by the window"
normalize(reference) = "the cat sat quietly on the warm mat by the window"
EM = 0

Exact match = 0.

Step 6 — read the three numbers together

One output, five very different scores, spanning 0 to 0.9. Nothing about the candidate changed between rows — only the question being asked. If this were a summarization task, ROUGE-L 0.900 says the system compressed well. If it were a translation task, BLEU 0.385 says it dropped meaning ("quietly", "warm") that a translation is obliged to carry. The metric encodes the task's tolerance for omission, and choosing the wrong one gives you a confidently wrong verdict rather than an error message.

Step 7 — demonstrate the two gaming attacks

Gaming ROUGE by copying. Suppose the candidate is the entire reference plus fifteen unrelated sentences from elsewhere in the source document. ROUGE-1 recall becomes 11/11 = 1.000 — perfect — because every reference unigram is present. ROUGE-1 precision collapses (11 matches over ~150 candidate unigrams ≈ 0.073), and F1 lands around 0.137. This is why bare ROUGE recall must never be a headline metric: it is trivially maximised by an extractive copier, and only the F1 form or an explicit length constraint stops it.

Gaming BLEU by brevity. Suppose the candidate is just the cat. Then p_1 = 2/2 = 1.0, p_2 = 1/1 = 1.0, and there are no 3-grams or 4-grams at all, so a naive implementation would have to smooth or would produce a degenerate score. But the brevity penalty with c = 2, r = 11 gives BP = exp(1 - 5.5) = exp(-4.5) = 0.0111, crushing any score to near zero. The brevity penalty exists exactly for this attack, and it is why "BLEU = clipped precision" alone is an incomplete definition.

The compressed decision rule, worth memorising as a sentence: translation → BLEU (precision); summarization → ROUGE (recall); one canonical answer → exact match; legitimate paraphrase → an embedding or learned metric; no reference at all → a rubric.

This is the single most-reported item in the exam's measurement content. Published candidate reports name BLEU scores explicitly among the topics that appear, and the BLEU-versus-ROUGE contrast is on the standard confusable list alongside stemming-versus-lemmatization and precision-versus-recall. The exam's own suggested-reading list for the Experimentation domain includes machine translation methods, which is BLEU's home task.

The question format is metric-to-task matching, and it is usually answerable from the orientation alone: given a described task, pick the metric; or given a metric, name what it rewards.

The objective-numbering defect, stated where the objectives are cited. The official study guide prints the Experimentation domain's objectives as 3.1–3.5, and those five lines are a verbatim duplicate of the Data Analysis domain's 2.1–2.5 — data mining awareness, comparing models using statistical performance metrics, conducting data analysis under supervision, creating charts, identifying trends. Read literally, they describe data analysis and visualization, not generation-quality metrics. The Experimentation section's own scope statement covers "AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)", and its suggested readings name machine translation, GLUE, hallucinations, and evaluating RAG applications. Candidate reports independently confirm BLEU on the exam. So the derived scope governs here; the printed 3.x text does not describe this lesson at all. The objective that legitimately applies is the duplicated pair 2.2 / 3.2, "compare models using statistical performance metrics" — BLEU and ROUGE are statistical performance metrics — together with 1.3 (build LLM use cases such as RAG, chatbots and summarizers), since a summarizer without ROUGE has no acceptance criterion.

Question phrasings to expect:

• "Which metric is most appropriate for evaluating a machine-translation system?" → BLEU.
• "Which metric is most appropriate for evaluating a summarization system?" → ROUGE.
• "BLEU is primarily a measure of what?" → n-gram precision against one or more references, with a brevity penalty.
• "What does the R in ROUGE stand for?" → Recall-Oriented. This is a real question shape, and the acronym is the answer key.
• "Why does BLEU include a brevity penalty?" → Because precision alone rewards producing very short output; BP penalises candidates shorter than the reference.
• "A summarization system copies whole sentences from the source and achieves very high ROUGE recall. What does this show?" → ROUGE recall can be gamed by copying; report F1 or constrain length.
• "Which metric would you use to evaluate short-answer question answering with a single canonical answer?" → Exact match, usually paired with token-level F1.
• "Why is exact match a poor metric for open-ended generation?" → It gives zero credit to correct paraphrases.
• "Which metric extends unigram matching with stems, synonyms and paraphrases?" → METEOR.
• "Can BLEU detect that a translation is fluent but factually inverted?" → No; it measures surface overlap only.

Distractor families. (1) BLEU and ROUGE swapped — BLEU offered for summarization or ROUGE for translation. This is the single highest-frequency distractor in the domain, and the orientation mnemonic defeats it. (2) Perplexity offered for a generation-quality question — a reference-free metric offered where a reference-based one is needed (09-02). (3) Exact match offered for open-ended text — plausible if you have not thought about paraphrase. (4) BLEU described as recall-based or ROUGE as precision-based — a direct inversion of the orientation. (5) A claim that BLEU or ROUGE detects hallucination or factual error — no surface metric does. (6) METEOR and BERTScore treated as interchangeable — one uses lexical resources, one uses learned embeddings.

ROUGE. The reason is structural rather than conventional: a summary is deliberately much shorter than its source, so the failure you must detect is omission of important content, and recall is the metric that moves when content is omitted. BLEU's precision orientation asks the opposite question — how much of what the system said was warranted — which a good summary passes trivially while a bad one that drops the main point also passes. Report ROUGE-1, ROUGE-2 and ROUGE-L together: ROUGE-1 for content coverage, ROUGE-2 as the stricter phrasing check, and ROUGE-L for ordering across sentences. Report them as F1 rather than bare recall, so a system cannot win by copying the source. And add at least one non-overlap metric — BERTScore for paraphrase tolerance (09-04) and a faithfulness check for truth (09-07) — because ROUGE cannot see either.

Neither is adequate on its own for a general-purpose LLM, because both need a reference and most LLM outputs have no single correct target. They remain useful in two specific places. First, on the sub-tasks where a reference genuinely exists: translation, summarization against a human-written summary, and structured extraction. Second, as a regression tripwire in CI: they are deterministic, cheap and fast, so they can run on every commit against a frozen set and fail the build when a score drops beyond a threshold (10-04). What they cannot do is score helpfulness, tone, appropriate refusal, instruction compliance, or groundedness — which is most of what makes an LLM feature good or bad. For those you need a rubric applied by a human (09-03) or by a judge model with its known biases (09-10). The mature answer is a layered harness: overlap metrics as fast tripwires, embedding metrics for paraphrase-tolerant scoring, RAG-specific metrics for groundedness, and a periodic human pass to keep the whole stack honest.

Exact match is all-or-nothing on the whole normalised string; token-level F1 gives partial credit for overlapping tokens between the predicted and reference answer spans. If the reference answer is the Treaty of Versailles and the prediction is Treaty of Versailles, a normaliser that strips leading articles scores EM = 1, and one that does not scores EM = 0 — while token-F1 scores about 0.86 either way, because three of the four reference tokens are present. This is why extractive-QA results are conventionally reported as an EM/F1 pair: EM tells you how often the system was exactly right, F1 tells you how close it was when it was not. If you must pick one, pick the one that matches the consumer of the answer — a downstream system that string-matches needs EM, a human reader is served by F1.

Because both operate on token strings and have no representation of meaning. An n-gram either matches or it does not; "postponed" and "delayed" are as different to ROUGE as "postponed" and "banana". The metrics were designed in an era when the systems under test produced output close to the reference's phrasing, and they degrade precisely as systems become more capable of legitimate rewording — which is a mildly perverse property: the better your generator gets at saying things its own way, the worse these metrics rate it.

Three partial remedies, in increasing order of cost. Multiple references widen the set of accepted phrasings and are the original intended fix, but each additional reference is another human-written text. METEOR buys tolerance from lexical resources — stems, synonym sets, paraphrase tables — and is the pre-neural answer. Embedding and learned metrics (BERTScore, BLEURT, COMET) buy it from vector geometry or from training against human ratings, which is 09-04's subject. All three fix the paraphrase problem and none of them fixes the factuality problem: an embedding metric will happily credit an antonym.

1. BLEU = precision = translation. ROUGE = recall = summarization. Memorise the triple; the R in ROUGE stands for Recall-Oriented.
2. BLEU = brevity penalty × geometric mean of clipped n-gram precisions for n = 1…4. Any zero order makes the whole score zero, which is why BLEU is a corpus-level metric.
3. The brevity penalty exists to stop the terseness attack. In the worked example, c=9, r=11 gave BP = 0.8007, and a two-token candidate would have been crushed to 0.011.
4. ROUGE recall is gamed by copying. Report ROUGE F1 and a length guardrail, never bare recall.
5. The same output scored ROUGE-1 recall 0.818, ROUGE-2 recall 0.600, ROUGE-L F1 0.900, BLEU 0.385, and EM 0. The metric encodes the task's tolerance for omission.
6. ROUGE-2 falls faster than ROUGE-1 because deleting one word destroys two bigrams; ROUGE-L is gentler because LCS does not need contiguity.
7. Exact match is entirely defined by its normaliser. Version it, and report EM with token-F1.
8. None of the three detects factual error. They answer "does this resemble the target?", never "is this true?"
9. All three punish valid paraphrase. Multiple references, METEOR, or embedding metrics (09-04) are the remedies.
10. Use them as fast, deterministic CI tripwires, not as the definition of quality for an open-ended LLM feature.
11. BLEU numbers are not comparable across implementations or tokenizations; pin one and record it.

Every metric so far compares an output to a reference. A retrieval-augmented system breaks that frame, because its answer is supposed to be grounded in a specific set of retrieved passages — and that gives you something better than a reference to check against: the evidence itself. It also gives you a new diagnostic problem. When a RAG answer is wrong, the fault could be in retrieval (the evidence was never fetched) or in generation (the evidence was there and the model ignored it), and a single answer-quality score cannot tell those apart.

Next: 09-07 decomposes RAG evaluation into four numbers that sit on either side of the retrieval/generation boundary — faithfulness, answer relevance, context precision and context recall — so that when a number moves you know which stage moved it.