BLEU vs. ROUGE vs. METEOR: Precision, Recall, and Semantic Alignment in Generation Metrics

BLEU (Bilingual Evaluation Understudy) is a precision-oriented n-gram overlap metric, built for machine translation. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) It asks: of the n-grams the candidate produced, how many also appear in the reference? A candidate that says very little, but says only things the reference also says, can score deceptively well on precision alone — which is exactly why BLEU pairs a brevity penalty with its n-gram precision, to stop short, safe outputs from gaming the score.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a recall-oriented overlap metric, built for summarization. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) It asks the opposite question: of the n-grams (or the longest common subsequence) in the reference, how many did the candidate actually cover? A summary that includes everything the reference says, padded with extra content, can score deceptively well on recall alone.

METEOR is an alignment-based metric, recall-weighted, that allows stem and synonym matches rather than only exact token matches. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) Where BLEU and ROUGE only count a match when the surface tokens are identical, METEOR's alignment step will match "running" to "run" (a stem match) or "quick" to "fast" (a synonym match, drawn from a resource like WordNet), and then scores the resulting alignment with a recall-weighted harmonic mean, penalized for how fragmented the alignment is.

[GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) The reversing of BLEU's and ROUGE's orientations — calling BLEU recall-oriented or ROUGE precision-oriented — is named directly in the domain's own trap list as the classic distractor for this pair, and it is worth internalizing as a mnemonic: BLEU was built to grade translation, where a fluent but overly long candidate that rambles past the reference should be punished, which is a precision concern; ROUGE was built to grade summarization, where missing key content from the source is the worse sin, which is a recall concern.

L1 — Intuition: precision asks about the candidate, recall asks about the reference

Picture two people independently describing the same event. BLEU asks: "of everything person A said, how much matches what actually happened (the reference)?" — every extra, unsupported detail A adds hurts the score. ROUGE asks: "of everything that actually happened, how much did person A manage to mention?" — every important detail A leaves out hurts the score. The two questions are genuinely different, and a candidate can score very differently on each: a short, cautious summary that only restates the reference's opening sentence can have perfect precision (everything it said is correct) and terrible recall (it left almost everything out).

L2 — Mechanism: the formulas, worked from scratch

BLEU's modified n-gram precision. For each n-gram order (typically 1 through 4), count how many of the candidate's n-grams also appear in the reference, but clip each n-gram's count at the number of times it appears in the reference — this stops a candidate from repeating one correct word many times to inflate its score.

modified precision_n = (sum over n-grams of min(count in candidate, count in reference))
                        ─────────────────────────────────────────────────────────────
                        (total n-gram count in candidate)

BLEU then combines precision across n-gram orders (typically a geometric mean of orders 1 through 4) and multiplies by a brevity penalty that shrinks the score if the candidate is shorter than the reference — otherwise a one-word candidate that happens to be a correct unigram could score perfectly on precision while saying almost nothing.

brevity penalty = 1                                   if candidate length >= reference length
                = exp(1 - reference length / candidate length)   otherwise

ROUGE-N recall. The mirror image of BLEU's precision — the denominator flips from the candidate's n-gram count to the reference's:

ROUGE-N recall = (sum over n-grams of min(count in candidate, count in reference))
                  ─────────────────────────────────────────────────────────────
                  (total n-gram count in reference)

ROUGE-L replaces n-gram overlap with the longest common subsequence (LCS) between candidate and reference — the longest sequence of words appearing in the same relative order in both, not necessarily contiguous — which rewards structural similarity that fixed-length n-grams can miss.

METEOR's alignment and score. METEOR first builds an alignment between candidate and reference tokens, preferring exact matches, then stem matches, then synonym matches, choosing the alignment that minimizes the number of "chunks" (contiguous matched spans) needed to cover the matched tokens. It then computes precision and recall over the aligned tokens, combines them into a recall-weighted harmonic mean (recall weighted roughly nine times more than precision in the reference formulation), and applies a fragmentation penalty based on how many disjoint chunks the alignment needed — a heavily scrambled word order, even with perfect word-level matches, produces a lower score than a fluent, contiguous match.

L3 — Why METEOR's extra step changes what each metric can and cannot see

BLEU and ROUGE are both blind to synonymy and morphology by construction: "the cat sat" and "a feline sat" share zero unigram overlap despite meaning nearly the same thing, so both metrics score them as almost entirely mismatched. METEOR's stem and synonym matching step exists specifically to close that gap — it will match "feline" to "cat" if its synonym resource links them, and it will match "sitting" to "sat" via stemming even though the surface forms differ. That one design choice is the entire reason METEOR shows stronger correlation with human judgment than BLEU on sentence-level scoring in the literature that motivated it, and it is also the entire reason METEOR requires a language-specific resource (a stemmer, a synonym database) that BLEU and ROUGE do not need — you cannot run METEOR on a language without first building or sourcing that resource, while BLEU and ROUGE run on raw tokens in any language.

Constructed scenario. All numbers below are computed from an invented example, not measured from any real system.

Reference: "the quick brown fox jumps over the lazy dog" (9 words) Candidate: "a quick fox jumps over a lazy dog" (8 words)

Step 1 — BLEU-1 (unigram precision) with clipping

Candidate unigrams: a, quick, fox, jumps, over, a, lazy, dog   (8 total, "a" appears twice)
Reference unigram counts (relevant ones): quick=1, fox=1, jumps=1, over=1, lazy=1, dog=1, a=0

Matches, clipped at reference count:
  "quick" -> min(1,1) = 1
  "fox"   -> min(1,1) = 1
  "jumps" -> min(1,1) = 1
  "over"  -> min(1,1) = 1
  "a"     -> min(2,0) = 0    (reference has no "a" at all)
  "lazy"  -> min(1,1) = 1
  "dog"   -> min(1,1) = 1

Clipped match total = 6
BLEU-1 precision = 6 / 8 = 0.750

Step 2 — brevity penalty

candidate length = 8, reference length = 9
since candidate length < reference length:
  BP = exp(1 - 9/8) = exp(-0.125) = 0.882

BLEU-1 (with brevity penalty) = 0.750 x 0.882 = 0.662

Step 3 — ROUGE-1 recall

Reference unigrams: the, quick, brown, fox, jumps, over, the, lazy, dog  (9 total)
Matches found in candidate, clipped at candidate's count:
  "quick"=1, "fox"=1, "jumps"=1, "over"=1, "lazy"=1, "dog"=1   (candidate has zero "the" and zero "brown")

Clipped match total = 6
ROUGE-1 recall = 6 / 9 = 0.667

Step 4 — reading precision against recall on the same pair

BLEU-1 (precision-side, with BP) = 0.662
ROUGE-1 (recall-side)            = 0.667

The two numbers land close together here because the candidate is a near-paraphrase with substitutions ("a" for "the", dropped "brown") rather than either padding or truncation — but they are answering different questions and would diverge sharply on a different pair. If the candidate had instead been the single word "dog", BLEU-1 precision would be a perfect 1.0/with a brutal brevity penalty (correct, but almost nothing said), while ROUGE-1 recall would collapse to 1/9 (correctly flagging that almost nothing of the reference was covered). That divergence is the entire reason both numbers, not one, belong on a report.

Step 5 — METEOR's synonym and stem credit

Exact matches (as in BLEU/ROUGE): quick, fox, jumps, over, lazy, dog   -> 6 tokens
Stem match: none additional in this pair (no morphological variants present)
Synonym match: "brown" has no aligned counterpart in the candidate at all -- omission, not
  a synonym substitution, so METEOR gets no extra credit here either

Matched candidate tokens = 6 of 8;  matched reference tokens = 6 of 9
Precision = 6/8 = 0.750;  Recall = 6/9 = 0.667
Recall-weighted harmonic mean (recall weight ~9x precision weight, reference formulation):
  Fmean = (10 x Precision x Recall) / (Recall + 9 x Precision) [simplified reference-style weighting]
        = (10 x 0.750 x 0.667) / (0.667 + 9 x 0.750)
        = 5.0025 / 7.417  =  0.674

Fragmentation: the six matched tokens fall into a small number of contiguous chunks (roughly
  2, since "a" breaks the run once), so the fragmentation penalty is modest, discounting
  Fmean by roughly 6-10% in this illustration:
  METEOR (approx.) = 0.674 x 0.92 = 0.620

In this particular pair, all three scores land in a similar 0.62-0.67 range, because the substitutions here happen to be function-word swaps ("a" for "the") rather than genuine content-word synonyms. METEOR's advantage would show up far more clearly on a pair like reference "the vehicle stopped suddenly" versus candidate "the car halted abruptly" — zero unigram overlap on the content words, near-zero BLEU and ROUGE, but a METEOR score lifted substantially by stem/synonym alignment on "vehicle"/"car" and "stopped"/"halted" and "suddenly"/"abruptly", which is exactly the case BLEU and ROUGE cannot see and METEOR was built to catch.

Constructed scenario, illustrating why precision and recall need each other's checks even within one metric family.

Gaming BLEU with a short, safe candidate. Reference: "the system successfully processed all twelve requests without error" (9 words). Candidate: "the system" (2 words). Every unigram in the candidate — "the", "system" — appears in the reference, so clipped-match precision is 2/2 = 1.000, a perfect score. The brevity penalty catches this: BP = exp(1 - 9/2) = exp(-3.5) = 0.030, driving the final BLEU-1 down to 1.000 x 0.030 = 0.030. Without the brevity penalty, a system could learn to say almost nothing and still score near-perfectly on raw precision — which is exactly the failure BLEU's brevity term exists to close.

Gaming ROUGE with a long, padded candidate. Same reference (9 words). Candidate: "the system successfully processed all twelve requests without error and then additionally logged extensive diagnostic information for the operations team to review later" (23 words, everything from the reference plus 14 extra words). Every reference unigram appears somewhere in the candidate, so ROUGE-1 recall is 9/9 = 1.000, a perfect score, despite the candidate padding the answer with unrequested content. ROUGE alone does not penalize the padding at all — a precision check, or a length-aware variant, would be needed to catch it. This is the mirror image of the BLEU-gaming case above, and it is why a report that shows only ROUGE recall, or only BLEU precision, is systematically blind to the failure mode its own metric was not designed to catch.

Reading the two gaming attacks side by side makes the precision/recall split concrete: BLEU is vulnerable to candidates that say too little; ROUGE is vulnerable to candidates that say too much. Neither vulnerability is a bug in the metric — each metric is doing exactly the job its orientation implies — but neither metric alone tells the whole story, which is the practical argument for reporting more than one of these numbers together rather than picking a favorite.

Six rows, six independent failure modes — none of them corrected by picking a "better" metric, since each mistake is a misuse of a correctly-computed number rather than a flaw in the metric itself.

Because the failure mode each task cares most about is different, and each metric was designed to punish exactly that failure mode. A bad translation is often bad because it says things the source did not say — an unsupported addition or a mistranslation — which is a precision problem, and BLEU's precision-plus-brevity-penalty design targets it directly. A bad summary is often bad because it leaves out something the source considered important — an omission — which is a recall problem, and ROUGE's recall design targets that instead. Using ROUGE for translation would under-penalize a rambling, unsupported translation as long as it happened to also cover the reference's content; using BLEU for summarization would under-penalize a summary that omits key content as long as whatever it did include was accurate.

Credit for stem and synonym matches, which lets METEOR partially score a paraphrase that BLEU and ROUGE would treat as almost entirely non-overlapping. "The vehicle stopped" and "The car halted" share one function word and zero content words at the surface level, so BLEU and ROUGE both score the pair near zero on unigram overlap despite the sentences meaning the same thing. METEOR's alignment step matches "vehicle" to "car" and "stopped" to "halted" through a synonym resource, producing a substantially higher score that better reflects the sentences' actual semantic equivalence. This comes at the cost of requiring a language-specific stemmer and synonym database, which is why METEOR is not a drop-in replacement for BLEU or ROUGE in every language or domain.

• BLEU is precision-oriented and built for translation; ROUGE is recall-oriented and built for summarization — reversing the two is this pair's standing exam distractor.
• BLEU's brevity penalty exists specifically to stop short, safe candidates from gaming precision.
• ROUGE-L's longest-common-subsequence approach catches structural similarity that fixed n-grams miss.
• METEOR adds stem and synonym matching on top of a recall-weighted score, which is the one property letting it credit a correct paraphrase that BLEU and ROUGE score as a near-total mismatch.
• METEOR's paraphrase credit requires a language-specific resource (stemmer, synonym database) that BLEU and ROUGE do not need.
• None of the three can verify factual correctness — all compare surface or aligned overlap against a reference string, not against ground truth.

⭐ THE EARNED INSIGHT

BLEU and ROUGE are the same arithmetic — clipped n-gram counts over a ratio — pointed in opposite directions by which string sits in the denominator; METEOR is not a third direction so much as a third question, "can these two texts be aligned as saying the same thing even where the words differ," and that third question is the only one of the three that a straightforward synonym swap can defeat both BLEU and ROUGE while leaving unanswered.

Surface overlap, however it is measured, still cannot tell you whether a fluent answer is actually correct. That is the subject of the next lesson: M6-03 covers LLM-as-a-judge scoring, human-in-the-loop review, and the systematic error analysis needed to catch a fluent-but-wrong answer that a high BLEU or ROUGE score would have let through.