Public Benchmarks (GLUE, MMLU) and Data Contamination Explained

A public benchmark has four parts, and all four have to be fixed for the comparison to mean anything: a dataset of items, a task definition that says what a correct output looks like, a metric that turns outputs into a number, and a protocol that specifies how the model is allowed to see the items (zero-shot, few-shot, fine-tuned, with or without chain-of-thought). Change any one of those four and two reported numbers stop being comparable, even when they carry the same benchmark name.

GLUE — the General Language Understanding Evaluation — is a collection of nine English sentence and sentence-pair tasks aggregated into a single score. Its constituent tasks are worth knowing by shape rather than by name-drill, because the shapes are the classical NLP task families the exam samples from:

The design intent matters more than the roster. GLUE was built in the pre-LLM era for models that were fine-tuned per task: you took a pretrained encoder, attached a small classification head, trained on each task's training split, and submitted test-set predictions to a leaderboard that kept the test labels private. SuperGLUE followed with eight harder tasks (including BoolQ, COPA, MultiRC, ReCoRD, WiC and the Winograd Schema Challenge) precisely because the strongest systems had passed the human baselines published alongside GLUE — the benchmark had been solved in the narrow sense that further gains no longer separated systems meaningfully.

MMLU — Massive Multitask Language Understanding — is a different animal. It is roughly sixteen thousand four-option multiple-choice questions spread across 57 subjects, grouped into humanities, social sciences, STEM and a catch-all "other". Subjects range from elementary mathematics through professional law and clinical knowledge. Nothing about MMLU is sentence-level; it is a knowledge-and-reasoning exam, scored as plain accuracy, and its canonical protocol is few-shot — a handful of worked examples per subject prepended to the prompt, drawn from a small development split, rather than any weight update at all.

That protocol difference is the single most useful contrast the two benchmarks give you. GLUE assumes you will train on the task; MMLU assumes you will prompt the model as it is. The move from one assumption to the other is the move from the encoder-fine-tuning era to the foundation-model era, and it is why 04-03 and 05-01 both matter as background to this lesson: which architecture family you are holding determines which benchmark protocol is even applicable to it.

Data contamination — also called benchmark leakage or train–test overlap at web scale — is the condition where items from the benchmark's evaluation split appear, verbatim or near-verbatim, in the corpus a model was pretrained on. It is not a hypothetical. Benchmarks are published on the open web precisely so people can use them; web-scale pretraining corpora are assembled by crawling the open web; the intersection is not empty and cannot be assumed empty. The consequence is a specific, directional bias: contamination inflates the reported score, and it inflates it most for the items the model has seen most often, which are typically the oldest and most-discussed benchmarks — exactly the ones with the longest leaderboards.

L1 — Intuition: a shared exam that leaked

Imagine a university that publishes every past paper, with answers, on its website — and then sets an exam by drawing questions from those papers. A student who has read the website scores highly. You cannot tell, from the score alone, whether that student understands the subject or has an excellent memory for the website. Worse, you cannot tell how much of each: one student may be 90% understanding and 10% recall, another the reverse, and their transcripts are identical.

Every part of the contamination problem is in that analogy. The benchmark is the past-paper archive. Pretraining is reading the website. The reported score is the transcript. And your product is the new question nobody has published, which is the only thing you actually care about. A private eval set built from your own data, as 01-08 teaches, is the unpublished question — and that is why it outranks every public number in your decision-making, regardless of how much smaller it is.

L2 — Mechanism: how a score gets inflated, step by step

The mechanism has four stages, and naming them is how you reason about a suspicious result.

1. Publication. The benchmark's items, and usually its answer key, are posted publicly — on a project page, in a paper appendix, in a GitHub repository, in a dataset hub, and then re-posted in tutorials, blog posts, Stack Overflow answers and scraped mirrors. One dataset becomes hundreds of copies with different formatting.
2. Crawling. A pretraining corpus is built by crawling and filtering the web. Deduplication at corpus-build time removes near-identical documents, not near-identical facts, so a benchmark question that appears in fifty differently-worded tutorials survives fifty times.
3. Memorisation. During next-token prediction training (01-01), a sequence seen many times is fit more tightly than a sequence seen once. The model does not need to store the whole benchmark to be helped by it; storing the mapping from a distinctive question stem to its keyed letter is enough to move a multiple-choice accuracy number.
4. Evaluation. The benchmark is then used as a held-out test set. It is not held out. The measurement is no longer an estimate of generalisation; it is a mixture of generalisation and recall, with unknown mixing weights.

The subtlety worth internalising is that stage 4 is where the error happens, not stage 3. Memorisation is not misconduct — a model that has read the internet has read the internet. The defect is treating a set the model has read as an estimate of performance on sets it has not read. That is an inference error made by the person reporting the number.

Two related but distinct problems often get called contamination and should not be:

• Task contamination / protocol drift — the model has seen the benchmark's format extensively (thousands of four-option quizzes) without seeing its items. This inflates scores relative to a genuinely novel format, but it is a fair reflection of a real capability. It is not leakage.
• Overfitting by iteration — no leakage into pretraining at all, but a team has tuned prompts, decoding parameters and retrieval settings against the public test split fifty times. The test set has become a training set through the researcher's hands rather than the crawler's. This is the same statistical damage by a different route, and it is the one you are most likely to commit on your own project.

L3 — Depth: detection without access to the training data

You will almost never be able to grep a frontier model's pretraining corpus. Contamination detection therefore has to work from the outside, and the practical toolkit is a set of behavioural probes. None is conclusive alone; together they build a case.

N-gram overlap scanning. Where you do control a corpus — your own fine-tuning data, your RAG index, a continued-pretraining mix — you can detect overlap directly by hashing every n-gram of every benchmark item and testing the corpus against it. The GPT-3 paper's contamination analysis used 13-gram overlap as its criterion, which is a reasonable default: long enough that natural coincidence is rare, short enough to catch a reworded copy. This is mechanical, cheap, and the only direct method available.

Canary strings. Some benchmarks embed a deliberately unique identifier — BIG-bench's canary GUID is the well-known example — with the instruction that anyone building a training corpus should exclude documents containing it. You can then prompt a model to reproduce the canary. If it can, the benchmark's files were in the corpus. This is a strong positive signal when it fires and proves nothing when it does not.

Perturbation testing. Take benchmark items and rewrite them while preserving the answer: paraphrase the stem, reorder the options, rename entities, change the numbers in an arithmetic item. A model whose competence is real degrades gracefully. A model whose score rests on recall degrades sharply and specifically — it often gets the perturbed version wrong while still nailing the original wording. This is the most informative probe available to an ordinary practitioner because it requires no privileged access at all.

Option-shuffle and answer-position probes. In a four-option multiple-choice benchmark, shuffle the options so the keyed letter changes. Accuracy that drops materially under shuffling suggests the model learned an item-to-letter association rather than an item-to-content one.

Completion probes. Give the model the first half of a benchmark item and ask it to continue. Verbatim continuation of a held-out test item is not something a well-generalising model does; it is something a model that has read the file does.

Temporal splitting. Build or select evaluation items that provably postdate the model's training cutoff. This is the cleanest design available and it is the reason "held-out by date" eval sets have become a standard tactic. Its cost is that you must keep making new ones, forever, because each set's freshness expires the moment it is published.

A discipline point that ties this tier together: every one of those probes is an experiment, with a hypothesis and a control. The control for a perturbation test is the unperturbed item on the same model. The control for a shuffle test is the unshuffled ordering. If you run the probe without the control you have a number and no inference — which is the failure the whole of this module is built to prevent.

The most valuable exam asset in this lesson is the mapping from evaluation instrument to the decision it is licensed to make. Confusing these is the distractor family that questions in this area are built on.

Read that table top to bottom and you have the shape of the entire experimentation module: benchmarks screen, private eval sets decide, offline A/B ranks variants, online A/B proves causality, monitoring watches. Public benchmarks sit at the top because they are cheapest and weakest; monitoring sits at the bottom because it is continuous and non-causal. 12-14 picks up the monitoring end of that chain.

A second comparison, narrower and more directly examinable — the benchmark families themselves:

Notice what changes down that table: the unit of evaluation moves from a labelled sentence pair to a multiple-choice item to an open-ended interaction judged by a human. That progression is the same one your own evaluation practice goes through as a project matures, and it is why 09-03 (human evaluation and rubrics) and 09-10 (LLM-as-a-judge) live where they do in the measurement thread.

Constructed scenario. All numbers below are invented to make the arithmetic legible. They are not measured results for any real model or benchmark, and you should not carry them out of this lesson as facts.

You inherit a model-selection recommendation from a colleague who has left. It says: "Model B beats Model A on our domain because B scores higher on a 200-item public multiple-choice benchmark in our vertical." You have the benchmark, the two models, and no access to either training corpus. Here is the sequence of experiments that turns that assertion into a defensible finding or kills it.

Step 1 — Reproduce the baseline, exactly. Re-run both models on all 200 items with the same prompt template, the same decoding settings, and the same answer-extraction rule. Suppose you observe Model A at 132/200 and Model B at 156/200.

Model A: 132 / 200 = 0.660
Model B: 156 / 200 = 0.780
Observed gap: 0.780 - 0.660 = 0.120  (12.0 percentage points)

Step 2 — Ask whether the gap could be noise before asking whether it is real. For a proportion, the standard error is the square root of p(1−p)/n. For Model B:

SE_B = sqrt(0.780 * 0.220 / 200) = sqrt(0.1716 / 200) = sqrt(0.000858) = 0.0293
SE_A = sqrt(0.660 * 0.340 / 200) = sqrt(0.2244 / 200) = sqrt(0.001122) = 0.0335
SE of the difference (treating the arms as independent):
  sqrt(0.0293^2 + 0.0335^2) = sqrt(0.000858 + 0.001122) = sqrt(0.001980) = 0.0445
95% interval on the gap: 0.120 ± 1.96 * 0.0445 = 0.120 ± 0.087 → [0.033, 0.207]

The interval excludes zero, so the 12-point gap is unlikely to be pure sampling noise on this benchmark. Note two things about that sentence. First, it says on this benchmark — the inference is about these 200 items' population, nothing wider. Second, the independence assumption is conservative here: both models answered the same items, so a paired analysis would give a tighter interval. 09-09 develops the paired case; the unpaired version above is the safe, easy-to-defend arithmetic.

Step 3 — Probe for contamination with a perturbation experiment. Take a random 60 of the 200 items. Rewrite each stem as a paraphrase that preserves the answer, and independently shuffle the four options so the keyed letter changes. You now have a matched pair for every item: original and perturbed. Score both models on both versions. Suppose you observe:

                Original (60)   Perturbed (60)   Drop
Model A          40 / 60 = .667   37 / 60 = .617   -0.050
Model B          47 / 60 = .783   32 / 60 = .533   -0.250

Model A loses 5 points under perturbation. Model B loses 25. Both models faced the same paraphrases and the same shuffles, so the perturbation is a shared treatment and the difference in drop is the interesting quantity: −0.250 − (−0.050) = −0.200. Model B's advantage on the original wording does not survive rewording. That is the signature of item-level recall rather than capability, and it is exactly what the original leaderboard-style comparison could not see.

Step 4 — Look at the perturbed comparison, which is the estimate you actually wanted.

Perturbed: Model A .617 vs Model B .533 → gap = -0.084 in A's favour
SE_A = sqrt(.617*.383/60) = sqrt(.2363/60) = sqrt(.003938) = 0.0628
SE_B = sqrt(.533*.467/60) = sqrt(.2489/60) = sqrt(.004148) = 0.0644
SE of difference = sqrt(.003938 + .004148) = sqrt(.008086) = 0.0899
95% interval: -0.084 ± 1.96*0.0899 = -0.084 ± 0.176 → [-0.260, 0.092]

The perturbed comparison is inconclusive: the interval spans zero, so on reworded items you cannot distinguish the two models with 60 items. This is the honest finding, and it is a far more useful deliverable than either "B is 12 points better" or "B is contaminated". Written up, it reads: the original benchmark favours B by 12 points; that advantage vanishes under paraphrase and option shuffling, and on reworded items 60 samples are too few to separate the models. Recommendation: build a 200-item private eval set from our own tickets before choosing.

Step 5 — Add the direct check where you can. You cannot scan the vendors' corpora, but you can scan your own RAG index and any fine-tuning data you plan to use, with a 13-gram overlap test against the benchmark. If your own index contains the benchmark, every future evaluation you run against it is compromised by your own hand, and that is squarely your responsibility rather than the vendor's.

Total cost of that investigation: four model runs over 200 items, one over 60 paraphrases, and about an hour of arithmetic. Compare that with the cost of picking the wrong base model for a quarter.

Public benchmarks are not worthless. They are misused. The distinction is entirely about which question you are asking.

The rule underneath the table: a public benchmark can support a comparison and can never support a prediction. It can tell you that two models differ on those items under that protocol. It cannot tell you what will happen when your users type something nobody has published.

One more caveat that the exam likes in scenario form: even an uncontaminated benchmark score is subject to Goodhart's law. Once a number becomes a target, effort flows to the number rather than to the capability it proxies. A model family that has been optimised for a benchmark for two years will score well on it whether or not the underlying capability improved — no leakage required, just selection pressure. That is a distinct mechanism from contamination and it produces the same misleading number, so when you see an implausibly high score your differential diagnosis has at least three entries: leakage, iterative overfitting, and target-chasing.

This lesson serves the Experimentation section of the blueprint, which carries 22% of the exam — the third-largest block, behind Core ML and AI (30%) and Software Development (24%). Two items from that section's suggested-reading list map directly onto this page: GLUE, named explicitly, and benchmarking elementary language tasks. The section's own scope statement is "the study of how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback" — and "evaluate and interpret" is precisely the skill a contaminated benchmark defeats.

The objective-numbering defect, stated plainly. If you go looking for the official objective id that authorises this lesson, you will find a documented source defect and you should know about it before it confuses you in a scenario question. In the official study guide, the objectives printed under Experimentation as 3.1–3.5 are a verbatim duplicate of the objectives printed under Data Analysis and Visualization as 2.1–2.5. Read literally, the Experimentation section's stated objectives are about extracting insights from large datasets via data mining, comparing models with statistical metrics, conducting data analysis under supervision, creating graphs and charts, and identifying relationships and trends. Four of those five describe data-analysis work, and taken at face value they would mean that 22% of the exam has no stated coverage of model evaluation or RLHF at all — which flatly contradicts the same section's own scope sentence, quoted above, that names AI model evaluation and RLHF as its subject.

The resolution used throughout this course is to treat the printed 3.x text as a transcription artefact and to derive the real scope from three surviving sources that agree with each other: the section's own scope statement, its listed course objectives, and its suggested-reading list (A/B testing, inference optimisation, zero-shot testing, machine-translation methods, hallucinations in LLMs, GLUE, evaluating RAG applications, cross-validation, benchmarking elementary language tasks). Published candidate reports independently confirm that BLEU, hallucination mitigation and RLHF appear on the exam, which validates the derived scope over the literal text. Practically, that means: where you see an objective id cited for anything in this module, treat the id as traceability, not as a description. In this lesson the closest genuinely applicable printed objective is "compare models using statistical performance metrics" (printed as 2.2 under Data Analysis and duplicated as 3.2 under Experimentation) plus "identify relationships and trends or any factors that could affect the results of research" (2.5 / 3.5) — and contamination is, exactly and literally, a factor that could affect the results of research. That is the honest mapping. Anyone telling you the printed 3.1 text describes benchmark contamination is reading a duplicate.

How this material is phrased in questions. Because the exam is reported to be general-level rather than deeply technical, expect identification and when-to-use framings rather than arithmetic:

Distractor families to recognise. Four recur, and knowing them by name is worth more than memorising benchmark rosters:

1. The leaderboard-as-proof distractor. An option asserts that because a model tops a public benchmark it is the right choice for the described application. Almost always wrong: the described application always has a distribution the benchmark does not contain.
2. The contamination-means-privacy distractor. An option defines contamination as PII leaking out of a model, or as training data being exposed by a prompt-extraction attack. Those are real problems — 13-05 treats them — but they are the opposite direction of information flow. Contamination is evaluation data flowing into training.
3. The wrong-benchmark-for-the-task distractor. GLUE offered for measuring generation quality; MMLU offered for measuring retrieval quality; a translation metric offered for a knowledge quiz. The fix is knowing what unit each instrument scores. 09-05 is the general form of this skill.
4. The more-data-fixes-it distractor. An option proposes fixing an inflated benchmark score by adding more benchmark items or running more seeds. More items shrink the standard error; they do nothing to a bias. Bias and variance are different failures and only one of them yields to sample size — a point 09-09 makes in general and this lesson makes concretely.

Directionally, yes — contamination biases a score upward, because the model has effectively seen the answer key and memorisation can only help on the items memorised. That is why contamination is dangerous rather than merely noisy: a noise source pushes both ways and averages out with more data, while a bias pushes one way and more data makes you more confident in a wrong number.

Two honest qualifications. First, the magnitude is unknown and item-specific — a model may have seen 3% of a benchmark or 40%, and the inflation scales with the share seen and with how distinctive each item's wording is. Second, contamination of the training split of a benchmark is not contamination at all: GLUE-style benchmarks ship training splits precisely so people can train on them, and a model pretrained on that text has done nothing wrong. The defect is specific to the evaluation split being used as though it were held out.

Use behavioural probes and stack them. The five that need no privileged access are: perturbation testing (paraphrase the stem, keep the answer, and look for a sharp drop), option shuffling (change which letter is keyed and look for a drop), completion probes (give half an item and see whether the model reproduces the rest verbatim), canary retrieval (ask for a benchmark's published canary string), and temporal comparison (score the model on matched items that provably postdate its training cutoff and compare with items that predate it).

Each probe needs its own control from the same model on the same items, or you have a number without an inference. And be careful about the negative result: none of these probes can prove a benchmark is clean. They can only raise or fail to raise the alarm. The correct posture is therefore not "detect contamination and then trust the benchmark" but "treat every public benchmark as possibly contaminated and let a private set carry the decision."

Yes, for the job it can still do, which is coarse screening and rough capability floors. A model that scores near chance on MMLU (25% for four options) is telling you something real. A model that scores in the high band is telling you something ambiguous, because contamination, target-chasing and genuine capability all produce that reading and the score does not distinguish them.

The practical stance: read a high MMLU number as "this model is in the class of models that people optimise against MMLU", which is genuine information about the model's maturity and training care, and then stop. Do not read it as a per-subject competence claim for your domain, and never read a two-point MMLU gap between two models as a ranking — on ~16k items a two-point gap is around the width of a sensible confidence interval before you even consider protocol differences. If you need per-domain competence, build the eval set for that domain.

They are the same statistical failure at two different scales, and the exam will happily ask you to tell them apart.

The unifying principle is the one 01-07 states for splits and 09-08 states for cross-validation: an evaluation number is only an estimate of generalisation if the evaluated items were genuinely unavailable to the fitting process. Contamination is that rule broken by a crawler instead of by a train_test_split call in the wrong place.

Report it when your audience needs comparability and you can state the protocol completely; skip it when your audience needs a prediction about their own use.

If you do report one, the minimum honest disclosure is: benchmark name and version, split evaluated, number of items scored, shot count and where the shots came from, the exact prompt template, decoding settings including temperature, the answer-extraction rule, the date of the run, and whether you audited your fine-tuning data for overlap with the evaluation split. That last item is the one everybody omits and the one a reviewer should ask for first — if you fine-tuned on scraped domain text, an overlap scan against the benchmark is cheap and its absence is conspicuous. Pair the public number with a private-eval number and the write-up becomes genuinely useful: one figure for comparability, one for decision-relevance, and an explicit statement that they answer different questions.

1. A public benchmark is dataset + task + metric + protocol. Two numbers are comparable only when all four match, so an unlabelled score is not evidence.
2. GLUE is nine sentence-level understanding tasks from the fine-tune-per-task era; MMLU is 57-subject four-option multiple choice from the prompt-the-model-as-is era. SuperGLUE exists because GLUE saturated.
3. Contamination is benchmark items leaking into pretraining. It biases scores upward, and because it is a bias rather than noise, adding items does not fix it.
4. Distinguish three causes of an implausibly high score: leakage (corpus contains the items), overfitting by iteration (you tuned against the split), and target-chasing (Goodhart's law). The number looks identical; the remedies differ.
5. Detect contamination from outside with perturbation, option shuffling, completion probes, canaries and temporal splits — each with its own control. A negative result is weak; a positive result is strong.
6. Run the n-gram overlap scan wherever you do control the corpus: your fine-tuning mix, your RAG index, your continued-pretraining data. Contaminating your own evaluation is the version of this failure you are personally responsible for.
7. A benchmark can support a comparison and cannot support a prediction. Use benchmarks to screen a shortlist; use a frozen private eval set to decide; use an online experiment to prove a user-visible effect.
8. Always attach an interval to a benchmark delta. Standard error for a proportion is √(p(1−p)/n); on 200 items a 12-point gap is roughly at the edge of what that arithmetic can defend.
9. Never publish your private eval set, and never use a public benchmark as your CI regression gate. Publication is how a good instrument becomes a contaminated one.
10. On this module's official objective ids, remember the documented defect: the printed 3.1–3.5 duplicate Data Analysis's 2.1–2.5, so cite them for traceability and derive the actual scope from the section's own scope statement and reading list.

You now know why a published number cannot decide anything for you, and that the instrument you actually need is one built from your own task. That raises the cheapest question in the whole project, and the one most teams skip: before you build a retrieval pipeline, before you fine-tune anything, what can the base model already do if you simply ask it? Zero-shot and few-shot capability testing is the smallest experiment that exists — a handful of your own items, no training, no infrastructure — and it frequently makes the rest of the roadmap unnecessary, or reveals in an afternoon that the roadmap was aimed at the wrong problem.

Next: 10-02 builds that first cheap experiment — how to run a zero-shot probe, when adding few-shot examples genuinely helps versus merely burns context, how to keep a capability test fair across models, and how to write up a result that either kills a project or justifies it.