Sample Size and Statistical Significance in LLM Evaluation: Standard Error, Intervals, and Multiple Comparisons

A statistically significant difference is one large enough that random sampling variation is an unlikely explanation for it. In LLM evaluation the "random sampling" is usually two things at once: which items happen to be in your evaluation set, and — because generation is not fully deterministic even at temperature 0 (09-11) — which outputs the model happened to produce this run.

Three quantities do all the work:

And two distinctions that separate a careful evaluation from a careless one:

• Statistical significance vs practical significance. A 0.4-point improvement measured on 50,000 items can be statistically airtight and completely irrelevant to users. Significance says "probably not zero"; it says nothing about "worth shipping". Always report the effect size — the actual difference in points — alongside any significance claim.
• Unpaired vs paired comparison. Comparing two systems' rates discards the fact that both were measured on the same items. A paired analysis looks at per-item differences, which removes item difficulty from the noise and is dramatically more sensitive. On a fixed evaluation set you should essentially always be doing paired analysis.

L1 — Intuition: the noise floor

Every evaluation has a noise floor — the size of movement you would see just from re-sampling, with no change to the system. If your noise floor is ±10 points and your improvement is 3 points, your measurement cannot see your improvement. You have not learned that the change did nothing; you have learned that your instrument is too coarse to tell.

The floor shrinks with the square root of the sample size, which is unforgiving: halving your noise floor costs four times the items.

L2 — Mechanism: the four formulas you need

Standard error of a proportion.

SE(p) = sqrt( p × (1 - p) / n )

Worst case is p = 0.5, which maximises p(1−p) at 0.25. Use p = 0.5 when planning, so your estimate is conservative.

Confidence interval (normal approximation).

95% CI ≈ p ± 1.96 × SE(p)
90% CI ≈ p ± 1.645 × SE(p)

The normal approximation is reasonable when n·p and n·(1−p) are both at least about 10. Below that — very high or very low pass rates, or small n — use a Wilson or Clopper–Pearson interval instead; the normal interval can extend below 0 or above 1, which is a visible sign it has broken.

Standard error of a difference between two independent proportions.

SE(p1 - p2) = sqrt( p1(1-p1)/n1 + p2(1-p2)/n2 )

Note what this means when both are measured on n = 100: the SE of the difference is about sqrt(2) ≈ 1.41 times the SE of a single proportion. Comparing two numbers is noisier than measuring one. That is the arithmetic behind the standard experience of "our two candidate prompts keep swapping places".

Standard error of a paired difference. If you evaluate both systems on the same n items, compute the per-item difference d_i (for pass/fail, d_i ∈ {−1, 0, +1}) and then:

mean difference d̄ = Σd_i / n
SE(d̄)             = s_d / sqrt(n)        where s_d is the sample std of the d_i

The gain is that items both systems get right, and items both get wrong, contribute d_i = 0 and add nothing to the variance. If the two systems agree on 90 of 100 items, only 10 items carry information — and the paired SE is computed from those, which is far smaller than the unpaired SE computed from all the noise in both rates.

McNemar's test is the standard formalisation for paired binary outcomes. Build the 2×2 table of agreement and look only at the discordant cells b (A right, B wrong) and c (A wrong, B right):

McNemar statistic ≈ (|b - c| - 1)² / (b + c)      compared against chi-square with 1 df

Everything the two systems agree on is irrelevant to the test, which is exactly the right instinct.

Sample size for a target detectable difference. A useful planning approximation for detecting a difference δ in proportions with 80% power at the 5% level, unpaired, equal groups:

n per group ≈ 16 × p(1-p) / δ²          (with p ≈ 0.5, this is 4 / δ²)

That approximation yields the table in §4 and is worth carrying because it makes the cost of precision vivid.

L3 — Depth: multiple comparisons, peeking, and the run-to-run floor

Multiple comparisons. If you run one comparison at the 5% significance level, you accept a 5% chance of a false positive. Run m independent comparisons and the chance that at least one is a false positive is:

P(at least one false positive) = 1 - (1 - α)^m

Twelve prompt variants is not an exotic number; it is a normal afternoon. And the arithmetic says that with twelve variants tested at the 5% level, the probability that at least one beats the baseline purely by chance is about 46%. Since you will naturally report the best one, your reported result is selected precisely for being lucky.

Two standard corrections:

• Bonferroni: test each comparison at α/m instead of α. With m = 12 and α = 0.05, each variant must clear 0.00417. Simple, conservative, and it makes it very clear how much evidence twelve variants actually demand.
• Benjamini–Hochberg (FDR): controls the expected proportion of false discoveries rather than the chance of any, which is less conservative and often more appropriate for exploratory screening.

The cheapest correction of all is procedural: screen many variants on a cheap set, then confirm the single winner on a fresh, held-out set. Confirmation on new data is worth more than any correction formula, because it converts a selection problem into a single test.

Peeking / optional stopping. Checking your evaluation repeatedly and stopping when it looks good inflates the false-positive rate for the same structural reason as multiple comparisons — each check is another chance to be lucky. Fix n in advance, or use a sequential-testing method designed for it. This matters most for online A/B tests (10-03), where the temptation to stop early is strongest.

The run-to-run floor. LLM evaluation has a noise source classical ML does not: the same prompt on the same items can produce different outputs across runs, even at temperature 0, because of non-deterministic kernel reductions, batching effects, and provider-side changes (09-11). Before believing any difference, run your baseline twice and measure the gap. If re-running the same configuration moves the score by 2 points, then 2 points is your empirical noise floor and no 2-point improvement means anything, whatever the arithmetic says. This single check — the same config, twice — is the highest-value ten minutes in evaluation work and it is almost never done.

Judge-based metrics compound the problem. If your metric is an LLM judge (09-10), the judge itself is stochastic and biased, so you have sampling noise on items, generation noise on outputs, and judgement noise on scores. Report the variance you measured rather than assuming it away.

These four get used interchangeably in write-ups and mean different things.

The two most damaging misreadings are worth spelling out. A p-value is not the probability that your improvement is real, and a p-value of 0.049 versus 0.051 is not a meaningful distinction. And a non-significant result is not evidence that the change did nothing — with n = 100 you are simply unable to resolve small effects, which is a statement about your instrument, not about your change. The correct report in that case is "we could not detect a difference; our design could only have detected differences larger than X points."

Also worth separating from all of the above: fold standard deviation in cross-validation (09-08) is not a standard error, because the folds share training data and are therefore not independent samples.

Constructed example throughout. Baseline prompt A and candidate prompt B are both evaluated on the same frozen 100-item set (09-01).

• A: 71 of 100 pass → p_A = 0.71
• B: 75 of 100 pass → p_B = 0.75

The observed improvement is +4 points. Three analyses, in increasing sophistication.

Analysis 1 — the single-rate interval

SE(p_A) = sqrt(0.71 × 0.29 / 100) = sqrt(0.2059 / 100) = sqrt(0.002059) = 0.04538
95% CI for A = 0.71 ± 1.96 × 0.04538 = 0.71 ± 0.0889 → [0.621, 0.799]

SE(p_B) = sqrt(0.75 × 0.25 / 100) = sqrt(0.1875 / 100) = sqrt(0.001875) = 0.04330
95% CI for B = 0.75 ± 1.96 × 0.04330 = 0.75 ± 0.0849 → [0.665, 0.835]

A: 71.0% [62.1%, 79.9%]. B: 75.0% [66.5%, 83.5%]. The intervals overlap heavily — B's interval contains A's point estimate and vice versa. Already this says the 4-point difference is not established.

Analysis 2 — the unpaired difference

SE(p_B - p_A) = sqrt( 0.75×0.25/100 + 0.71×0.29/100 )
              = sqrt( 0.001875 + 0.002059 )
              = sqrt( 0.003934 )
              = 0.06272

observed difference = 0.04
z = 0.04 / 0.06272 = 0.638

A z of 0.638 is nowhere near the 1.96 needed for 95% confidence; the two-sided p-value is about 0.52. The 95% CI for the difference is:

0.04 ± 1.96 × 0.06272 = 0.04 ± 0.1229 → [-0.083, +0.163]

The difference is +4 points, 95% CI [−8.3, +16.3]. The interval comfortably includes zero and even includes B being 8 points worse. Treating this as an improvement and shipping it is exactly the self-deception this lesson is about.

Analysis 3 — the paired analysis, which is the right one

Both prompts ran on the same 100 items, so build the agreement table. Constructed but realistic agreement pattern:

The two prompts agree on 68 + 22 = 90 items. Only the 10 discordant items carry any information: b = 3 (A right, B wrong) and c = 7 (A wrong, B right). Note the margins reproduce the rates: 71 and 75, and c − b = 4, which is the 4-item, 4-point difference.

Paired standard error. The per-item differences are: +1 on 7 items, −1 on 3 items, 0 on 90 items.

d̄ = (7×(+1) + 3×(-1) + 90×0) / 100 = 4 / 100 = 0.040

Σ(d_i - d̄)²:
  7 items at +1:  7 × (1 - 0.04)²    = 7 × 0.9216  = 6.4512
  3 items at -1:  3 × (-1 - 0.04)²   = 3 × 1.0816  = 3.2448
 90 items at  0: 90 × (0 - 0.04)²    = 90 × 0.0016 = 0.1440
                                        sum        = 9.8400

s_d   = sqrt(9.8400 / 99) = sqrt(0.09939) = 0.31527
SE(d̄) = 0.31527 / sqrt(100) = 0.031527

t = 0.040 / 0.031527 = 1.269
95% CI = 0.040 ± 1.96 × 0.031527 = 0.040 ± 0.0618 → [-0.022, +0.102]

The paired 95% CI is [−2.2, +10.2] points, versus the unpaired [−8.3, +16.3]. Pairing halved the interval width (12.4 points versus 24.6) because the 90 items both prompts agreed on stopped contributing noise. And yet the interval still includes zero: t = 1.269 falls short of 1.96, so even the sensitive test cannot establish a 4-point difference at 100 items.

McNemar's test on the same table:

(|b - c| - 1)² / (b + c) = (|3 - 7| - 1)² / (3 + 7) = (4 - 1)² / 10 = 9 / 10 = 0.90

A chi-square statistic of 0.90 on 1 degree of freedom corresponds to a p-value of about 0.34. Same conclusion: not significant.

Analysis 4 — so what would it take?

Using the planning approximation n ≈ 4 / δ² per group (unpaired, p ≈ 0.5, 80% power, α = 0.05):

The paired column is approximate — it depends on the agreement rate, and higher agreement helps more — but the message is robust: pairing is worth roughly a 3× reduction in required sample size at 90% agreement, which is the cheapest statistical improvement available to you. Use the same items, always.

The honest report for our example: "Prompt B scored 75% versus prompt A's 71% on the frozen 100-item set (paired difference +4 points, 95% CI [−2.2, +10.2], McNemar p ≈ 0.34). We could not detect a difference; this design could only have resolved differences larger than about 10 points. B is not worse, and may be modestly better; we did not establish it."

That paragraph is what a competent evaluation looks like, and it is dramatically more useful than "B improved by 4 points" — because it tells the reader what the measurement could and could not have seen.

Constructed example. A team generates twelve prompt variants and evaluates each against the same 100-item set. The baseline scores 71%. Results:

V4 at 81% looks like a clear winner: +10 points, which by the §4 table is roughly the detectable threshold at n = 100. Ship it?

Step 1 — the family-wise error rate.

P(at least one of 12 false positives at α = 0.05) = 1 - (0.95)^12
(0.95)^12 = 0.5404
P = 1 - 0.5404 = 0.4596

About a 46% chance that at least one variant would clear the 5% bar by luck alone, even if all twelve were identical to the baseline. Reporting the maximum of twelve noisy draws is reporting a selected extreme.

Step 2 — Bonferroni-corrected threshold.

corrected α = 0.05 / 12 = 0.004167
critical z for two-sided 0.004167 ≈ 2.87   (versus 1.96 uncorrected)

V4's unpaired z against the baseline:

SE = sqrt(0.81×0.19/100 + 0.71×0.29/100) = sqrt(0.001539 + 0.002059) = sqrt(0.003598) = 0.05998
z  = 0.10 / 0.05998 = 1.667

z = 1.667 does not even clear the uncorrected 1.96, let alone the corrected 2.87. V4's +10 points is not significant even before correcting for the twelve comparisons.

Step 3 — sanity-check against the spread of the whole family. The twelve differences range from −4 to +10, a 14-point spread, with a mean of about +0.4 points. That spread is roughly what you would expect from pure noise at n = 100: the SE of a difference is about 6 points, so draws of ±10 are unremarkable. The shape of the family is itself evidence. If your best variant's advantage is comparable to the family's natural spread, you have found the top of a noise distribution.

Step 4 — the procedure that actually works.

1. Screen all twelve on the 100-item set. Keep the top two or three as candidates, and treat their scores as unreliable estimates, not results.
2. Confirm the finalists on a fresh set of items they have never been scored on — or, if you cannot get fresh items, on a substantially larger set.
3. Use paired analysis on the confirmation run, which reduces the required size by roughly 3× at typical agreement rates.
4. Re-run the baseline in the same session so run-to-run noise (09-11) is measured, not assumed.
5. Report the effect size with its interval, not just the winner's score.

Screening then confirming is the whole discipline. It costs one extra evaluation run and it converts a 46% false-discovery risk into a single clean test.

Step 5 — what if you cannot afford a confirmation set? Then say so, and downgrade the claim. "V4 was the best of twelve variants on our 100-item set at 81% versus a 71% baseline; with twelve comparisons this is not statistically distinguishable from chance selection, and we are treating it as a hypothesis rather than a result." A team that writes that sentence will not spend a quarter defending V4.

The Experimentation domain is 22% of the exam and its official scope statement is explicitly about experimental method: "the study of how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)." Interpreting an experiment correctly is precisely this lesson. The domain's suggested-reading list also names A/B testing, which is where these ideas are usually first met.

The objective-numbering defect, stated where the objectives are cited. The official study guide prints this domain's objectives as 3.1–3.5, and those five lines are a verbatim duplicate of the Data Analysis domain's 2.1–2.5: data-mining awareness, comparing models using statistical performance metrics, conducting data analysis under supervision, creating graphs and charts, and identifying relationships and trends. Data Analysis is 14% of the exam and Experimentation is 22%, yet the printed objective text is identical, so read literally the larger domain has no stated coverage of model evaluation or RLHF at all. The scope statement above and the suggested-reading list both contradict that reading, and published candidate reports independently confirm evaluation, hallucination and RLHF content on the exam. The derived scope governs. The objectives that legitimately apply to this lesson are the duplicated pair 2.5 / 3.5 — "identify relationships and trends or any factors that could affect the results of research", which is a remarkably good description of sampling noise, peeking and multiple comparisons — plus 2.2 / 3.2 (compare models using statistical performance metrics) and 1.5 (ML fundamentals including model comparison).

Question phrasings to expect:

• "A team evaluates a prompt change on 20 examples and observes a 10% improvement. What is the main problem with concluding the change helped?" → The sample is far too small; the confidence interval at n = 20 is roughly ±22 points, so a 10-point move is well inside the noise.
• "What does the standard error of a proportion depend on?" → The observed proportion and the sample size, as sqrt(p(1−p)/n); it shrinks with the square root of n.
• "To halve the width of a confidence interval, how much more data do you need?" → Four times as much, because SE falls as sqrt(n).
• "A team tests 12 prompt variants and reports the best one as a significant improvement. What error has been made?" → Multiple comparisons / selection bias; the family-wise false-positive probability is far above the nominal 5%.
• "What does a p-value of 0.04 mean?" → The probability of observing a difference at least this large if there were truly no difference is 4%. It is not the probability that the effect is real.
• "Why is a paired comparison more sensitive than comparing two independent rates?" → Because both systems are measured on the same items, item difficulty cancels and only the discordant items contribute variance.
• "A result is statistically significant but the difference is 0.3 points. Should you ship it?" → Significance is not practical importance; weigh the effect size against cost and risk.
• "What is the risk of checking results repeatedly and stopping when the difference becomes significant?" → Optional stopping inflates the false-positive rate, the same mechanism as multiple comparisons.
• "A non-significant result is obtained. What can you conclude?" → Not that there is no effect — only that the study lacked the power to resolve an effect of that size.

Distractor families. (1) A small sample presented as sufficient — the single most common. (2) p-value misinterpreted as the probability the hypothesis is true. (3) "Not significant" read as "proven no effect." (4) Statistical significance conflated with practical importance. (5) More comparisons offered as more evidence — testing more variants described as strengthening the finding, when it weakens it. (6) Standard deviation offered where standard error belongs — SD does not shrink with n, SE does. (7) An unpaired comparison offered when the items are identical — technically valid, needlessly weak, and often the wrong option when a paired alternative is listed.

Roughly 1,600 per group with an unpaired comparison at conventional 80% power and a 5% significance level, or roughly 500 with a paired design on the same items at typical agreement rates. That is the honest answer, and it is why most reported 5-point prompt improvements in the industry are not real.

Three ways to make the problem tractable rather than just accepting defeat. Pair everything — same items, both systems, analyse the per-item differences; this is free and buys about a 3× reduction. Use a graded metric rather than pass/fail where the task allows — a 0–4 rubric score or a continuous metric carries more information per item than a binary outcome, so the same n resolves smaller differences. And accept larger detection thresholds early in a project: at a hundred items you can reliably see 20-point moves, which is exactly the size of improvement that early work produces. Chase the 5-point refinements only once the 20-point ones are exhausted, and by then you will likely have enough automated metric coverage to afford the items.

Statistical significance answers "is this difference distinguishable from sampling noise?" Practical significance answers "is this difference big enough to matter?" They are independent, and all four combinations occur.

The bottom-left cell is where most damage happens, in both directions. Teams abandon promising changes because a small study did not reach significance, and teams ship changes on the strength of a non-significant "improvement". The discipline that prevents both is the same: always report the effect size and its confidence interval, and always state the smallest effect your design could have detected. A report containing those two things cannot be misread in either direction.

Paired, essentially always, because the standard practice in this module is to run every configuration against the same frozen evaluation set — which is the definition of paired data. The gain is not marginal. In the §4 example, pairing cut the 95% interval from 24.6 points wide to 12.4 points wide on identical data, purely by recognising that the 90 items both prompts handled the same way contain no information about their difference.

The mechanics for the common cases: pass/fail outcomes → McNemar's test on the discordant cells, or a paired proportion interval; graded scores → paired t-test or Wilcoxon signed-rank on the per-item differences; preference comparisons → a sign test or binomial test on the wins, discarding ties. Use an unpaired test only when the two systems genuinely saw different items — for instance a live A/B test where traffic is randomly assigned to arms (10-03), which is unpaired by construction and therefore needs the larger sample sizes.

One caution: pairing removes item-difficulty noise but not run-to-run generation noise. If you re-run the same configuration and it scores differently, that variance is still in your paired differences, and the only way to know its size is to measure it — the same-config-twice check from §2.

Because a modern LLM evaluation has at least three independent noise sources, and only the first is the one people think about.

Item sampling. If you changed the evaluation set, the score changes. This is why the set must be frozen and versioned (09-01).

Generation non-determinism. Even at temperature 0, the same prompt can produce different outputs across runs. Floating-point reductions on GPU are not associative and their order can depend on batch composition; kernel selection and hardware can vary; and a hosted model behind an API can change under you without notice. 09-11 treats this in full. The practical consequence: run the same configuration twice and use the observed gap as your empirical noise floor, then refuse to believe improvements smaller than that.

Metric non-determinism. If your metric is an LLM judge, the judge is itself a stochastic generator with its own biases and its own version drift (09-10). Deterministic metrics — exact match, ROUGE, BLEU, BERTScore with a pinned encoder — remove this source entirely, which is one of their underrated advantages as CI tripwires (10-04).

Add a fourth for completeness: the corpus, for RAG systems. Re-indexing, an embedding-model change, or new documents will move every retrieval metric with no change to the prompt or model at all (12-12, 09-07). Version the corpus alongside the eval set, or your time series is measuring two things at once.

1. SE = sqrt(p(1−p)/n). At n = 100, p = 0.5, SE = 0.05 and the 95% interval is about ±10 points.
2. Error falls as sqrt(n). Halving the interval costs four times the items.
3. Comparing two rates is sqrt(2) noisier than measuring one. The worked example's unpaired 95% CI for a +4-point difference was [−8.3, +16.3].
4. Pair everything. The same data analysed paired gave [−2.2, +10.2] — half the width — because 90 agreeing items stopped contributing noise.
5. McNemar's test uses only the discordant cells. With b = 3 and c = 7, the statistic was 0.90 and p ≈ 0.34.
6. A 4-point improvement needs roughly 2,500 items unpaired, or about 800 paired. At 100 items you can see 20-point moves, not 4-point moves.
7. Twelve variants at α = 0.05 gives a 46% chance of a spurious winner. 1 − 0.95^12 = 0.4596.
8. Screen then confirm. One extra run converts a selection problem into a single clean test; it beats any correction formula.
9. Run the baseline twice. The gap is your empirical noise floor, and it costs ten minutes.
10. A p-value is not the probability the effect is real, and "not significant" is not "no effect" — report the minimum detectable effect instead.
11. Always report the effect size with its interval. Statistical significance and practical importance are independent, and both directions of confusion are expensive.
12. Size per slice, not in total. A 500-item set says nothing about a nine-item subgroup.

Everything above assumed you had a score per item. On a hundred items with a human rubric, getting those scores costs an afternoon; on a thousand items it costs a week, and the sample sizes this lesson just derived make a thousand items look necessary rather than luxurious. The obvious move is to have a model do the scoring — which works better than sceptics expect and fails in specific, documented, reproducible ways that will bias your numbers if you do not know their names.

Next: 09-10 covers LLM-as-a-judge: how a judge is constructed, how to validate one against human labels, and the four biases — position, verbosity, self-preference, and rubric drift — that make an unvalidated judge worse than no metric at all.