Bias and Fairness Auditing: Why Disaggregated Evaluation Is the Only Way to Detect Disparate Impact

Identity statement: bias, in the sense this domain tests, is a systematic performance or representation gap between subgroups that a model inherited from its training data and can amplify through training, not a property the model invents independently of what it was shown. [GROUND TRUTH] (Sources/ncp-genl/domain-10-safety-ethics-compliance.md): "Bias is inherited and can be amplified from training data." Two words in that sentence carry the whole domain's weight: "inherited" locates the origin outside the model's architecture, in the data; "amplified" adds the further claim that a model's training process does not merely reflect a skew in its input data, it can make that skew worse than it was in the source.

The amplification claim deserves a concrete mechanism, not just a label. A training objective that rewards a model for predicting the statistically likely continuation, applied to a corpus where one subgroup is systematically underrepresented or is represented in a narrower range of contexts, teaches a model that the majority pattern is simply "the pattern" — and a model optimizing hard for average loss across a training set will, without any adversarial intent anywhere in the pipeline, learn to perform best on whatever pattern dominates the data it saw most of, at the direct expense of whatever pattern it saw the least of. Nothing about this requires a mistake in model design; it is a predictable consequence of optimizing an average loss over an unrepresentative population, which is exactly why the fix belongs in the data the average was computed over, not in the optimization procedure itself.

Where bias does not come from, as often as people assume

The domain's named exam trap runs in the opposite direction from where intuition points many people first: assuming the algorithm is where bias is introduced, when the source material's framing places the origin squarely in the data. A gradient-descent optimizer, a transformer's attention mechanism, a tokenizer — none of these have any notion of demographic categories at all; they optimize whatever objective and whatever data they are given, with total indifference to what that data happens to represent. A model trained on a perfectly balanced, representative corpus with the exact same algorithm, architecture, and hyperparameters as a model trained on a skewed one will not reproduce the skewed model's disparate performance, because the algorithm was never where the skew lived. This does not mean algorithmic choices are irrelevant to fairness — some training procedures can amplify an existing data skew more than others — but it does mean that "redesign the algorithm" is very often the wrong lever to reach for first, when "audit and rebalance the data" is the lever the actual mechanism points at.

L1 — Intuition: an average is a summary that discards exactly the information you need

A weighted average compresses many numbers into one by giving each number a weight proportional to how much of the total population it represents. That is a completely reasonable thing to want most of the time — it is precisely why averages exist — but it means a subgroup's contribution to the overall number is scaled down exactly in proportion to how small that subgroup is. A subgroup that is 10% of a population can perform arbitrarily badly and still move the overall average by, at most, 10% of the gap between its own performance and everyone else's. The smaller the subgroup, the more room the aggregate number has to look fine regardless of how badly that subgroup is actually served.

L2 — Mechanism: the exact formula, and why "high overall accuracy" is compatible with "one subgroup is failing badly"

Take a system evaluated across four subgroups, with subgroup sizes n₁, n₂, n₃, n₄ and per-subgroup accuracies a₁, a₂, a₃, a₄. The aggregate accuracy is the size-weighted sum:

aggregate accuracy = (n1×a1 + n2×a2 + n3×a3 + n4×a4) / (n1 + n2 + n3 + n4)

Every term in that sum is scaled by its subgroup's share of the total. If one subgroup is small — say, 10% of the evaluated population — and its accuracy is catastrophically low, its contribution to the aggregate is still just 10% of the gap between its performance and the rest. A subgroup at 40% accuracy, in a population where every other subgroup performs at 97% and that failing subgroup is 10% of the total, moves the aggregate down by only about 5.7 percentage points relative to a hypothetical where every subgroup performed at 97% — which is enough to notice if you are looking closely, but easily lost in ordinary measurement noise, rounding, or a report that states one number to one decimal place and moves on. Make the failing subgroup smaller — 5% of the population instead of 10% — and the same catastrophic 40% accuracy moves the aggregate by roughly half as much again, while the underlying harm to that subgroup has not improved at all. The aggregate number's insensitivity to a small subgroup's failure gets worse, not better, exactly as that subgroup gets smaller and more in need of scrutiny — the arithmetic actively works against the populations most likely to be minorities in a training or evaluation set.

L3 — The exam-relevant edge case: disaggregated evaluation is not "more evaluation," it is a different measurement entirely

The critical distinction the professional exam tests is not "do more testing" — it is that disaggregated evaluation is not an intensified version of aggregate evaluation, it is a structurally different measurement that answers a question aggregate evaluation cannot answer at any sample size. No amount of additional aggregate-level testing — more test examples, more runs, tighter confidence intervals on the one blended number — will ever reveal a subgroup-specific gap, because the aggregate metric was never computing anything at the subgroup level to begin with. It is not a statistical-power problem that a bigger aggregate test set fixes; it is a specification problem, where the metric being reported was never designed to answer the question being asked. [GROUND TRUTH] (Sources/ncp-genl/domain-10-safety-ethics-compliance.md) states the fix directly and without qualification: "audit across demographic slices" and "disaggregated (per-group) evaluation is the right way to detect bias" — not a better aggregate statistic, a different kind of measurement altogether, sliced by group from the start.

The row worth re-reading is the fourth one, because it directly answers a specific and tempting wrong move: running the same aggregate evaluation on a much larger test set, or running it many times to shrink a confidence interval, produces a more precise aggregate number, but that number remains structurally blind to subgroup-level disparate impact no matter how large the sample gets, because subgroup identity was never part of what the aggregate computation tracked in the first place.

A constructed scenario, invented for teaching, with illustrative numbers. An LLM-based resume-screening assistant is evaluated for accuracy in flagging qualified candidates for human review, against a labeled held-out set of 2,000 resumes spanning four applicant subgroups defined by a protected demographic characteristic. The system reports a single number at launch: 94.1% aggregate accuracy.

Step 1 — the number the launch report actually showed.

Constructed scenario, invented for teaching — not a measured result from any real system.

Aggregate evaluation, launch report:
  n = 2,000 resumes, overall accuracy = 94.1%

Step 2 — the disaggregated table nobody built before launch.

Step 3 — verify the arithmetic that made this possible. Group C — 300 resumes, 15% of the evaluated population — performs at 75%, nineteen points below the next-lowest subgroup and twenty-three points below the top-performing subgroup. Recompute what the aggregate would have been if Group C matched Group A's 98% instead of its actual 75%: (1,078 + 480 + 294 + 82) / 2,000 = 96.7%. The actual aggregate of roughly 93.25% is only about 3.5 points below that hypothetical fully-fair aggregate — a gap easily absorbed by ordinary reporting precision, sampling variation between evaluation runs, or a report that simply states "94%" and moves on. A twenty-three-point subgroup gap produced a three-and-a-half-point aggregate effect. That ratio — a severe subgroup harm producing a mild aggregate signal — is the mechanism section 2 derived in the abstract, now shown with real numbers attached to a real-feeling scenario.

Constructed illustration, invented for teaching — not a measured result from any real system.

Group C, n = 300 (15% of the evaluated population):
  Actual accuracy:                 75.0%
  Gap vs. best-performing group:   23.0 points below Group A's 98.0%
  Aggregate accuracy, as reported: 93.25% (rounds to 94.1%)
  Aggregate accuracy, if Group C matched Group A: 96.7%
  Difference the disparate impact actually made to the aggregate: only ~3.5 points

A 23-point subgroup gap moved the reported number by roughly 3.5 points. Anyone reading
"94% accuracy" and concluding the system treats every subgroup fairly has drawn a
conclusion the number was never capable of supporting.

Step 4 — trace the cause, and name where the fix belongs. Investigation of Group C's training-relevant history finds that resumes from this subgroup were systematically underrepresented in the labeled training data used to build the screening model's fine-tuned judgment layer, and that the phrasing conventions common in this subgroup's resumes — different formatting norms, different ways of describing comparable experience — were poorly represented in what the model had seen enough of to judge confidently. This is a training-data composition and representation problem, exactly the channel [GROUND TRUTH] (Sources/ncp-genl/domain-10-safety-ethics-compliance.md) names: "representative data, rebalancing, debiasing techniques, and post-hoc filtering" are the stated mitigation categories, and every one of them operates on the data or the training process — none of them is a request-time check on any single resume being screened.

⭐ THE EARNED INSIGHT

The reason this domain treats disaggregated evaluation as the load-bearing fact of the whole bias-and-fairness objective, rather than one good practice among several, is that the arithmetic actively conceals harm in exact proportion to how small and how underrepresented the harmed group already is — the group most likely to be missing from a training set is also the group whose failure an aggregate metric is most capable of hiding, which means the populations most vulnerable to being poorly served are structurally the ones an aggregate-only evaluation practice will notice last, if it notices them at all.

This is the boundary the module's guiding question is built around, and it deserves to be stated with no hedging: a guardrail is a request-time check, evaluated on one input or one output at a time, deciding whether to allow, block, or rewrite that single request. Bias, as demonstrated in section 4, is a property of a model's learned behavior across a population — it only becomes visible when you compare performance across many requests grouped by subgroup, and it lives inside weights the training process set, not inside any single request a rail could inspect and flag. [GROUND TRUTH] (Sources/ncp-genl/domain-10-safety-ethics-compliance.md) states this precisely: "Mitigation happens at the data/training level: representative data, rebalancing, debiasing techniques, and post-hoc filtering" — and every item on that list operates before or during training, or as a correction applied across many outputs at once, never as a rule a guardrail evaluates against one incoming request.

Concretely: no rail, however well-configured, can look at a single resume-screening decision and determine that it is part of a pattern of worse performance for one subgroup, because that determination requires aggregating outcomes across hundreds or thousands of decisions and comparing them by subgroup — information no single request carries. A guardrail can restrict what a model is allowed to output about a protected characteristic, and that is a real and useful control for a different problem (for instance, blocking an output that explicitly states a demographic characteristic as a factor in a decision), but restricting what a model says about bias is not the same claim as fixing whether the model's underlying judgment actually performs equally well across groups. M10-01's own material makes the same point from the other direction: guardrails constrain outputs, they do not debias the underlying model, and this lesson is the place that claim gets its full mechanism and its full worked demonstration.

Post-hoc filtering is worth a specific caution, because it is the mitigation category most likely to be confused with a guardrail: both operate at or near serving time, and the distinction that keeps them separate is what they act on. Post-hoc filtering adjusts a model's decision thresholds per subgroup, informed by the disaggregated evaluation that found the gap in the first place — it is a correction calibrated against known subgroup identity and known subgroup performance data. A guardrail has no subgroup-performance data to calibrate against at all; it evaluates a single request against fixed rules, with no memory of how the model has performed across a population. Post-hoc filtering is a data-informed adjustment layered near the model; a guardrail is a rule-based or classifier-based screen with no model-performance awareness whatsoever.

This lesson covers objective 10.2 of the domain's three objectives, flagged for the deepest treatment in this module because the source material identifies this as one of the two most consequential traps in the whole domain, alongside the guardrail-debiasing misconception M10-01 and this lesson both address from their own angles. The scope note for the domain states the expectation directly: professional-level questions test whether you understand fairness must be measured across groups, not as one number — this lesson's entire arithmetic demonstration in section 2 and section 4 exists to make that requirement impossible to forget.

How the question tends to be phrased

Expect a scenario stating a system's high aggregate accuracy and asking what evaluation practice would have caught a described subgroup harm — the keyed answer is disaggregated, per-group evaluation, not a larger sample, not a different aggregate metric, and not a guardrail. Also expect an item that offers "the algorithm is biased" and "the training data is biased" as two options for a described disparate-impact finding, testing whether the source material's stated attribution — that bias usually originates in data — is correctly recalled over the more intuitive but less accurate algorithmic explanation.

What the distractors typically look like

The standing traps: offering a larger or more rigorously sampled aggregate evaluation as sufficient to detect subgroup harm, when no amount of aggregate-only measurement can surface what the metric was never computing; offering "retrain the algorithm" as a fix when the actual lever is the data the algorithm was trained on; and, the trap this domain names as its signature misconception, offering a guardrail or content filter as a fix for a described bias finding, when guardrails restrict outputs at request time and have no mechanism that touches what a model learned.

Because an aggregate accuracy figure is a size-weighted average, and a subgroup's contribution to that average is scaled down in exact proportion to how small a share of the evaluated population it represents. A subgroup that is a small fraction of the total can perform far worse than every other subgroup and still move the aggregate number only a modest amount — this lesson's worked example showed a twenty-three-point subgroup gap producing roughly a three-and-a-half-point effect on the reported aggregate. The smaller and more underrepresented the harmed subgroup, the less the aggregate metric moves in response to its failure, which is precisely why the populations most likely to be missing from a dataset are also the ones an aggregate-only evaluation habit is least likely to protect.

No, and this is the domain's most explicitly named misconception. A guardrail evaluates one request at a time against fixed rules or a classifier score; it has no aggregated view of how a model performs across a population of requests grouped by subgroup, which is precisely the information disaggregated evaluation exists to surface. Bias mitigation — representative data collection, rebalancing, debiasing techniques applied during training, or post-hoc filtering calibrated against known subgroup performance data — operates on the data the model learned from or on thresholds informed by that same data. None of those mitigation categories is a rule a guardrail evaluates against a single incoming request, and treating a runtime content check as a substitute for a data-level fix leaves the underlying subgroup performance gap completely unchanged.

• A high aggregate accuracy number can completely conceal a severe subgroup-specific harm. This lesson's worked example showed a twenty-three-point subgroup gap moving the reported aggregate by only about three and a half points.
• The masking effect gets worse, not better, as the harmed subgroup gets smaller. The populations most likely to be underrepresented in training data are structurally the ones an aggregate-only evaluation habit protects the least.
• Disaggregated evaluation is a different measurement, not a bigger aggregate one. No amount of additional aggregate-level testing reveals a subgroup gap, because the aggregate metric was never computing anything at the subgroup level.
• Bias is inherited and can be amplified from training data — it usually does not originate in the algorithm. Audit training-data composition and representation before redesigning anything algorithmic.
• Mitigation happens at the data and training level: representative data, rebalancing, debiasing techniques, and post-hoc filtering — never as a rule a guardrail evaluates against a single request.
• A guardrail cannot mitigate bias, full stop. It has no aggregated view across a population of requests, which is exactly the view disaggregated evaluation exists to provide and exactly what a data-level fix requires.

Detecting and mitigating bias is one distinct mechanism in this module's guiding question about what a safety control can and cannot do — a different one comes next. Next: M10-04 turns to a failure mode that looks similar on the surface (a model producing output that is wrong in a way that damages trust) but has an entirely different mechanism and an entirely different mitigation path: hallucination, where grounding a model's generation in retrieved, citable context measurably reduces the problem without ever fully eliminating it, because a model can still misuse or ignore context it was actually given.