Human Evaluation of LLMs: Rubrics, Inter-Annotator Agreement, and Cohen's Kappa

Human evaluation is the direct scoring of model outputs by people, against criteria written down in advance. It is the only evaluation method that can assess properties no reference string encodes — helpfulness, tone, whether an answer is actually responsive, whether a refusal was appropriate, whether a summary distorted the source's emphasis. Every reference-based metric in this module is ultimately a cheap approximation of a human judgement that someone made once.

Inter-annotator agreement (IAA) — also called inter-rater reliability — is the degree to which independent annotators assign the same label to the same item. It is a property of the instrument, not of the model under test. You compute it on a deliberately-overlapped subset: give annotators A and B the same 30 items, without letting them see each other's work, and compare.

The three artefacts that make human evaluation reproducible:

And the statistic that binds them: agreement, measured, reported, and improved before the labels are used for anything.

L1 — Intuition: percent agreement flatters you

Two annotators label 100 outputs as "faithful" or "unfaithful". They agree on 90. Ninety percent — excellent, surely?

Not necessarily. Suppose 95% of outputs really are faithful, and both annotators know it. If both simply labelled everything "faithful" without reading anything, they would agree 100% of the time and their labels would carry zero information. High agreement on a skewed task is cheap. What you want to know is: how much better than chance is this agreement? That is exactly what kappa computes — it subtracts the agreement you would expect from the annotators' marginal label frequencies alone, and rescales what remains.

L2 — Mechanism: Cohen's kappa, step by step

Cohen's kappa for two annotators on a categorical label set:

κ = (P_o - P_e) / (1 - P_e)

P_o = observed agreement  = (# items both labelled the same) / N
P_e = expected agreement  = Σ over labels L of [ P(A picks L) × P(B picks L) ]

The numerator is "how much agreement above chance you achieved". The denominator is "how much agreement above chance was available". The ratio is the fraction of achievable non-chance agreement you actually got.

Kappa's range and reading:

Those bands are a widely-used convention, not a law of nature, and different fields set different thresholds — a safety-critical label deserves a higher bar than a tone preference. Treat them as a triage guide.

Which coefficient for which situation:

L3 — Depth: kappa's paradoxes and what to do about them

Kappa is better than percent agreement and still not a truth serum. Three known pathologies:

The prevalence paradox. When one label dominates, P_e is very high, so the denominator 1 - P_e is tiny, and kappa becomes hypersensitive: a couple of disagreements can crash it from 0.8 to 0.3 while percent agreement barely moves. A task where 97% of outputs are faithful can show high percent agreement and low kappa simultaneously, and both numbers are "correct". The response is not to discard kappa but to report both, plus the label distribution, and to consider enriching the annotation sample with likely-positive cases so the label distribution is less skewed. That is the same over-sampling logic as the stratified evaluation set in 09-01.

The bias paradox. Kappa can be higher when annotators have systematically different marginal rates than when they have similar ones, for the same observed agreement, because different marginals lower P_e. So a rising kappa does not always mean a better instrument. Always look at the confusion matrix between the two annotators, not just the scalar.

Multi-label and free-text collapse. Kappa needs a fixed label set. If your rubric asks annotators to write a reason, or to select any number of applicable error tags, kappa is not directly defined. Use Krippendorff's alpha with an appropriate distance function, or reduce the task: score each error tag as its own binary decision and compute agreement per tag. The per-tag view is usually far more useful anyway, because it tells you which criterion is ambiguous.

One more depth point that matters for RLHF. Preference data — "which of these two responses is better?" — has a structurally low agreement ceiling, because many pairs are genuinely close. It is standard practice to allow a "tie" or "about equal" option specifically so that annotators are not forced to manufacture a preference where none exists; forcing a choice on close pairs injects pure noise into the reward model. 11-07 covers what that noise does to a reward model, and 11-06 covers where preference data sits in the pipeline.

The relationship is hierarchical, and getting the hierarchy right is the point: human labels define the target; automatic metrics and judges are approximations whose quality is measured by how well they agree with human labels. That means every LLM judge you deploy should be validated against a human-labelled subset, and the validation statistic is the same agreement statistic from this lesson — judge-versus-human kappa. If your judge agrees with humans at kappa 0.5 and your humans agree with each other at kappa 0.55, the judge is nearly as good as a human and you should ship it. If your humans agree at 0.85 and the judge at 0.45, the judge is not ready. You cannot interpret a judge's agreement number without knowing the human-human number, which is the reason this lesson comes before 09-10.

Two annotators independently label 100 RAG answers as Faithful (F) or Unfaithful (U). These counts are constructed for the arithmetic, not measured from a real annotation run.

The confusion matrix between annotators:

Step 1 — observed agreement. The diagonal is where they agree.

P_o = (70 + 12) / 100 = 82 / 100 = 0.820

Eighty-two percent. Sounds respectable.

Step 2 — expected agreement from the marginals. Annotator A said Faithful 80/100 = 0.80 of the time; B said Faithful 78/100 = 0.78. A said Unfaithful 0.20; B said Unfaithful 0.22.

P_e = P(both say F)         + P(both say U)
    = (0.80 × 0.78)         + (0.20 × 0.22)
    = 0.6240                + 0.0440
    = 0.6680

Two annotators with those habits would agree 66.8% of the time even if they were labelling at random within those habits.

Step 3 — kappa.

κ = (P_o - P_e) / (1 - P_e)
  = (0.820 - 0.668) / (1 - 0.668)
  = 0.152 / 0.332
  = 0.4578

κ ≈ 0.46 — "moderate". The 82% headline collapses to a coefficient that says the rubric is only capturing about 46% of the achievable non-chance agreement. This gap between percent agreement and kappa is the single most important thing to internalise about IAA, and it is worth being able to reproduce the arithmetic cold.

Step 4 — read the disagreement structure. The off-diagonal cells are 10 and 8, roughly symmetric, so neither annotator is systematically stricter — this is genuine ambiguity, not a calibration offset between people. Had the cells been 18 and 0, we would have a bias problem: annotator A calls things unfaithful that B calls faithful, consistently, which is fixed by a calibration session rather than a rubric rewrite.

Step 5 — improve the rubric and recompute. Reading the 18 disagreements reveals that most of them are answers that are partly supported: the main claim is grounded, an incidental detail is not. The rubric offered only two labels, so annotators were splitting on where they personally drew the line. We rewrite the rubric with three labels and an explicit decision rule:

• Fully grounded — every factual claim in the answer is supported by the retrieved context.
• Partially grounded — the answer's primary claim is supported but at least one secondary claim is not.
• Ungrounded — the primary claim is not supported by the context.

Re-annotating 100 items with the new rubric gives:

P_o = (61 + 16 + 10) / 100 = 0.870

P_e = (0.65 × 0.66) + (0.23 × 0.22) + (0.12 × 0.12)
    = 0.4290        + 0.0506        + 0.0144
    = 0.4940

κ = (0.870 - 0.494) / (1 - 0.494) = 0.376 / 0.506 = 0.7431

κ ≈ 0.74 — "substantial". Note what happened: percent agreement rose only modestly (0.820 → 0.870, five points), but kappa rose enormously (0.46 → 0.74). The reason is in P_e: adding a third label and spreading the distribution dropped expected chance agreement from 0.668 to 0.494, so there was far more achievable agreement to capture, and the sharper rubric captured it. Splitting an ambiguous binary into a scale with a named middle category is the single highest-yield rubric fix, and this arithmetic is why.

How many annotators? Two on an overlap subset is the minimum that produces an agreement number at all, and it is enough to diagnose a rubric. Three lets you adjudicate by majority instead of by discussion, which is what you want once labelling is a production process. More than three rarely pays for itself unless the task is genuinely contested, in which case the disagreement is the finding and should be reported as such rather than averaged away. The overlap subset does not need to be the whole set: a standard design is 100% overlap during rubric development, then 10–20% overlap during production labelling as an ongoing drift check.

This topic is named directly in the Experimentation domain's official scope statement, which is worth quoting because it is the strongest textual support anything in this module has: the domain covers "how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)". Human subjects in labelling is not a derived topic here — it is one of two activities the scope statement names explicitly.

The objective-numbering defect. The official study guide prints Experimentation's objectives as 3.1–3.5, and those five lines are a verbatim duplicate of the Data Analysis objectives 2.1–2.5: data mining, data analysis under supervision, chart creation, trend identification. Read literally, they contain no mention of human labelling, RLHF, or model evaluation at all — which directly contradicts the scope statement above. The printed IDs are retained for traceability, but the domain's real content is derived from its own scope statement and its suggested-reading list. The objectives that legitimately apply to this lesson are 2.3 / 3.3 (conduct data analysis under the supervision of a senior team member — annotation work is exactly the supervised-contributor activity the job-role frame describes), 2.5 / 3.5 (identify factors that could affect the results of research — annotator disagreement is such a factor), and from Trustworthy AI, 5.2 (data privacy and consent, which governs how human labellers and the data they see are treated) and 5.4 (minimising bias, since annotator pools are a bias source).

Question phrasings to expect:

• "Two annotators agree on 90% of labels. Why might inter-annotator agreement still be considered poor?" → Because percent agreement does not correct for chance; with a skewed label distribution, expected agreement is already high, so kappa can be low.
• "What does Cohen's kappa measure?" → Agreement between two annotators corrected for the agreement expected by chance.
• "Inter-annotator agreement on a rubric is 0.35. What is the most appropriate next step?" → Revise the rubric and re-train annotators. Distractors: discard the low-agreement items, average the two labels, hire more annotators, use a larger model.
• "Why measure inter-annotator agreement before training a reward model?" → Because label noise from an ambiguous rubric propagates into the reward model and is then optimised against, producing reward hacking.
• "Which is the appropriate agreement statistic for five annotators on an ordinal scale?" → Krippendorff's alpha (or weighted/Fleiss variants); Cohen's kappa handles exactly two annotators on nominal labels.
• "What is the main advantage of human evaluation over BLEU for an open-ended generation task?" → It can assess properties no reference string encodes, and it does not penalise valid paraphrase.

Distractor families. (1) Percent agreement offered as sufficient — the most common. (2) Blame the annotators — options proposing to replace or retrain people when the rubric is the defect; retraining helps calibration bias, not ambiguity. (3) Average away the disagreement — options that mean-pool conflicting labels instead of adjudicating, which destroys exactly the information you needed. (4) Skip humans entirely and use an LLM judge — plausible-sounding, but a judge with no human-labelled validation set has no known accuracy, and validating it is what human labels are for. (5) More annotators as a fix for a bad rubric — more annotators on an ambiguous rubric produces more precise measurement of the ambiguity.

By the common convention, κ above 0.8 is almost perfect, 0.6–0.8 substantial, 0.4–0.6 moderate, and below 0.4 means the labels are not reliable enough to build on. Those bands are a heuristic, not a standard, and the right threshold depends on stakes: a tone-preference label can be useful at 0.5, while a label determining whether content is harmful should be far higher before it gates anything. Two calibrations matter more than the absolute number. First, compare your κ to the best achievable κ for that task — some judgements are irreducibly subjective, and 0.6 on "which summary is more helpful" may be near the human ceiling. Second, compare it to the κ of anything you plan to substitute for humans: a judge that matches human-human agreement is as good as a human, whatever the absolute figure. And always report κ together with percent agreement and the label distribution, because any one of the three alone can mislead.

In this order, because the order is the cheapest-first order:

1. Read the disagreements. Pull the items where annotators differ and look for a pattern. Almost always, one or two specific situations account for most disagreement — a partially-grounded answer, an out-of-scope question, an answer that is correct but rude. That situation is the rubric's missing category.
2. Split or merge labels. Add a named middle category, or collapse two labels annotators cannot distinguish. Both moves raise κ, and the arithmetic in §4 shows why splitting can raise it dramatically.
3. Replace adjectives with observable conditions. "Accurate" is an adjective; "every numeric figure appears in the retrieved context" is an observable condition. Rubric criteria should be checkable by pointing at the text.
4. Add worked edge cases to the guideline. Three examples of each borderline call are worth more than a paragraph of definition.
5. Run a calibration session. Everyone labels the same ten items, then discusses. This fixes systematic bias between annotators (asymmetric off-diagonal cells), which is a different disease from ambiguity (symmetric off-diagonal cells).
6. Only then change the annotator pool, and only if the task genuinely requires expertise the pool lacks.

Note that steps 1–4 change the instrument and step 5 changes the people, and you diagnose which you need from the shape of the annotator confusion matrix, not from the scalar κ.

Because both are trained or validated on human labels, and a model cannot learn a distinction its labels do not contain. In RLHF the pipeline is strict — supervised fine-tuning, then a reward model trained from human preference labels, then policy optimisation against that reward model — so the reward model's accuracy is bounded by the consistency of the preference labels it was trained on. If annotators agree on 60% of close pairs beyond chance, the reward model is fitting a signal that is 40% noise on those pairs, and the policy optimisation step will then find and exploit whatever spurious regularity survives. That is one route to reward hacking, covered in 11-07. For an LLM judge the mechanism is different but the bound is the same: a judge's measured accuracy is its agreement with human labels, so if the humans disagree with each other, "judge accuracy" has no well-defined upper value to approach. The practical instruction is short: measure human-human agreement first, and report every judge or reward-model number relative to it.

Six things, and a guideline missing any of them will produce drift:

• The task, in one sentence, phrased as the question the annotator answers about each item.
• The label set, with each label defined by an observable condition rather than an adjective.
• A decision procedure for the order in which criteria are checked — for example, check groundedness before helpfulness, because an ungrounded answer's helpfulness is irrelevant.
• Edge cases with worked answers: partially correct, correct but incomplete, correct but unsafe, refusal when refusal was right, refusal when it was not.
• The tie/uncertain option and when to use it, plus what happens to those items downstream.
• Practical rules: how long to spend, whether to consult the source document, whether to use external search (usually no — it changes the task), what to do with items containing personal data, and how to flag content that should not have been shown to a human at all.

Version the guideline the same way you version the evaluation set in 09-01, and record which guideline version produced which labels. A label without a guideline version attached is an orphan measurement.

1. Human evaluation is the only method that can score properties no reference encodes — helpfulness, faithfulness, appropriate refusal, tone.
2. Percent agreement flatters you. In the worked example, 82% observed agreement was κ = 0.46, because chance agreement was already 66.8%.
3. Cohen's kappa is (P_o − P_e) / (1 − P_e). Know it cold, and know that two annotators plus nominal labels is its scope.
4. Use Krippendorff's alpha for more annotators, ordinal scales, or missing labels; weighted kappa for ordinal with two raters.
5. Low agreement is a rubric defect, not an annotator defect, roughly nine times out of ten. Read the disagreements to find the missing category.
6. Adding a named middle category is the highest-yield fix. Splitting binary faithfulness into full/partial/ungrounded took κ from 0.46 to 0.74 in the worked example, on a five-point rise in percent agreement.
7. Diagnose from the annotator confusion matrix. Symmetric off-diagonals mean ambiguity (fix the rubric); asymmetric mean bias (fix calibration).
8. Agreement is the ceiling. No automatic metric, LLM judge, or reward model can be more reliable than the labels used to validate it.
9. Allow a tie option in preference labelling. Forcing a preference on near-identical pairs injects noise straight into the reward model.
10. Blind, shuffle, cap session length, and version the guideline. Every one of these prevents a specific, documented failure.
11. On the exam, treat the printed 3.1–3.5 objectives as defective — the scope statement's phrase "the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)" is the text that puts this lesson on the syllabus.

Human labels are the ground truth and they are also the bottleneck: at two minutes an item, a hundred items is an afternoon and a thousand is a project. The rest of this module is about approximations — automatic metrics that agree with human labels closely enough to run in a build pipeline. The first family worth understanding is the one that fixes the most obvious defect of string-overlap scoring: a correct answer that happens to use different words than the reference.

Next: 09-04 covers BERTScore and embedding-based evaluation metrics — how they score meaning rather than surface form, exactly what they still cannot see, and why they inherit the blind spots of whichever embedding model they are built on.