Error Analysis for LLMs: Turning Failures into a Prioritized Fix List

Error analysis is a structured, repeatable process for reading failing model outputs, assigning each one a specific root-cause category from a small fixed taxonomy, and aggregating those categories into counts that reveal which failure mode is actually dominant — as distinct from which failure mode is most memorable, most recently reported, or most annoying to the person doing the reading. The deliverable is not a score. It is a table: category name, count, percentage of all failures, one or two representative examples, and a proposed fix for the categories worth fixing first.

This sits downstream of every metric in this module and answers the question none of them can answer alone: why. Perplexity (09-02) tells you the model's predictions were surprising to itself; it does not tell you whether that surprise came from a genuinely hard input or a broken tokenizer. BLEU and ROUGE (09-06) tell you the output diverged from a reference; they do not tell you whether that divergence was a wrong fact, a valid paraphrase, or a missing detail. LLM-as-a-judge (09-10) gives you a score and, if prompted well, a rationale — but a rationale on one item is not a pattern across a hundred. Error analysis is the discipline of reading enough of those individual explanations, together, to see the pattern the aggregate score was hiding.

The single sentence version, worth memorizing: a metric tells you that something is wrong; error analysis tells you what is wrong, how often, and therefore what to do about it.

L1 — Intuition: read the failures before you guess the fix

The most common and most expensive mistake in applied LLM work is skipping straight from "the score is disappointing" to "let's try fixing X," where X is whatever the last article the team read happened to recommend — more retrieval chunks, a bigger model, a longer system prompt. Sometimes X is right. Often it is not, because the actual dominant failure mode was something nobody looked at directly: a chunking boundary cutting a fact in half, a prompt that never specified the required output format, or a metric that was penalizing correct answers phrased differently than the reference. The fix for "we guessed" is simple to state and consistently skipped under time pressure: read the failing examples themselves, one at a time, before deciding what to build.

L2 — Mechanism: the five-stage process

Stage 1 — Pull the failures, not a random sample of everything. Start from your evaluation run's results (09-01) and filter to the items that scored below your threshold — failed exact match, low judge score, low faithfulness. Reading passing cases is not useless, but it is not error analysis; the signal you need lives in the failures specifically. If you have too many failures to read all of them, take a random sample of a fixed size (50–100 is a workable starting point) rather than the first N or the most dramatic-looking N, because both of those introduce selection bias into the very analysis meant to correct for bias in intuition.

Stage 2 — Read each failure and write down, in your own words, what actually went wrong. Not "the answer was bad" — that is a restatement of the score, not an analysis. Write the mechanism: "retrieved passage did not contain the requested date," "model stated a number not present in any retrieved passage" (extrinsic hallucination, 09-12), "output was correct but in the wrong format for the downstream parser," "reference answer itself was outdated." This stage is slow by design. It is where the actual information is.

Stage 3 — Build the taxonomy from what you actually saw, not from a template. After reading 20–30 failures, patterns start repeating. Name them as categories, in language specific to your system: "retrieval miss — relevant doc not in top-k," "retrieval hit, generation ignored context," "format violation," "genuine hallucination," "ambiguous ground truth," "metric artifact" (the metric penalized a correct answer). A good taxonomy has somewhere between five and twelve categories — fewer than five usually means categories are too coarse to act on; more than twelve usually means some categories should be merged, because a fix list with fifteen line items is not a fix list, it is a list of everything.

Stage 4 — Tag every failure in the sample against the taxonomy, and count. Each failure gets exactly one primary category (a secondary tag is fine if a failure genuinely has two independent causes, but resist tagging everything as multi-cause — it is usually a sign the taxonomy needs a merge or a split). Produce a frequency table: category, count, percentage of the sample.

Stage 5 — Rank by count times fixability, not by count alone. The category with the most failures is not automatically the first thing to fix. A category with 40% of failures that requires a multi-week model swap and a category with 15% of failures that is a one-line prompt fix are not equally worth prioritizing first — the second may deliver more realized value per hour of engineering time even though its raw count is smaller. This is the step that converts a frequency table into an actual fix list, and it is the step most teams skip, stopping at the table and calling it done.

L3 — Depth: taxonomy design, inter-rater agreement, and re-running the analysis

Where the taxonomy should live in the pipeline, not just in the output. A mature taxonomy separates failures by the stage of the pipeline that owns the fix, because that is what determines who acts on the finding: retrieval-stage failures (wrong or missing context), generation-stage failures (context was right, output was wrong anyway — including hallucination types from 09-12), format/parsing-stage failures (content was right, structure was wrong), and evaluation-stage failures (the model's answer was actually fine; the metric or the reference was wrong). This last category is genuinely common and genuinely under-reported, because it requires admitting the measurement, not the system, was at fault — 07-10's retrieval-versus-generation split for RAG debugging is a direct ancestor of this broader four-way stage split.

Inter-rater agreement matters here too. 09-03 established that human rubric scoring needs an agreement check because two people can read the same guidance differently. The same risk applies to error-analysis tagging: if two people categorize the same 20 failures and agree on the category for only 12 of them, the taxonomy's category boundaries are too fuzzy to trust, and the count in Stage 4 is not a reliable count of anything. A quick agreement spot-check — have a second person independently tag a subset, compare — is cheap insurance against building a fix list on top of noise.

Error analysis is not a one-time event. After the top category is fixed, the failure distribution changes — sometimes a fixed category shrinks and a previously-minor category becomes the new dominant one, a pattern familiar from any bottleneck-removal process. The discipline is to re-run the sampling and re-tagging after each meaningful fix, not to run it once at project kickoff and treat the resulting fix list as permanent. 09-01's frozen evaluation set is exactly what makes this re-run comparable across iterations — a fixed set of items lets you attribute a shrinking category to the fix, rather than to a different, easier sample.

Slicing the analysis, not just aggregating it. A single overall frequency table can still hide a serious problem if it is dominated by an easy majority slice. If your evaluation set spans several task types, user segments, or languages, run the categorization within each slice separately as well as in aggregate — a failure mode affecting 5% of all traffic but 60% of one specific, important slice (a particular customer segment, a particular query type) is easy to miss in an aggregate table and easy to catch the moment you slice by that dimension. This mirrors the general warning from 09-09 that a conclusion drawn from an aggregate sample says nothing reliable about a small subgroup within it — the same caution applies to error categories, not just to statistical significance.

Three related activities get conflated, and the exam-relevant distinction is what each one actually produces.

Error analysis and root-cause analysis are complementary, not competing: error analysis tells you which problem matters most across the whole failure population; root-cause analysis, applied to the category error analysis surfaced as dominant, tells you exactly why that category happens so you can fix the actual mechanism rather than a symptom of it. Skipping error analysis and going straight to root-cause analysis on the first failure you happen to notice risks doing deep, careful work on a problem that turns out to be a small minority of total failures.

Constructed scenario throughout — an illustrative evaluation run, not a measured result from any real system. A RAG-based support assistant scores 68% on a frozen 100-item evaluation set (09-01), each item graded pass/fail by a validated rubric (09-03). That leaves 32 failing items. Following Stage 1, a random sample of those 32 is read in full (here, all 32, since the count is manageable).

Stage 2–3: reading and naming categories. After reading through the 32 failures, six recurring patterns emerge, each written in the team's own language for their system:

1. Retrieval miss — the correct information exists in the corpus but was not among the top-k retrieved chunks.
2. Retrieval hit, context ignored — the correct chunk was retrieved, and the model answered incorrectly anyway, drawing on its own memory instead.
3. Extrinsic hallucination — the model added a specific claim (a fee, a policy detail) not present in any retrieved chunk (09-12).
4. Format violation — the content was substantively correct, but the required structured-output schema was not followed, breaking the downstream parser.
5. Ambiguous ground truth — the reference answer in the evaluation set is itself outdated or debatable, and the model's answer is arguably also acceptable.
6. Judge miscalibration — the automated judge scoring this run mis-scored a correct answer as a fail (a known judge failure mode from 09-10).

Stage 4: tagging and counting.

What the raw pass rate alone would never have told you: the single number "68%" is consistent with dozens of completely different underlying pictures — a generation problem, a retrieval problem, a metric problem, or some mix. Only after this table exists can anyone say anything concrete about what to do next.

Stage 5: ranking by count times fixability.

The actual fix list this produces, in order: (1) investigate and improve retrieval — the largest single lever; (2) tighten grounding instructions and add citation requirements to cut hallucination; (3) enforce output schema with constrained decoding — cheap, fast, and immediately shippable, so it may well be done in parallel with item 1 rather than strictly after it; (4) correct the four ambiguous reference answers in the evaluation set itself; (5) spot-check the judge's two miscalibrated cases against a human rater; (6) only after all of the above, consider a deeper faithfulness intervention for the "context ignored" cases, since it is both the smallest category and the most expensive fix.

What this table demonstrates that the pass rate could not. Two of the six categories — ambiguous ground truth and judge miscalibration — are not model problems at all; they are measurement problems, together accounting for 6 of the 32 failures. Fixing the model to satisfy an ambiguous or wrongly-scored reference would have been effort spent chasing an artifact of the evaluation setup, not a real capability gap. This is a completely ordinary and completely invisible-without-error-analysis finding: a meaningful fraction of "the model's fault" was, on inspection, the evaluation's fault.

The generalizing rule: run the full five-stage process whenever you are about to commit real engineering effort based on the result, and use the lighter, taxonomy-reuse version for ongoing monitoring where the categories are already established and the question is just "has the mix shifted." Never skip straight from a score to a fix for anything above a trivial change — the worked example's judge-miscalibration and ambiguous-ground-truth categories are exactly the kind of finding that only shows up when someone actually reads the failures.

The Experimentation domain's real scope — derived, per the objective-numbering defect documented fully in 09-09 and 09-12 (the printed objectives 3.1–3.5 are a verbatim duplicate of the Data Analysis domain's 2.1–2.5, so the domain's actual coverage of model evaluation and experimentation is read from its own scope statement and suggested-reading list rather than the literal printed text) — explicitly covers interpreting experiments correctly, and error analysis is the concrete activity that operationalizes "interpret" rather than merely "measure." This lesson directly serves objective 4.5 (monitoring the functioning of data collection, experiments, and other software processes — error analysis is the monitoring activity that catches a process degrading in a specific, nameable way rather than a vague aggregate dip) and 4.7 (writing software components under supervision — an engineer who can name the dominant failure category writes a far more targeted fix than one who is guessing). It also reflects the job-role frame's core claim that the associate role is about recognizing correct practice and correct tool choice under supervision: a candidate who can correctly identify, from a described scenario, that the next step is to categorize failures before proposing a fix is demonstrating exactly that judgment.

Question phrasings to expect:

• "A model scores 70% on an evaluation set. What should be the immediate next step before making any changes?" → Perform error analysis on the failing items — categorize the failures by cause — before deciding what to fix.
• "What is the main limitation of a single aggregate evaluation score?" → It indicates that failures are occurring, but not why, or which failure mode dominates — it cannot direct a specific fix.
• "A team categorizes model failures into retrieval errors, generation errors, and format errors, then counts each category. What is this process called?" → Error analysis.
• "After reading failing examples, a team discovers two of six failure categories are actually caused by incorrect reference answers, not the model. What does this indicate?" → The evaluation set or metric itself needs correction — not every apparent model failure is a model failure.
• "Why should error-analysis samples be tagged by more than one person at least occasionally?" → To check inter-rater agreement on the taxonomy; low agreement means the categories are too ambiguous to trust the resulting counts.
• "What should determine which failure category gets fixed first — the count alone, or something else?" → Count weighed against fixability/effort; the largest category is not automatically the highest-priority fix.
• "Why is error analysis usually re-run after a fix is deployed, rather than performed once?" → The failure distribution shifts after each fix; a previously minor category can become dominant once the largest one is addressed.

Distractor families, and why each is wrong:

A confusion matrix (09-05) is a specific tool for classification tasks with a fixed, known set of labels — it tells you which true label got predicted as which other label, and at what rate, and it is exact and exhaustive by construction because every prediction falls into exactly one cell. Error analysis is a broader, more open-ended process applicable to any task, including free-form generation where there is no fixed label set at all — a hallucinated summary, a malformed JSON output, and a retrieval miss are not "labels" in the way a confusion matrix's rows and columns are; they are categories a human reader constructs by looking at what actually happened. A confusion matrix is something you compute directly from predictions and true labels with no reading required. Error analysis requires a person (or a carefully validated automated proxy) to read the actual content of the failure and decide what happened. Where a task genuinely has a fixed label set, a confusion matrix is a useful input to error analysis — a large "predicted refund policy, true label shipping policy" cell tells you where to start reading — but it does not replace the reading itself, because the matrix cannot tell you why those two labels are being confused.

There is no single universal number, but a workable rule of thumb is enough to see each real category repeat at least two or three times, which in practice tends to land in the 30–100 range for most applied projects, and can be much smaller if the total failure count is small to begin with. The goal is not statistical significance in the 09-09 sense — you are not trying to estimate a precise population proportion with a tight confidence interval, you are trying to discover which categories exist and get a rough, directionally trustworthy sense of their relative size. If you stop after 10 examples, you risk missing rarer-but-real categories entirely. If you insist on reading every single failure in a large evaluation before drawing any conclusion, you likely spend disproportionate time for very little additional taxonomic discovery past the first 50–80, because new categories become rare once the dominant patterns have surfaced. A practical middle path: read until you go 10–15 examples in a row without discovering a new category, then stop discovering and start just tagging and counting the rest of your sample against the taxonomy you have.

Whoever can correctly judge whether a given output is actually right or wrong for this specific task — which is very often a domain expert or the engineer who built the system, not a generic annotator, and not, as a sole method, the model under evaluation itself. A model used to categorize its own failures (an LLM-as-judge pass over the failure set) can genuinely speed up the tagging step at scale, and is a reasonable way to get an initial pass over a large failure set — but it inherits every bias 09-10 named for judges in general, and a model may be systematically blind to exactly the kind of failure it itself is prone to, since it is applying the same generative habits to the categorization task that produced the failures in the first place. The practical pattern that works well: use a model-assisted first pass to propose categories and tags at scale, then have a human — ideally someone with real domain knowledge of the task, not just familiarity with LLMs generally — spot-check a meaningful subset for agreement, exactly as 09-03's inter-annotator-agreement discipline recommends. Treat full automation of error analysis with the same skepticism this whole module has applied to every other automated shortcut: useful for scale, never sufficient on its own for a conclusion you are about to act on.

• Error analysis — the process of reading, categorizing, and counting a sample of failing model outputs by root cause, producing a ranked, actionable fix list rather than a score.
• Failure taxonomy — the fixed, small set of named categories (typically 5–12) that every failure in a sample gets tagged against.
• Root-cause analysis — the deeper investigation, applied to one specific failure category, of exactly why that category occurs mechanistically.
• Retrieval-stage failure — a failure attributable to the retrieval step in a RAG pipeline (missing or irrelevant context), as distinct from generation, format, or evaluation-stage failures.
• Evaluation-stage / measurement-stage failure — a case where the apparent failure is caused by an ambiguous reference answer or a miscalibrated metric or judge, not by the system under test.
• Fix priority (count × fixability) — ranking failure categories for remediation by combining how often they occur with how cheaply they can be addressed, rather than by raw frequency alone.
• Inter-rater agreement (in tagging) — the extent to which independent taggers assign the same category to the same failure; low agreement signals an ambiguous taxonomy.
• Slicing — running a categorization separately within meaningful subgroups (task type, segment, language) rather than only in aggregate, to avoid diluting a concentrated problem.
• Paired error analysis — reading the same failing items across two candidate systems side by side, to see whether they fail for the same or different reasons.

1. A score tells you that something failed; error analysis tells you why, how often, and what to do about it. Every metric in this module produces the first; only error analysis produces the second and third.
2. Read the failures directly — do not guess the fix from the score alone. This single habit is the difference between targeted engineering and wasted effort.
3. Build the taxonomy from what you actually read, not a generic template, and keep it to roughly 5–12 categories — enough to be specific, few enough to be actionable.
4. Rank by count times fixability, not count alone. A cheap fix for a mid-sized category can outperform an expensive fix for the largest one.
5. Not every failure is the model's fault. Ambiguous ground truth and judge/metric miscalibration are real, common categories that point at the measurement, not the system.
6. Check inter-rater agreement on your tags, the same discipline 09-03 applies to human rubric scoring — an ambiguous taxonomy produces an untrustworthy count.
7. Re-run the analysis after each fix. The failure distribution shifts, and the next-most-important category is only visible once the current top one is addressed.
8. Slice the analysis, not just the aggregate. A serious problem concentrated in a small but important segment can hide inside an unremarkable overall rate.
9. Error analysis and root-cause analysis are complementary, applied in order. Establish which category dominates first; go deep on mechanism only after that.
10. Model-assisted tagging speeds up scale but is not sufficient alone. Spot-check with a human, ideally a domain expert, before committing to the resulting fix list.

Module 9 has been about measuring and diagnosing your own system on your own evaluation set — the honest, labor-intensive way. There is a faster-looking shortcut that every team eventually reaches for: borrowing a published, standardized benchmark instead of building and maintaining one. That shortcut carries its own, entirely different failure mode, one that has nothing to do with retrieval, hallucination, or judge bias, and everything to do with whether the test you are running was ever a fair test in the first place.

Next: 10-01 covers public benchmarks like GLUE and MMLU and the problem of data contamination — how benchmark data leaks into training corpora, why a model can score well on a benchmark for reasons that have nothing to do with the capability the benchmark claims to measure, and how to tell whether a benchmark result is actually trustworthy before you cite it.