Explainability and Testing Data/Model Quality and Consistency

Identity statement: explainability, in this domain's framing, means using tools that reveal what evidence a model relied on to produce a given output — most commonly attention maps, which show which image regions or tokens drove an output, or textual rationales, which have a model explain its own reasoning in words.

When it matters: any scenario asking how to validate that a multimodal model is relying on the right evidence, or asking what an attention map does and does not prove.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) frames this directly: "multimodal models can improve interpretability — e.g., attention maps showing which image regions/tokens drove an output, or generating a textual rationale. This helps validate that a model relies on the right evidence." Two words in that sentence set the honest boundary this lesson works inside: "helps validate," not "proves." Explainability tools are evidence toward an answer, not a certificate of correctness.

What an attention map actually is, and what it is not

An attention map is a visualization of where a model's attention weight — the internal signal that governs how much influence different parts of the input have on a given output — was concentrated when producing that output. If a model captioning an image produces the word "dog" and its attention map lights up over the region of the image containing a dog, that is a good sign: the output and the evidence the model leaned on line up sensibly. If the same model produces "dog" while its attention map lights up over an unrelated corner of the image, that is a red flag worth investigating, because the model reached a plausible-sounding conclusion by relying on something that, at least visibly, has nothing to do with it.

M2-04, earlier in this course, already established the caution this domain repeats here from a different angle: an attention map is suggestive, not a guaranteed causal explanation of the output. High attention on a region correlates with that region's influence on the output, but attention weight is a mechanism internal to the model's computation, not a verified causal audit trail — a model can attend heavily to a region and still, through some interaction elsewhere in the network, produce an output that a naive reading of the attention map alone would not fully explain.

⭐ THE EARNED INSIGHT An attention map answers "where did the model look," not "why did the model decide." Those two questions feel like the same question, and treating them as identical is the single easiest way to over-trust an explainability tool. A model can look at the right place and still reach the wrong conclusion for reasons the attention map itself never reveals — which is exactly why explainability is one input into validating a model's reasoning, never the whole validation on its own.

Textual rationales: a model explaining itself, and the same caution applied to that

A textual rationale is a model-generated explanation, in natural language, of why it produced a given output — "I identified this as a golden retriever because of its coat color and ear shape." The same caution from the attention-map case applies here in a sharper form: a language-generation component producing a plausible-sounding rationale is not the same as that rationale being the model's actual internal reasoning process. A model can generate a fluent, convincing-sounding rationale that does not accurately describe the mechanism that actually produced its output — the rationale-generation step is itself a generation task, subject to the same fluency-without-correctness risk that governs any other generated text.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names test data quality and consistency as checking for "missing/mislabeled entries, cross-modal misalignment, duplicates, and distribution shift before trusting results." Four distinct checks, each catching a different failure mode a dataset can carry silently into a result you would otherwise trust.

Missing or mislabeled entries. A missing entry is an absent value; a mislabeled entry is a present value that is simply wrong — a caption that describes a different image than the one it is attached to, a class label assigned incorrectly. M3-02 already covered the missing-modality-entry case in depth for multimodal data specifically; mislabeling is the sibling failure where the data is present but incorrect rather than absent.

Cross-modal misalignment. This is M3-02's alignment material by another name, applied here as a testing checklist item rather than a management practice: verify, don't just assume, that a caption genuinely corresponds to its paired image, or a transcript to its paired audio, before trusting any result computed from the pairing.

Duplicates. A duplicated example inflates its own influence on any aggregate statistic computed over the dataset, and — more seriously for evaluation specifically — a duplicate that appears in both a training set and an evaluation set produces a result that looks like generalization but is actually partial memorization, the same integrity failure M3-01 named when it flagged evaluating on the training set as a trap.

Distribution shift. A dataset's underlying distribution can change between when a model was trained and when it is being evaluated, or between different sources feeding the same pipeline — the same category of concept the source material's "confounding factors" material covers, applied specifically to whether a dataset still resembles the distribution a model was validated against.

Why these four checks are listed together rather than folded into one general "clean the data" instruction

Each of the four failure modes is invisible to the other three's detection method. A missing-value scan will not catch cross-modal misalignment, because both files can be individually complete. A duplicate-detection pass will not catch distribution shift, because duplicates are a within-dataset property and shift is a between-datasets-or-between-times property. Treating "test data quality" as one vague instruction rather than four specific, separately-checkable items is exactly how a real pipeline ends up passing a general "looks fine" review while still carrying one of these four failures undetected.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) frames testing model accuracy and effectiveness as evaluating "on held-out data with task-appropriate metrics, probe edge cases, and compare against a baseline." This sentence packs together three separate obligations that a single overall accuracy number, computed once, satisfies none of on its own.

Task-appropriate metrics, evaluated on held-out data. This is M3-01's fixed-evaluation-set discipline and M3-08's task-to-metric matching, applied together: the right metric, computed on data the model did not train on.

Probing edge cases. A single aggregate score computed over an entire held-out set can look strong while a model fails badly on a specific, important subgroup or edge case that the aggregate simply averages away — the same subgroup-masking risk M2-06's confounding-factors material and this course's later Trustworthy AI module both name from their own angles. Testing model effectiveness means deliberately looking at performance on edge cases and subgroups, not only at the one number the whole set produces.

Comparing against a baseline. A model's raw score means little in isolation; it means something once compared against a fixed reference point, exactly the baseline discipline M3-01 opened this module with. Stating that a model scores 82% on a task answers a distinctly less useful question than stating that it scores 82% against a baseline's 74% on the identical evaluation set, since only the second framing tells you whether anything actually improved.

[GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names this domain's own single most common evaluation shortcut directly: "judging a RAG system only on the final answer, ignoring whether retrieval was relevant and the answer was faithful to sources." M3-08's evaluation-metrics survey already established why RAG needs retrieval quality and faithfulness tracked as two separate metrics; this section names the failure mode that happens when a team skips that separation and judges only the finished output.

The shortcut is tempting precisely because it is the easiest thing to measure: read the final answer, decide whether it is correct, done. What that shortcut cannot do is tell you why a wrong answer was wrong. A RAG system's final answer can fail for two structurally different reasons — the retrieval stage fetched the wrong or incomplete material, or the generation stage drifted beyond what perfectly good retrieved material actually supported — and a team that measures only the final answer has no way to localize which of the two happened, which means no way to know which stage to actually fix.

Why this specific shortcut is worth calling out as the domain's most common trap

Every other data-quality and model-testing check in this lesson is a checklist item you can forget to run. The RAG shortcut is different: it is not a check that gets skipped by oversight, it is a check that gets replaced by an easier-looking one that appears to answer the same question. "Is the final answer correct" sounds like it should be sufficient — it is, after all, the thing the user actually experiences — but it silently substitutes a simpler, insufficient question for the two-part question RAG evaluation actually requires. That substitution is what makes it a shortcut worth naming specifically, rather than just another item on a checklist.

A support-bot RAG system answers a policy question incorrectly. Trace the diagnosis with, and without, the shortcut this lesson names.

User question: "What is the refund window for a damaged item?"
System's answer: "You have 60 days from the delivery date to request
                   a refund for a damaged item."
Ground truth: the actual policy window is 30 days.

WITH the shortcut (final-answer-only judgment):
  Verdict: "Wrong answer." Full stop.
  -> No information about which stage caused the error.
  -> No actionable next step beyond "something is broken somewhere."

WITHOUT the shortcut (retrieval quality + faithfulness, checked
                       separately):
  Step 1 - Retrieval quality check: did the system retrieve the
           passage that actually states the refund policy?
           Result: YES — the correct policy passage, stating "30 days,"
           was retrieved among the top results.
  Step 2 - Faithfulness check: does the generated answer match what
           the retrieved passage actually says?
           Result: NO — the passage says 30 days; the generated
           answer says 60 days. The generation stage introduced a
           number the retrieved source never stated.
  -> Diagnosis: a faithfulness failure, not a retrieval failure.
  -> Actionable next step: investigate the generation stage
     specifically — retrieval does not need attention here.

This is a constructed scenario — the specific policy numbers are invented for the walkthrough, not a measured result from any deployed system — but the diagnostic gap it demonstrates is exactly the one [GROUND TRUTH] (Sources/nca-genm/domain-3-experimentation.md) names: final-answer-only judgment correctly flags that something is wrong, and then stops, leaving the team to guess at which of two structurally different problems actually caused it. Checking retrieval quality and faithfulness separately turns "something is wrong" into "the generation stage specifically hallucinated a number the source material never stated" — a diagnosis specific enough to act on.

A team has trained a multimodal classifier and wants to report its results with confidence. Walk through applying every check this lesson has named, in order, before trusting the reported number.

Step 1 - Missing/mislabeled entries: scan the evaluation set for
         blank fields and spot-check a sample of labels against the
         actual content.
  Found: 12 examples with blank labels (excluded from evaluation);
         3 examples with labels that, on inspection, do not match
         the content (corrected before evaluation).

Step 2 - Cross-modal misalignment: verify a sample of caption-image
         pairs against a trustworthy shared key, per M3-02's method.
  Found: alignment holds; no drift detected in this sample.

Step 3 - Duplicates: check for the same example appearing in both
         the training set and the evaluation set.
  Found: 8 duplicated examples present in both splits; removed from
         the evaluation set before scoring.

Step 4 - Distribution shift: compare the evaluation set's class
         balance and general characteristics against the training
         set's.
  Found: no material shift detected between the two.

Step 5 - Task-appropriate metric, fixed evaluation set, baseline
         comparison: score the cleaned evaluation set with the
         correct classification metrics (precision/recall/F1, per
         M3-08), against a recorded baseline.
  Result: F1 improved from a 0.71 baseline to 0.79 on the cleaned set.

Step 6 - Edge-case probing: check performance specifically on the
         smallest and historically hardest subgroup in the dataset,
         not just the aggregate F1.
  Result: subgroup F1 is 0.61 — meaningfully below the aggregate,
          flagging a specific weakness the aggregate score alone
          would have hidden.

This is a constructed scenario — the specific counts and scores are invented for the walkthrough — but the sequence itself is the real, testable discipline this lesson is built around: every one of the six steps catches something the others would not have, and the final aggregate F1 of 0.79 would have been reported with false confidence had steps 1 through 4 not first removed the duplicated and mislabeled examples inflating it, and had step 6 not surfaced the subgroup weakness the aggregate number was quietly averaging away.

Experimentation is the largest domain on the NCA-GENM blueprint at 25% of the exam, and this lesson closes out the domain's own framing of what "interpreting an experiment" actually requires beyond running the numbers: validating that a model relies on the right evidence, and validating that the data and the model itself can be trusted before any metric computed from them means anything. This is the domain's own final layer of discipline, sitting on top of every metric and every generative mechanism the rest of the module covered.

The question tends to arrive in a small number of recognizable shapes.

1. Explainability-scope items. A scenario describes an attention map or rationale and asks what it does or does not prove. The keyed answer treats it as suggestive validation, not a guaranteed causal explanation.
2. Data-quality-check identification. A scenario describes one specific failure (a duplicate in both splits, a caption-image mismatch, a shifted distribution) and asks which named check would catch it.
3. Model-testing completeness items. A scenario reports a single aggregate accuracy number and asks what is missing before the result can be trusted — the keyed answer names edge-case probing and baseline comparison as the missing pieces.
4. The RAG-shortcut item. A scenario judges a RAG system by its final answer alone and asks what is wrong with that evaluation approach — the keyed answer names the missing retrieval-quality and faithfulness checks directly.

What the distractors typically look like

Expect an attention map or textual rationale offered as definitive proof of a model's reasoning, when the correct framing treats either as suggestive evidence requiring further validation. Expect one data-quality check named as sufficient on its own — "we checked for missing values, so the data is clean" — when the domain names four separate checks, each catching something the others miss. And expect a RAG evaluation described as complete because the final answer was checked for correctness, when the keyed answer insists on retrieval quality and faithfulness as the two additional, necessary checks.

An attention map shows where a model's attention weight was concentrated when it produced a given output — which image regions or which tokens received the most internal weight. It does not prove that those regions or tokens are what causally determined the output, because attention is a signal internal to the model's computation, not a verified account of why the model reached its conclusion. Treat an attention map as one useful piece of evidence toward validating a model's reasoning, not as a standalone proof that the reasoning was correct.

Because it looks sufficient — the final answer is, after all, what the user actually experiences — while silently failing to distinguish between two structurally different causes of a wrong answer: a retrieval failure, where the system fetched the wrong or incomplete material, and a faithfulness failure, where the system generated content beyond what perfectly good retrieved material actually supported. Judging only the final answer tells you a result was wrong without ever telling you why, which leaves a team unable to target a fix at the stage that actually needs one.

1. A model's attention map for an image-captioning output lights up over the correct object in the image. What can you conclude?
   • A. The caption is definitely correct.
   • B. The model's internal reasoning has been fully verified.
   • C. The evidence is suggestive and consistent with correct reasoning, but not a guaranteed causal proof.
   • D. No further validation is needed.
2. Which of the following is NOT one of the four named data-quality checks this lesson covers?
   • A. Missing or mislabeled entries.
   • B. Cross-modal misalignment.
   • C. Learning rate scheduling.
   • D. Distribution shift.
3. A duplicated example appears in both a training set and its paired evaluation set. What is the risk?
   • A. None, since duplicates only affect training.
   • B. The evaluation score reflects partial memorization rather than genuine generalization.
   • C. The model's attention maps become unreliable.
   • D. The duplicate will be automatically excluded by any evaluation script.
4. A model reports 88% aggregate accuracy on a held-out set. What is missing before this result can be fully trusted?
   • A. Nothing — an aggregate held-out score is sufficient on its own.
   • B. Edge-case/subgroup probing and a comparison against a fixed baseline.
   • C. A larger training set.
   • D. A different random seed.
5. A RAG system's final answer is wrong. Judging only the final answer, what can you conclude about the cause?
   • A. It must be a retrieval failure.
   • B. It must be a faithfulness failure.
   • C. Nothing specific — the final answer alone cannot distinguish a retrieval failure from a faithfulness failure.
   • D. The evaluation set must be too small.
6. Why is a textual rationale generated by a model not automatically trustworthy as an explanation?
   • A. Rationales are always shorter than the actual reasoning process.
   • B. Generating a rationale is itself a generation task, subject to producing fluent but inaccurate text.
   • C. Rationales are only available for image tasks, not text tasks.
   • D. Rationales require a discriminator to verify them.

Answers

1. C. An attention map lighting up over the correct region is a good, suggestive sign consistent with correct reasoning, but attention weight is an internal computational signal, not a verified causal account — it supports further validation, it does not complete it.
2. C. The four named checks are missing/mislabeled entries, cross-modal misalignment, duplicates, and distribution shift; learning rate scheduling is a training-time hyperparameter concern, unrelated to this lesson's data-quality checklist.
3. B. A duplicate crossing the train/evaluation boundary means the model may have already seen that exact example during training, so a strong score on it reflects memorization rather than genuine generalization to new data.
4. B. A single aggregate score can mask poor performance on a specific subgroup or edge case, and a raw score without a baseline comparison does not establish whether anything actually improved; both edge-case probing and baseline comparison are needed alongside the aggregate number.
5. C. This is precisely the diagnostic gap the final-answer-only shortcut creates: a wrong answer could stem from bad retrieval or from unfaithful generation on top of good retrieval, and judging the final answer alone provides no way to distinguish which.
6. B. A model producing a rationale is performing a text-generation task, which carries the same risk as any other generated text — fluency and plausibility do not guarantee the rationale accurately describes the model's actual internal mechanism.

• Attention maps and textual rationales help validate a model's reasoning; they do not prove it — both are suggestive evidence, not a guaranteed causal account.
• Testing data quality means four separate checks: missing/mislabeled entries, cross-modal misalignment, duplicates, and distribution shift — each catches a failure the others miss.
• Testing model accuracy requires more than one aggregate held-out score: task-appropriate metrics, edge-case probing, and a fixed baseline comparison, together.
• Judging a RAG system on its final answer alone is this domain's named most common shortcut — it cannot distinguish a retrieval failure from a faithfulness failure, which means it cannot tell a team what to actually fix.
• This lesson's discipline sits on top of every generative mechanism and every metric this module has covered: a sound experiment, the right metric, and validated data and reasoning are three separate, all-necessary conditions for trusting a result.

This closes Module M3, Experimentation — the largest domain on the NCA-GENM blueprint at 25% of the exam, covering how to design a fair experiment, generate a multimodal output with diffusion, GANs, and Riva, evaluate it with the right task-specific metric, and validate the data and reasoning behind the result. Next: Module M4, Multimodal Data, picks up the modality-handling and fusion material this module's diffusion and CLIP-adjacent lessons have already gestured toward — how each modality becomes numbers a model can use, when to fuse early versus late, and what to do when a modality is simply missing.