Exploratory Data Analysis Before Fine-Tuning: The Five-Step Checklist

Identity statement: exploratory data analysis before fine-tuning is a structured, five-part inspection of a dataset's distributions, lengths, vocabulary, labels, and quality, performed deliberately before any training run begins, so that defects surface as a cheap diagnostic finding rather than an expensive, hard-to-trace training failure.

The reason timing matters as much as content here is what a fine-tuning failure actually looks like without EDA. A model fine-tuned on a dataset with severe class imbalance does not throw an error — it trains successfully, reports a loss curve that looks reasonable, and produces a checkpoint. The imbalance shows up later, as a model that performs conspicuously worse on the minority class in evaluation, or worse, as a model that looks fine on evaluation but fails in a specific, hard-to-reproduce way in production because the evaluation set shared the same imbalance the training set did. By the time that failure surfaces, days or weeks of GPU-hours have already been spent, and tracing the failure back to its actual root — a label distribution nobody checked before training started — is far more expensive than the five checks that would have caught it up front.

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names the trap this material exists to prevent directly: "Skipping EDA and jumping straight to fine-tuning hides imbalance, leakage, and truncation problems that quietly wreck results." Every word in that sentence is doing specific work — "hides" rather than "risks," because these problems do not announce themselves; "quietly," because the training run itself gives no warning sign; and "wreck results" rather than "slightly degrade results," because each of the five failure modes this lesson covers is capable of a severe, not marginal, impact on a fine-tuned model's real performance.

L1 — Intuition

Before anything else, look at how the dataset's classes or categories are actually represented. A dataset that looks reasonably sized in aggregate can still be badly skewed at the class level — most of its bulk concentrated in one or two categories, with everything else thinly represented.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names this step as "class balance" specifically. In practice, distribution analysis means tabulating how many examples fall into each class or category the fine-tuning task cares about, and looking at that tabulation as a distribution rather than a single aggregate count. A dataset with 100,000 examples sounds ample until distribution analysis reveals that 95,000 of them belong to one class and the remaining five classes share 5,000 examples between them — a fact the raw count alone never surfaces.

L3 — The exam-relevant edge case: distribution analysis is a precondition, not a fix

Distribution analysis's job is detection, not correction — running this check tells you the shape of the imbalance, but fixing it is a separate step (oversampling, augmentation, or reweighting, from M3-01's material). The exam-relevant point worth holding is that this check has to run before any of those fixes are applied, because you cannot correct a distribution you have not measured, and a team that jumps straight to "let's oversample the minority class" without first quantifying how minority that class actually is risks either under-correcting or over-correcting the imbalance relative to its real severity.

L1 — Intuition

A model has a fixed context window — a maximum number of tokens it can process in a single input. Any training example longer than that window does not fail loudly; it gets silently truncated, and the model trains on a partial version of that example with no indication anything was cut off.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names this step as "length statistics — truncation risk vs the model's context window," and the phrase "vs the model's context window" is load-bearing: this check is not asking "are the examples long" in the abstract, it is asking "how many of these examples exceed the specific context window of the specific model being fine-tuned." A length distribution — the count of examples falling into short, medium, and long length buckets, measured in the model's own tokens rather than characters or words — reveals what fraction of the dataset is actually at truncation risk for the model in question, which is a fact that changes depending on which model you are fine-tuning, not a fixed property of the dataset alone.

Truncation is a genuinely silent failure mode. A training pipeline that truncates an over-length example to fit the context window does not raise an error or a warning by default in most frameworks — it simply drops the tokens past the window and proceeds, and the model trains on whatever fragment survived. If the truncated portion happened to contain the label-relevant content — the actual answer in a long question-answer pair, or the conclusion of a long document being summarized — the model is effectively being trained on an example whose supervision signal has been silently deleted, and nothing in the training logs will flag that this happened.

L3 — The exam-relevant edge case: truncation risk is model-specific, and the check has to be run against the deployment target

The detail that separates a shallow read of this check from a correct one: length statistics computed once and assumed to generalize across different models is a mistake, because "how many examples exceed the context window" is a function of the length distribution and the specific context window being fine-tuned against. A dataset that was safely within a 32,000-token context window's truncation risk becomes a meaningfully different risk profile if the same dataset is later used to fine-tune a model with an 8,000-token window instead. Length statistics have to be recomputed, or at minimum re-checked, against whatever specific model's context window the fine-tuning run actually targets.

L1 — Intuition

A dataset's vocabulary — the actual distinct words, terms, and tokens that appear in it — carries information that neither the length statistics nor the label distribution surfaces on their own. Vocabulary analysis looks specifically at what terms appear, how often, and whether the rare end of that distribution is meaningful domain content or noise.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names this step as "vocabulary analysis — rare/noisy terms," which connects directly back to M3-03's tokenization material: a corpus dominated by noise (malformed text fragments, encoding artifacts, boilerplate remnants that survived deduplication) inflates the apparent vocabulary size with entries that carry no real signal, while a corpus with genuinely rare but meaningful domain terms — the specialized jargon M3-03 discussed — needs those terms recognized as valuable rather than mistaken for noise and filtered out. Vocabulary analysis is the check that distinguishes those two cases: tabulating term frequency across the dataset and inspecting the rare tail specifically, rather than assuming every rare term is either automatically noise or automatically signal without actually looking.

L3 — The exam-relevant edge case: this check catches what cleaning missed, not what cleaning already fixed

Vocabulary analysis at the EDA stage is not redundant with M3-01's cleaning pass, even though both touch text quality — cleaning is a corrective step applied once, upstream, while vocabulary analysis at the EDA stage is a verification step that checks whether the cleaning actually succeeded on this specific dataset, immediately before it is used. A cleaning pipeline that worked well on 99% of a corpus can still leave a residue of noisy terms in the remaining 1%, and vocabulary analysis run right before fine-tuning is the check that catches that residue rather than assuming an earlier, generic cleaning pass caught everything.

L1 — Intuition

Label distribution analysis looks specifically at the target labels a supervised fine-tuning task is training toward, as distinct from the general class-balance check in step 1, which can apply to any categorical property of the data, labeled or not.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names this step as "label distribution — imbalance detection," and the reason this is listed as a separate step from step 1's distribution analysis, rather than folded into it, is that a dataset can have a perfectly reasonable distribution across some general property (source, length, topic) while still carrying severe imbalance specifically in the label the model is being trained to predict. A sentiment classification dataset might be well-balanced across document source and length while still being 90% positive-sentiment examples and 10% negative — a fact that step 1's general distribution check, if it were checking the wrong property, could miss entirely, and that only a check aimed specifically at the label itself reliably catches.

L3 — The exam-relevant edge case: label imbalance and general-distribution imbalance are independent findings

The trap worth naming: assuming that checking a dataset's general distribution (by source, by topic, by length) is equivalent to checking its label distribution. They are independent findings, and a dataset can pass one check while failing the other. The five-step checklist lists them separately precisely because a single combined "distribution looks fine" conclusion, drawn from checking the wrong property, would miss the specific imbalance that actually matters for a supervised fine-tuning objective — the label the loss function is computed against.

L1 — Intuition

The final check is a direct data-integrity pass: are there duplicate records in this specific dataset, are any labels wrong, and are any fields null or missing that the training pipeline expects to be populated.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names this step as "quality assessment — duplicates, mislabels, nulls." This is, in a sense, a second-pass verification of M3-01's cleaning and curation discipline, run specifically on the dataset about to be used for this particular fine-tuning job rather than assumed to have been handled once, generically, upstream. Duplicates surviving into a fine-tuning dataset carry the same over-weighting risk M3-01 described for pretraining corpora, at a smaller scale but with a comparable mechanism: a fine-tuning example repeated several times gets repeated gradient-update weight, and can produce a model that overfits specifically to that repeated example's phrasing. Mislabels — examples where the target label is simply wrong, whether from an annotation error or a pipeline bug — are a direct source of noisy training signal that a model has no way to distinguish from correct supervision; it will dutifully try to learn the wrong association if enough mislabeled examples share a pattern. Nulls — missing fields the training pipeline expects — are the same schema-integrity concern M3-02's formatting material covered, checked here one final time on the specific dataset actually about to be used.

L3 — The exam-relevant edge case: quality assessment is where the earlier steps' blind spots concentrate

Quality assessment is deliberately listed last, and it functions as a catch-all check for defects that the other four steps, each aimed at a specific property, might not surface on their own. A duplicate record does not necessarily disturb the class distribution (step 1) if the duplicated example happens to belong to the majority class already, does not necessarily create a length-statistics anomaly (step 2) if it is an ordinary-length example, and does not necessarily stand out in vocabulary analysis (step 3) if its content is unremarkable — it can pass all three other checks cleanly and still be a genuine defect that only a direct duplicate/mislabel/null scan will catch.

Reading this table as a set rather than five isolated rows is the point: no single step subsumes any other, and a dataset can pass four of the five checks cleanly while failing the fifth in a way that still wrecks a fine-tuning run.

⭐ THE EARNED INSIGHT: > None of the five EDA steps is redundant with any of the others, because each one is aimed at a genuinely different axis of a dataset's structure — what categories exist, how long examples run relative to a specific model's window, what vocabulary appears, what the supervised label distribution looks like, and whether individual records are duplicated, mislabeled, or incomplete. A dataset can pass any four of these checks cleanly and still fail catastrophically on the fifth, which is exactly why the checklist has five named items rather than one combined "does this data look okay" judgment call — a single holistic impression is not a substitute for five independent, specifically targeted checks.

Treat the following as a constructed scenario built to make the diagnostic reasoning legible, not a report of any real incident. A team fine-tunes a model on a 40,000-example customer-support ticket classification dataset (five categories: billing, technical, shipping, account, other) without running EDA first. Fine-tuning completes with a normal-looking loss curve. Evaluation results:

Overall accuracy:            91.2%
Per-class recall:
  billing:      96.8%
  technical:    94.1%
  shipping:     93.5%
  account:      89.7%
  other:         8.3%   <-- conspicuously low

The "other" category's near-collapse is the symptom. Running the five-step checklist retroactively on this dataset reveals the root cause:

Step 1 (distribution analysis):
  billing:    14,200 examples (35.5%)
  technical:  12,800 examples (32.0%)
  shipping:    8,900 examples (22.3%)
  account:     3,600 examples ( 9.0%)
  other:         500 examples ( 1.2%)   <-- severe minority class

Step 4 (label distribution): confirms the same finding at the label level,
  since "category" is this dataset's actual training label — steps 1 and 4
  point at the identical defect here because the general distributional
  property being checked and the supervised label happen to be the same field.

The other four steps, run on this same dataset, would have come back largely clean: length statistics show no unusual truncation risk (support tickets are short relative to the model's context window), vocabulary analysis shows no unusual noise, and quality assessment finds no meaningful duplicate or null-field problem. The failure is isolated entirely to steps 1 and 4 — the "other" category was so thinly represented (1.2% of the dataset, 500 examples against 39,500 for the other four classes combined) that the fine-tuned model essentially never learned to recognize it, defaulting to whichever of the four majority classes a borderline "other" example most superficially resembled.

If EDA had run before fine-tuning:
  Step 1/4 would have flagged the 1.2% minority class immediately
  -> correction options per M3-01: oversample "other," augment it
     synthetically, or reweight the loss to counteract the imbalance
  -> a five-minute distributional check, run before training,
     versus a full fine-tuning run plus a post-hoc failure diagnosis

This is a constructed scenario — the specific percentages are illustrative, chosen to make the diagnostic reasoning clear rather than measured from a real dataset — but the mechanism it demonstrates is exactly the one [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names as this domain's central trap: the problem was entirely visible in the data before training started, and skipping the check that would have surfaced it let the defect quietly wreck one category's results while the aggregate accuracy number looked reassuring.

Consider a second constructed scenario, isolating a different one of the five checks. A team fine-tunes a model with an 8,000-token context window on a long-document summarization dataset of 20,000 document/summary pairs, again without running EDA first.

Reported evaluation ROUGE score: 0.41 (reasonable-looking)

The aggregate score looks acceptable, and without EDA, nothing in the training or evaluation pipeline flags a problem. Running length statistics retroactively:

Document length distribution (in the model's own tokens):
  Under 4,000 tokens:    11,200 documents (56.0%)
  4,000-8,000 tokens:     5,300 documents (26.5%)
  Over 8,000 tokens:      3,500 documents (17.5%)  <-- exceeds the context window

17.5% of the dataset — 3,500 documents — exceeds the model's 8,000-token context window and gets silently truncated during training. For a summarization task specifically, the truncated tail of a long document frequently contains exactly the concluding content a good summary needs to reference, so a meaningful fraction of this dataset's training examples were, in effect, teaching the model to summarize based on an incomplete document while still being paired with a summary written against the full, untruncated original.

If length statistics had run before fine-tuning:
  17.5% truncation-risk rate would have been visible immediately
  -> options: filter documents over the window, use a model with a
     larger context window, or chunk long documents and adjust the
     summarization framing accordingly
  -> the aggregate ROUGE score of 0.41 never reveals this on its own,
     because it is an average across a mix of fully-intact and
     silently-truncated examples, diluting the truncated subset's
     specific degradation into a still-plausible-looking overall number

This is again a constructed illustration, not a measured result — but it demonstrates precisely why length statistics has to be checked specifically against the deployment model's context window before fine-tuning, and why an aggregate evaluation metric computed after the fact is a poor substitute for the five-minute check that would have caught the same defect before any training compute was spent on it.

Objective 3.5 covers EDA before fine-tuning directly, closing out Data Preparation's five objectives. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) frames EDA as "an explicit best practice," which signals that this material is tested as a procedural discipline — a checklist to apply — rather than as a single conceptual fact to recall, and the module's own "deepest" flag on this lesson reflects that professional-level exam items here tend to be scenario-based diagnoses rather than bare definition recall.

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) frames EDA explicitly as a before-fine-tuning practice, and that timing framing is itself a recurring exam angle: a "why does this matter" item testing whether EDA is understood as a pre-training discipline rather than a post-hoc diagnostic exercise run only once a result already disappoints.

Beyond that timing angle, expect this material in a few other recurring shapes: a scenario describing a fine-tuning failure with a specific symptom (one category collapsing, unexpectedly poor performance on long inputs, an unexplained accuracy gap) and asking which EDA step, run beforehand, would have caught it — testing whether the five steps' distinct scopes from section 7's table are understood well enough to map a symptom back to its corresponding check; and a direct-recall item asking which of the five named steps addresses a specific concern (truncation risk maps to length statistics; duplicates map to quality assessment).

What the distractors typically look like

The standing traps in this material: presenting EDA as optional or as something to run only after a training result disappoints, when the source material's own framing is that it is a best practice performed beforehand; collapsing the five distinct steps into a single vague "check the data quality" step, losing the specific-symptom-to-specific-check mapping the exam actually tests; and confusing step 1's general distribution analysis with step 4's label-distribution check, when a scenario's actual defect lives specifically in the supervised label rather than in some other categorical property of the data.

There is no single step that substitutes for the other four, and the checklist's own structure argues against picking just one — each step catches a genuinely different failure mode, and a dataset that fails only on, say, quality assessment (duplicates and mislabels) would sail through distribution analysis, length statistics, vocabulary analysis, and label distribution cleanly, meaning skipping quality assessment specifically because the other four looked fine would still let that defect through untouched. If time is genuinely constrained, the most defensible priority order follows expected severity of impact — label distribution and quality assessment tend to have the most direct effect on a supervised fine-tuning outcome — but "run one check instead of five" is not the discipline the source material describes, and the exam's own framing treats this as a five-item checklist rather than a menu to pick from.

No — they are complementary rather than redundant, and the distinction is about scope and timing. M3-01's cleaning and curation is a corrective pass applied generically to a corpus, often far upstream of any specific fine-tuning job, addressing missing values, deduplication, and imbalance at the level of the broader dataset or pretraining corpus. EDA before fine-tuning is a verification pass applied specifically to whatever exact dataset is about to be used for a specific fine-tuning run, immediately before that run begins, checking whether the earlier cleaning actually succeeded on this particular slice of data and catching anything that slipped through. A dataset can have been cleaned thoroughly at the corpus level and still fail an EDA check on the specific subset selected for a given fine-tuning job — a filtering or sampling step applied between corpus-level cleaning and fine-tuning-specific dataset assembly is exactly the kind of place a new defect can be introduced that EDA, run immediately before training, is positioned to catch.

• EDA before fine-tuning is five distinct checks, not one holistic judgment: distribution analysis, length statistics, vocabulary analysis, label distribution, and quality assessment, each catching a category of defect the other four cannot.
• It is an explicit best practice performed before fine-tuning, not a diagnostic tool reserved for after a result disappoints — the whole value of the checklist is catching a defect while it is still a five-minute check rather than a multi-day root-cause investigation.
• Length statistics has to be checked against the specific model's context window, not treated as a fixed property of the dataset independent of which model is being fine-tuned.
• Label distribution and general distribution analysis are independent findings. A dataset can look balanced on one general property while being severely skewed on the actual supervised label.
• Skipping EDA hides imbalance, leakage, and truncation risk specifically — the source material's own named consequence — and each of those three failure modes maps to a distinct step in the checklist rather than one generic "bad data" category.
• A dataset can pass four of the five checks and still fail catastrophically on the fifth, which is exactly why all five are run independently rather than treated as redundant with each other.

This closes Module M3. A dataset that has cleared cleaning and curation, correct formatting, a deliberate tokenizer merge-rule choice, a considered vocabulary-size tradeoff, and this five-step EDA pass is, at last, safe to fine-tune on — but being safe to fine-tune on says nothing yet about whether the resulting model is small and fast enough to actually deploy.

Next: M4-01 picks up exactly there, opening Model Optimization, the exam's single largest domain at 17% of the blueprint — PTQ versus QAT versus GPTQ, the first of several levers for making a trained model smaller and faster without wrecking the accuracy this module's five checks just worked to protect.