Exploratory Data Analysis (EDA) on a Text Corpus: Profiling Before You Trust It

EDA on a text corpus is the systematic measurement of a corpus's shape and defects before it is used, producing a profile you can compare against later. Two clauses are load-bearing. "Before it is used" places it upstream of indexing and training, not as a debugging step afterwards. "A profile you can compare against later" makes it the baseline for drift detection — the numbers you record today are what tells you six months from now that the corpus has changed.

Text EDA differs from tabular EDA in four ways that matter:

That last row has practical force. Tabular intuition expects roughly bell-shaped numeric distributions and treats extreme values as outliers to consider removing. Text length distributions are typically right-skewed with a long tail — many short documents, a few enormous ones — and the enormous ones are usually real and important, not errors. Applying an outlier-removal reflex to a document-length distribution deletes your appendices, your policy manuals, and your most information-dense sources.

L1 — Intuition: seven questions, cheapest first

Run these in order. Each one can end the investigation by revealing something that makes the later measurements meaningless.

1. How many documents, and how many are usable? Count parsed successfully, parsed empty, and failed.
2. How long are they, in tokens? Distribution, not mean.
3. How much is duplicated? Exact, then near-duplicate.
4. What languages and encodings? Including the mojibake check.
5. How large is the vocabulary, and how diverse? Type–token ratio, top-n terms.
6. What structural failures are present? Empty extractions, boilerplate-only documents, broken tables.
7. How are labels or categories distributed? If it is a labeled set.

The reason for the ordering is that step 3 invalidates steps 5 and 7 if skipped. If 27% of your corpus is near-duplicates — the figure from the illustrative pipeline in 08-01 — then your vocabulary statistics and label proportions are statements about a corpus that repeats itself, not about your domain.

L2 — What each measurement is, and the number that should worry you

Document count and parse status. Three counts, always reported together: successfully parsed, parsed to empty or near-empty, parse failed. The trap is that PDF extraction failures do not throw — they return an empty string or a page of ligature garbage, and a pipeline that filters on non-null keeps them. 06-01 covers this failure class in full; the EDA obligation is to count it. Worry when: more than a few percent parse to under ~50 tokens, or when the failure rate differs by source (which means one source is systematically broken).

Length distribution in tokens. Report the full shape: minimum, 25th percentile, median, 75th, 90th, 99th, maximum. Never the mean alone — a right-skewed distribution's mean sits above its median and describes nothing you can act on. The reason tokens are mandatory rather than preferable:

• Your context window is a token budget (04-06).
• Your API cost is per token (12-09).
• Your chunk size is set in tokens (06-02).
• Your model's hard input limit is in tokens.
• The characters-per-token ratio is not constant. It varies with language (languages that the tokenizer's training data under-represented fragment into more tokens per character), with content type (code, URLs, tables, and identifiers fragment heavily), with numerals, and with the tokenizer itself — BPE, WordPiece, and SentencePiece produce different counts for the same string, which is the substance of 02-04.

So a character count converted to tokens by a fixed divisor gives you an estimate whose error is systematic and correlated with exactly the documents you are most likely to mis-handle. Count with the actual tokenizer you will use. 02-03 is the lesson on this, and it exists as its own lesson precisely because the "tokens are not words" point is Tier-1 exam material.

Worry when: the 99th percentile exceeds your context window (you have a truncation problem you have not planned for), the median is under ~30 tokens (your "documents" may be fragments), or the distribution is bimodal (you probably have two document populations mixed together and should profile them separately).

Duplication rate. Exact duplicate rate by hash, then near-duplicate rate by shingling/MinHash or embedding similarity. Also measure boilerplate frequency: the most common repeated line or paragraph across documents, which in a real corpus is a navigation menu, a confidentiality footer, or a cookie banner. Worry when: exact duplicates exceed a few percent, near-duplicates exceed ~10%, or any single paragraph appears in more than a small fraction of documents. The retrieval consequence is in 06-04; the statistical consequence is that every other number in your profile is inflated by the duplication factor.

Language and encoding. Run language identification per document, report the distribution, and separately check for encoding damage — the ’-class mojibake that indicates a UTF-8 string decoded as Latin-1 somewhere upstream. Worry when: any unexpected language exceeds a fraction of a percent (either your source is broader than you thought or your language ID is misfiring on short documents, and both are findings), or when mojibake appears at all, because it is never isolated.

Vocabulary size and lexical diversity. Vocabulary size is the count of distinct types. Lexical diversity is typically measured as the type–token ratio — distinct words divided by total words — and it is a measure of vocabulary richness. It must be distinguished from syntactic complexity, which measures sentence structure: clause depth, sentence length, subordination. Those two are separate metrics of separate properties, and confusing them is on the explicitly reported confusable list for this exam [FIELD]. A corpus of short, simple sentences using a huge specialised vocabulary has high lexical diversity and low syntactic complexity; legal prose typically has the opposite profile. One important caveat when you compute type–token ratio: it is length-dependent — longer texts trend toward lower TTR simply because common words repeat — so comparing TTR across documents of very different lengths is comparing lengths, not diversity. Normalise by sampling equal-length windows if the comparison matters.

Report the top 20–50 terms too. In a healthy corpus they are stopwords plus your domain's obvious nouns. When something surprising appears in the top 20, that is usually boilerplate you have not noticed.

Label distribution. Counts per class, and the ratio of largest to smallest. This is where the class imbalance of 08-02 becomes visible, and where an intent taxonomy from 08-01 gets checked for empty cells.

L3 — Where the preprocessing pipeline fits, and stemming vs lemmatization

Text EDA raises a question tabular EDA does not: should I normalise the text before measuring it? The answer depends on what you are measuring, and it forces the classical preprocessing pipeline into view.

The canonical classical-NLP text preprocessing order is:

lowercase  →  strip special characters / punctuation  →  tokenize
           →  remove stopwords  →  lemmatize (or stem)

Each step destroys information deliberately. Lowercasing destroys the Apple/apple distinction. Stopword removal destroys negation (not is a stopword in many default lists, which is how a sentiment pipeline learns that "not good" is "good"). Stemming destroys morphology.

The last two steps are the same slot filled two different ways, and this is one of the most heavily reported items on this exam [FIELD]:

Memorise the one-line discriminator: stemming chops, lemmatization looks up. A stem may not be a word; a lemma always is. If a question gives you studies → studi, that is stemming, unambiguously. If it gives you better → good or was → be, that is lemmatization, because no truncation rule can produce those. Lesson 02-05 is the owning lesson for this pair and drills it fully; the EDA-side relevance is that whether you lemmatize changes your vocabulary count, often by a large margin, so a vocabulary statistic without a stated preprocessing recipe is not comparable to anyone else's.

The modern caveat matters as much as the pair itself: transformer-based LLMs need far less of this pipeline than classical NLP did. Subword tokenization handles morphology implicitly — running and run share subword structure that the model's embeddings relate — and lowercasing, stopword removal, and lemmatization typically destroy signal an LLM would have used. The pipeline remains correct for bag-of-words and TF-IDF feature extraction (02-06), for classical classifiers, and for keyword search normalisation. It is wrong as a default preprocessing step before feeding text to a transformer. Two practical EDA consequences: profile the raw text when your downstream consumer is an LLM, and profile the normalised text when your downstream consumer is a TF-IDF model — and never mix the two in one table without labelling which is which.

Which brings the rule: profile raw for token budgeting and cost, profile normalised for vocabulary and diversity, and label every statistic with its recipe.

The distinction the exam actually tests here is inspect versus modify. EDA and profiling are read-only; cleansing and transformation change the data. A stem describing "identifying which columns have missing values and what proportion" is EDA. A stem describing "filling missing values with the column median" is cleansing. And the sequencing rule stated in the domain's own scope statement — inspect, cleanse, transform, model — is itself a testable ordering, because cleansing before inspecting means you decided what was wrong before you looked.

One more comparison worth having, because it is a live decision in real work:

Automated profilers are genuinely useful and genuinely blind to the token question. That blindness is the reason this lesson exists.

All figures below are constructed for illustration to expose the reasoning; none are measured from a real corpus.

You inherit 12,000 files intended as a RAG corpus for a support assistant: exported help-centre articles, PDF product manuals, and a dump of resolved-ticket threads.

Step 1 — counts and parse status.

The 10% near-empty share is the first finding, and it is not spread evenly. Broken down by source: 4.1% of help-centre articles, 2.8% of ticket threads, and 41% of the PDF manuals. That single cross-tabulation localises the problem to PDF extraction. Nothing about the aggregate 10% would have told you that; the cross-tab did. This is the general principle that reappears in 08-04 — a number aggregated across groups can hide a total failure in one group.

Step 2 — length distribution, characters vs tokens. Here is the whole lesson in one table. Same 10,412 documents, measured both ways, with a naïve 4-characters-per-token conversion shown next to the real tokenizer count:

Read the right-hand column. The estimate is roughly acceptable at the median and catastrophically optimistic in the tail — and the tail is where the decisions are. Why does the error grow? Because the long documents are the PDF manuals, and they are dense with part numbers, tables, configuration snippets, and units, all of which fragment into many tokens per character. The naïve estimate says your largest document is 53,500 tokens; it is 96,300. If you had sized a pipeline on the estimate, the manuals would have silently truncated.

The actionable finding: with a hypothetical 8,000-token context limit, the naïve estimate says ~4% of documents exceed it; the real count says ~11% exceed it. That is the difference between "an edge case" and "a chunking requirement," and it changes the design.

Step 3 — duplication.

The footer is the finding that matters most. It appears in 85.5% of the corpus, contributes roughly 96 × 8,904 ≈ 855,000 tokens of pure noise, and in a vector index it becomes a near-neighbour of any query whose phrasing resembles legalese. It also inflates every vocabulary and diversity statistic. Strip boilerplate before you compute anything else.

Step 4 — language and encoding. 97.1% English, 2.2% Spanish (a genuine finding — a support region nobody mentioned), 0.7% "unknown," which on inspection is mostly documents so short that language ID is unreliable. Mojibake check: 217 documents contain â€ sequences, all from one export batch. One upstream encoding bug, one batch to re-export.

Step 5 — vocabulary and lexical diversity, computed after boilerplate stripping and deduplication, and reported twice because the recipe changes the number:

Lemmatization and lowercasing together halve the vocabulary. So "our corpus has a 71,400-word vocabulary" and "our corpus has a 148,300-word vocabulary" are both true statements about the same corpus, and neither is meaningful without its recipe. Note also that the top-20 term list before boilerplate stripping was led by confidential, notice, and intended — the footer — and after stripping was led by ordinary stopwords and product nouns. The top-terms list is a boilerplate detector.

Step 6 — structural failures. Beyond the parse counts: 340 documents whose entire content is a table that extracted as a single run-on line, and 88 whose content is a navigation menu. Both classes are technically non-empty and would survive a null filter.

Step 7 — the profile, frozen. Record all of the above with a date and a corpus version. Six months later, re-run it: if the median token length has moved 30% or a new language has appeared at 4%, you have detected covariate drift in your corpus before it degrades retrieval — the 08-02 pathology, caught by having a baseline.

Summary of what the profile changed: the corpus goes from "12,000 documents" to 8,600 usable, deduplicated, boilerplate-stripped documents with a known token distribution, a known 11% over-limit fraction driving the chunking design, one localised PDF extraction bug, one encoding bug, and a Spanish subset that needs a product decision. None of those were visible in the file count.

When to stop: when every additional measurement stops changing a decision. EDA is instrumental, not decorative — a profile with 40 charts and no design consequence is a hobby. The discipline is to state, next to each number, the decision it informs. If you cannot name the decision, do not compute the number.

One thing never to skip, at any depth: reading actual documents. Read twenty at random, and read five from each tail of the length distribution. Statistics tell you the shape; only reading tells you that the shortest documents are all cookie banners and the longest are all one manual chunked wrongly. Every experienced practitioner has a story where the finding came from reading, not from a histogram.

Objectives served. Directly: 2.1 "awareness of the process of extracting insights from large datasets using data mining, data visualization, and similar techniques", 2.3 "conduct data analysis under the supervision of a senior team member", and 2.5 "identify relationships and trends or any factors that could affect the results of research" [OFFICIAL]. The Core ML domain's 1.2 restates 2.1 verbatim (the duplicate pair 08-02 owns), and 1.6 / 1.10 — familiarity with Python NLP packages including spaCy and NumPy — is where the preprocessing and tokenization tooling is examined. The domain is 14% of the exam, about 8 of 60 questions [OFFICIAL].

Where the exam's weight actually sits inside this lesson. Not on EDA process — that is Tier 3 in the [FIELD] priority tiers. It sits on two items that this lesson happens to be the natural home for:

• Stemming vs lemmatization is explicitly reported as an exam item and appears on the confusable-pairs list [FIELD]. It is Tier 1. If you learn one thing here cold, learn that stemming truncates crudely and may produce a non-word while lemmatization uses a dictionary and always produces a valid root.
• Tokens are not words is Tier 1 and underlies the "measure in tokens" rule.

Everything else in this lesson is context that makes those two stick and protects you against the scenario-shaped questions where a described symptom traces back to a corpus nobody profiled.

Question phrasings to expect

• "Which preprocessing technique reduces a word to its dictionary base form, guaranteeing the result is a valid word?" → lemmatization.
• "studies becomes studi. Which technique produced this?" → stemming.
• "What is the correct order of a classical text preprocessing pipeline?" → lowercase → remove special characters → tokenize → remove stopwords → lemmatize.
• "Why should document length be measured in tokens rather than characters when preparing a corpus for an LLM?" → because context windows, cost, and chunk sizes are token-denominated and characters-per-token is not constant across languages, content types, and tokenizers.
• "Which metric measures vocabulary richness rather than sentence structure?" → lexical diversity (type–token ratio); syntactic complexity is the sentence-structure one.
• "What should be done before modelling a newly acquired dataset?" → exploratory data analysis / inspect the data. The EDA-before-modelling rule is stated in the domain's own scope.
• "A team removes stopwords before training a sentiment classifier and accuracy drops. Why?" → negation words are on default stopword lists.
• "Which Python library provides tokenization, POS tagging, NER, lemmatization, and dependency parsing?" → spaCy (named in the official objectives).

Distractor families

Because every constraint you are actually managing is denominated in tokens and the conversion factor is not stable. The context window is a token count; the API bill is per token; the chunk size you pass to a splitter is in tokens; the model's hard input limit is in tokens. Characters relate to tokens through a ratio that varies with the language (text in languages under-represented in the tokenizer's training data fragments into more tokens per character), with content type (code, URLs, product codes, tables, and long numbers fragment heavily), and with the tokenizer itself — BPE, WordPiece, and SentencePiece all count the same string differently, per 02-04. Critically, the error is not random: it is largest for exactly the dense, technical, long documents that are most likely to overflow a limit. The illustrative table in section 4 shows a naïve estimate that is 8% low at the median and 80% low at the maximum. Count with the tokenizer you will actually use; 02-03 is the lesson that makes this computable.

Stemming applies crude rule-based truncation to strip suffixes and produces a stem that need not be a real word — studies becomes studi, was becomes wa. Lemmatization performs a dictionary and morphology lookup, often using part-of-speech context, and produces a lemma that is always a valid dictionary word — studies becomes study, was becomes be, better becomes good. Stemming is faster and needs no lexicon; lemmatization is slower, needs a lexical resource, and is more accurate. The discriminator to carry into the exam is the output: if the result is not a word, it was stemming. And note the second-order examples, because they are the ones that make the distinction unmistakable — no truncation rule can turn better into good or was into be, so those transformations can only be lemmatization. 02-05 owns and drills this pair.

Far less of it, and applying the full classical pipeline to transformer input is usually harmful. Subword tokenization already handles morphology, so lemmatization is redundant; contextual embeddings use case and punctuation as signal, so lowercasing and stripping punctuation destroy information; and stopword removal deletes negation and function words that carry meaning the model uses. What is still required for LLM work is different in kind: correct text extraction from source formats, encoding normalisation, boilerplate and duplicate removal, and chunking at token boundaries. The classical pipeline remains fully correct where it always was — bag-of-words and TF-IDF feature extraction (02-06), classical classifiers, and keyword-search index normalisation, which is why sparse retrieval in 07-01 still cares about it. So the honest answer is that preprocessing did not disappear; it moved from normalising words to repairing documents.

Seven blocks, each with the decision it informs stated next to it. Counts and parse status, cross-tabulated by source, informing which extraction pipeline to fix. Token-length percentiles (p25/50/75/90/99/max) plus the fraction exceeding your context limit, informing chunk size and truncation strategy. Duplication rates (exact, near, and top repeated paragraph), informing dedup and boilerplate stripping. Language and encoding distribution, informing whether you need multilingual handling and which export batch to fix. Vocabulary and lexical diversity, labelled with the preprocessing recipe, informing feature choices and comparability. Structural failure counts (empty extractions, table-only, menu-only), informing filters. Label or category distribution against your intent taxonomy, informing coverage and metric choice. Then a date and a corpus version, because the report's second job — arguably its more valuable one — is to be the baseline you diff against at the next refresh.

1. Seven questions, in order: counts and parse status, token-length distribution, duplication, language and encoding, vocabulary and diversity, structural failures, label distribution. Duplication must be measured before vocabulary or labels, or those numbers describe a corpus that repeats itself.
2. Measure length in tokens, never characters. The naïve character-divided-by-four estimate is roughly right at the median and badly wrong in the tail — and the tail sets your chunking and truncation design.
3. Report percentiles, not the mean. Document length is heavy-tailed; the mean describes nothing you can act on, and the long documents are usually real rather than outliers.
4. Cross-tabulate by source. A 10% aggregate near-empty rate concealed a 41% failure in one source in the worked example. Aggregates hide localised failure — the theme 08-04 formalises.
5. The most frequent repeated paragraph is your highest-yield single measurement. In the illustration an 85.5%-prevalence footer contributed ~855,000 tokens of noise and would have polluted retrieval.
6. Stemming chops, lemmatization looks up. A stem may not be a word (studies → studi); a lemma always is (studies → study, was → be). Tier-1, explicitly reported [FIELD].
7. Lexical diversity ≠ syntactic complexity. Vocabulary richness versus sentence structure — a separate explicitly-reported confusable.
8. Label every text statistic with its preprocessing recipe. Lowercasing plus lemmatization halved vocabulary size in the worked example, so an unlabelled count is not comparable to anything.
9. The classical pipeline is right for TF-IDF and wrong before a transformer. For LLM work, preprocessing means repairing documents, not normalising words.
10. Read the documents. Twenty at random, five from each length tail. No statistic substitutes for it.
11. Freeze the profile. Its second job is to be the baseline that reveals corpus drift.

You have distributions, rates, cross-tabs, and a percentile table. Every one of them is now a candidate for a chart, and the wrong chart type does not merely look worse — it answers a different question from the one you asked.

Next: 08-04 maps chart type to purpose exactly — histogram for a distribution, box plot for spread and outliers, scatter for a relationship, bar for comparison, heatmap for a matrix or correlation, line for a trend over time — and then makes the argument that matters more than the mapping: an aggregated chart is structurally incapable of showing group-level harm, which is precisely why the cross-tabulation habit from this lesson turns honest visualization into a fairness instrument rather than a presentation skill.