Deduplication and corpus cleaning for RAG

Corpus cleaning is the set of ingestion operations that decide which text should not be indexed. It has three distinguishable jobs, and the exam-relevant skill is naming which job a described symptom belongs to.

Boilerplate removal is the one people skip, and it is the one the design note for this lesson points at: a footer repeated across 4,000 pages becomes the nearest neighbour of every query. It is not a duplicate document. Every page it sits on is genuinely distinct. The repetition is inside the documents, which means document-level deduplication will not touch it and will report the corpus as clean.

Two framings worth holding onto. First, this is the RAG-specific instance of a general data-quality discipline the blueprint names directly: the canonical defect list in data cleansing is missing values, duplicates, outliers, format inconsistencies, and data leakage, and duplicates are on it. [OFFICIAL] on the objective's scope; the defect list is this course's canonical framing of it. Second, deduplication and quality filtering at scale is exactly what NVIDIA's NeMo Curator exists to do — curation as a distinct stage in the stack, separate from NeMo for build and customise, NeMo Retriever for retrieval, and NIM for packaged deployment. [NVIDIA-DOC] For the exam, know that identity: which NVIDIA component curates and deduplicates data at scale? NeMo Curator.

L1 — Why a footer becomes everybody's nearest neighbour

Take a running footer: © 2024 Acme Corporation. Confidential. Page 14 of 400. It carries almost no topical content. Embed it and you get a vector that is not pointing towards any subject in particular — it sits in a bland region of the space, near the centre of the cloud rather than out at an edge where "warranty periods" or "study leave" live.

Now consider what a query does. A query embedding lands somewhere in the space, and similarity search returns whatever is nearest. A vector out at the "warranty" edge is very near warranty queries and very far from leave queries. A vector near the centre is moderately near everything. Across a thousand diverse queries, the centre-ish vector is a plausible top-20 candidate for most of them, whereas each edge vector is a strong candidate for only its own neighbourhood. Multiply by 4,000 copies and you have 4,000 moderately-good-for-everything candidates crowding the shortlist for every query in your system.

The damage is worse than one bad result, for two reasons. The chunks are near-identical, so they do not fail independently — when one surfaces, several surface together, and they can take multiple slots of a top-5. And every slot they take is a slot a real answer did not get. This is why the failure is described as crowding out rather than as noise.

L2 — The four mechanisms of duplicate harm

The third mechanism is the most dangerous because it produces a confidently wrong answer with a valid citation, which is the hardest failure to catch in review. A reader who checks the citation finds the quoted text exactly where the system said it was. The document is real. The page is right. It is simply the 2019 edition.

L3 — Detection at scale: hashing, shingles, and thresholds

Exact deduplication is cheap and should always be on. Normalise the text — collapse whitespace, normalise Unicode, optionally strip case for the comparison only — hash it, and keep the first occurrence of each digest. This is O(n) and catches the multiple-copies-in-multiple-drives case that accounts for a great deal of real corpus duplication.

Near-duplicate detection is harder because naive pairwise comparison is O(n²), which is fine for 10,000 chunks and impossible for 10 million. The standard family of solutions turns similarity into a hashing problem:

• Shingling. Represent a document as the set of its overlapping k-word sequences (shingles). Two documents differing by a date share almost all their shingles. Similarity is then the Jaccard overlap of the two sets.
• MinHash. Compress each shingle set into a short signature such that the probability two signatures agree in a position equals their Jaccard similarity. Comparing signatures is far cheaper than comparing sets.
• Locality-sensitive hashing (LSH). Bucket signatures so that similar items collide, and only compare within buckets. This is what turns the O(n²) problem into something tractable.
• SimHash. An alternative producing a single fingerprint whose Hamming distance approximates similarity — cheap, and commonly used for web-scale near-duplicate detection.
• Embedding similarity. You have the vectors already; near-duplicates are pairs above a high cosine threshold. Convenient, but it costs an embedding first, and it conflates "same text" with "same meaning" — two genuinely different passages saying the same thing are semantically near-duplicates, and whether you want to remove those is a judgement call.

You are not expected to implement MinHash for this exam. You are expected to know that near-duplicate detection at scale is done with hashing-based approximation rather than pairwise comparison, and that this is the kind of work a curation tool built for scale performs on your behalf.

Boilerplate detection uses a different signal entirely — cross-document frequency. Count how many distinct documents (not occurrences) contain each candidate line or short span. Real content appears in one or a few documents. Boilerplate appears in nearly all of them. A line present in 92% of documents in a heterogeneous corpus is furniture, whatever it says.

Boilerplate detection by document frequency — constructed example

line                                            docs containing it   verdict
"© 2024 Acme Corporation. Confidential."               3,914 / 4,002   boilerplate
"Page __ of __"  (after digit normalisation)            3,880 / 4,002   boilerplate
"This document is uncontrolled when printed."           3,102 / 4,002   boilerplate
"Unsubscribe from these notifications"                  1,455 / 4,002   boilerplate (email footer)
"Study leave is granted at the manager's discretion"        2 / 4,002   content
"Operating temperature range for the XR-40 unit"            1 / 4,002   content
Constructed illustration, not measured.

Note the digit normalisation on the second row. Page 14 of 400 and Page 15 of 400 are different strings, so raw counting misses them; normalising digit runs to a placeholder before counting reveals the pattern. Templated boilerplate with a variable slot in it is extremely common and is invisible to exact matching.

The document-level dedup catches it? row is the diagnostic that separates the three in a question stem. If the described symptom is a short string surfacing for unrelated queries, the answer is boilerplate and no amount of document deduplication touches it. If the symptom is contradictory answers with valid citations, the answer is near-duplicate versions. If the symptom is the same passage repeated in a retrieved set, exact duplicates.

A constructed diagnosis. All numbers are invented for teaching and are not measured results. Constructed scenario.

A team indexes a 4,000-page technical documentation set exported as PDFs, chunked recursively at 512 tokens with 64-token overlap, roughly 9,600 chunks. Users report that answers are "vague" and that the assistant "keeps talking about confidentiality".

Step 1 — count the most frequent chunk prefixes.

Top repeated 40-character prefixes across 9,600 chunks

"© 2024 Acme Corporation. Confidenti"            2,918 chunks
"Acme Technical Documentation — Rev "            2,461 chunks
"This document is uncontrolled when "            1,004 chunks
Constructed counts.

The parser extracted the page furniture on every page. The chunker, working on a stream of text that now had a footer spliced in every ~600 tokens, distributed those strings across nearly a third of all chunks. Note what happened: the footer is not a separate chunk in most cases — it is embedded inside real content chunks, so it contaminates their vectors rather than merely adding junk ones.

Step 2 — look at where those chunks rank.

20 held-out real user questions, top-5 retrieved from the current index

slots occupied by chunks whose text is >50% boilerplate       17 of 100
questions where at least one top-5 slot went to boilerplate   13 of 20
questions where the known answering passage was in top 5      11 of 20
Constructed measurement on a constructed index.

Thirteen of twenty questions lost at least one of their five slots to furniture. That is the crowding mechanism, quantified in the only way that means anything: against a fixed question set, which is the by-hand retrieval test from 03-04 applied to a cleaning decision.

Step 3 — clean and re-index.

Cleaning applied
  1. Strip lines whose document frequency exceeds 60% of documents,
     after normalising digit runs and collapsing whitespace
  2. Drop chunks whose remaining text is under 40 tokens
  3. Exact-dedup on normalised chunk hash

Result
  chunks before                       9,600
  chunks after                        8,730     (−9%)
  boilerplate-dominated chunks            0
  chunks removed as exact duplicates     214     (mostly identical warning
                                                  blocks repeated verbatim)

Same 20 questions, re-measured
  slots occupied by boilerplate               0 of 100
  answering passage in top 5             16 of 20   (was 11)
Constructed measurement.

The corpus got 9% smaller and the retrieval outcome on the fixed question set improved substantially. The direction of that result is the point, not its magnitude — your corpus will move by a different amount, and the only way to know how much is to run the same three steps.

Two details of the fix deserve highlighting. The 60% document-frequency threshold is a knob, not a law: set it too low and you delete a standard clause that genuinely is content in a contract corpus; too high and templated furniture survives. And step 2 — dropping chunks that are almost nothing after cleaning — matters because a chunk stripped down to Page of is a fragment whose vector is even blander than the footer's was.

Repetition is not always a defect, and the judgement of when to keep it is the part a naive cleaning pipeline gets wrong.

The contract-clause row is the one that stops this from becoming a rule you apply blindly. High repetition means "this is probably furniture" and not "this is definitely furniture", and the difference is decided by what the corpus is for. In a documentation corpus, a paragraph appearing in 300 documents is boilerplate. In a contract corpus, it is the indemnity clause, and which contracts contain it is the question the whole system exists to answer.

This topic sits at the intersection of two blueprint areas, which is why it earns exam attention out of proportion to how much of the tutorial literature covers it.

From the Data Analysis and Visualization scope: the official statement is "inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information," and objective 2.3 is "Conduct data analysis under the supervision of a senior team member." Duplicates are a named member of the canonical defect list — missing values, duplicates, outliers, format inconsistencies, data leakage — and naming the defect behind a described symptom is a repeatedly emphasised skill for this material. From Core ML and AI: objective 1.4, "Curate and embed content datasets for RAGs," makes curation an explicit, tested activity, and deduplication is the largest single part of curating a corpus. [OFFICIAL] on objective wording.

There is a second, quieter reason. Duplication is also the mechanism behind train/test contamination — the same passage present in both an evaluation set and a training or index set inflates measured performance without improving anything real. That is a data-quality failure with the same root cause and a completely different symptom, and it connects this lesson to 10-01 and 08-02. If your evaluation questions were drawn from documents that also exist as duplicates in the index, your retrieval numbers are measuring the duplication.

Question phrasings to expect:

• "A RAG assistant returns the company's confidentiality notice as a top result for many unrelated queries. What is the cause?" — repeated boilerplate producing a cluster of low-content vectors near the centre of embedding space. Not an embedding-model problem.
• "Which data defect is described: the same record appears multiple times in the dataset?" — duplicates, from the canonical defect list.
• "Which NVIDIA tool is used for large-scale data curation, including deduplication and quality filtering?" — NeMo Curator.
• "A RAG system gives an answer that contradicts current policy but cites a real document. What ingestion problem is most likely?" — multiple versions of the document indexed with no version metadata or filtering.
• "What is the effect of indexing four copies of the same document?" — retrieved results crowd with duplicates, effective k falls, context budget is wasted.
• "When should deduplication be performed in a RAG pipeline?" — at ingestion, before embedding and indexing. Afterwards, every duplicate is already a vector competing in the index.
• "Why is duplication a problem for evaluation as well as retrieval?" — contamination inflates measured performance.

Distractor families:

Because low-content text embeds to a low-specificity location, and a low-specificity location is moderately close to everything.

Put it geometrically. An embedding model maps text into a space where direction encodes meaning. A passage about warranty periods points firmly in the warranty direction: very close to warranty queries, far from everything else. A footer reading © 2024 Acme Corporation. Confidential. Page 14 of 400. has almost no topical direction to encode — it is generic administrative language — so its vector lands in a bland central region rather than out at a topical edge. Its similarity to any given query is mediocre. Its similarity to every query is mediocre, and mediocre-for-all beats excellent-for-one whenever the excellent-for-one candidate is not the current query's subject.

Then multiply. Four thousand copies of that footer are 4,000 vectors packed into that central region. Every query in your system now has thousands of moderately-similar candidates that no query is actually about. Some of them clear the top-k threshold. Because they are near-identical, they clear it together, and because they are individually plausible rather than absurd, no threshold or score filter cleanly excludes them.

Two consequences follow that are worth stating separately. First, the failure scales with corpus size, so it gets worse exactly as the system gets more valuable — a 40-page corpus barely notices, a 40,000-page corpus is ruined. Second, this is not fixed by a better embedding model. A better model represents that footer more accurately, and the footer accurately represented is still generic administrative text sitting in the middle of the space. The defect is in the corpus, and corpus defects are fixed in the corpus.

Both, at different levels, and the order matters.

The organising principle is the one line worth memorising from this section: clean before you embed, because embedding is where a text problem becomes an index problem. Text is cheap to edit. A vector in an index is a thing with storage, latency, and ranking consequences, and it has already cost you the embedding call. Everything upstream of embedding is reversible with a re-run; everything downstream is a migration (12-12).

One frequency pass, and it is genuinely a short piece of work:

1. Normalise for comparison. Collapse runs of whitespace, normalise Unicode, and replace digit runs with a placeholder so Page 14 of 400 and Page 15 of 400 become the same string. Keep the original text; the normalisation is only for counting.
2. Count by document, not by occurrence. For each distinct line or short span, count how many distinct documents contain it. Occurrence counts are dominated by long documents; document counts reveal furniture.
3. Sort descending and read the top hundred by eye. This is the step no automation replaces. Furniture is instantly recognisable to a human and the list is short.
4. Set a threshold and write down why. A line present in more than some large fraction of documents is a candidate. The number depends on the corpus, so record it as a decision.
5. Check the boundary cases against the corpus's purpose. Is a paragraph in 300 of 4,000 documents furniture, or the indemnity clause? Only you can say, and this is where blind automation deletes content.
6. Score the change against a fixed question set. Retrieve for the same 20 questions before and after. If the numbers do not move, you have at least made the index smaller for free; if they get worse, one of your rules ate content.
7. Re-run it on every ingestion. New templates bring new furniture, and a rule set that was complete last quarter is not complete now.

Two extra checks pay for themselves. Look at the shortest chunks in the corpus, because near-empty chunks are almost always cleaning residue or parsing debris, and they embed to blander vectors than the boilerplate did. And look at the chunks with the highest average similarity to all other chunks — that is a direct measurement of centroid pull, and whatever sits at the top of that list is what is competing with every query you will ever ask.

• Deduplication is a retrieval-quality decision, not a storage optimisation. Framing it as disk space misses the whole mechanism and is a distractor family in its own right.
• A repeated footer becomes moderately similar to every query. Low-content text embeds near the centre of the space, thousands of copies pack that centre, and mediocre-for-all beats excellent-for-one whenever the excellent candidate is off-topic.
• Three distinct problems, three distinct detections. Exact duplicates by hash, near-duplicates by shingling and LSH, boilerplate by cross-document frequency. Document-level deduplication finds boilerplate never.
• Duplicate versions produce confidently wrong answers with valid citations — the hardest RAG failure to catch in review. Fix it with version metadata and default filtering, not deletion.
• Clean before you embed. Text problems are cheap to fix; index problems are a migration.
• Repetition is sometimes the signal. A clause in 300 contracts is content. High frequency means probably furniture, and the corpus's purpose decides.
• Duplication also contaminates evaluation, inflating retrieval scores that production will not reproduce.
• Score every cleaning rule against a fixed question set, one rule at a time, or you will never know which rule ate your content.
• NeMo Curator is NVIDIA's answer for curation and deduplication at scale, distinct from NeMo Retriever and from the AI Blueprint for RAG. Identity depth is what the exam asks for. [NVIDIA-DOC]

The corpus is now parsed, chunked, described by metadata, and cleaned. That completes the ingestion half of RAG, and it means you finally know what your corpus looks like after a machine has read it — which was the question this module opened with. What you still have no way to do is find anything in it. Everything so far has been about the shape of the haystack; nothing yet has been about search. And the first search technique to learn is deliberately not the fashionable one: keyword matching, which is old, cheap, exact, unbeatable at rare terms and identifiers, and the baseline you must measure against before you are entitled to claim that any vector store improved anything.

Next: 07-01 Sparse retrieval and BM25 keyword search — the baseline that decides whether your embeddings are earning their keep.