Bag-of-words, TF-IDF, and n-grams explained

Bag-of-words (BoW) represents a document as a vector with one dimension per vocabulary term, where each value is the count of that term in the document. The name is literal: imagine emptying the document's words into a bag and shaking it. What survives is which words appeared and how many times. What is destroyed is the order, the syntax, and every relationship that depended on adjacency. A document collection becomes a document-term matrix — rows are documents, columns are vocabulary terms, cells are counts — and it is overwhelmingly sparse, because any single document contains a tiny fraction of the corpus vocabulary.

TF-IDF (term frequency–inverse document frequency) is a weighting scheme applied to a bag-of-words matrix, not a separate representation. It multiplies two factors: term frequency, how often the term appears in this document, and inverse document frequency, a factor that rises as the term appears in fewer documents across the corpus. The result: a term that is frequent here and rare elsewhere gets a high weight, which is exactly the profile of a term that distinguishes this document. A term appearing in every document — the, is, and — gets a weight near zero automatically, which is why TF-IDF partly obviates the stop-word lists of 02-05.

N-grams are contiguous sequences of n tokens treated as single terms. Unigrams are single tokens (not, good); bigrams are adjacent pairs (not good); trigrams are triples (not very good). Adding bigrams and trigrams to the vocabulary lets a bag-of-words model see local order — it can now distinguish not good from good — at the cost of a much larger and sparser feature space. N-grams are also a concept in their own right beyond feature engineering: n-gram language models estimate the probability of the next word from the previous n−1 words, which is the statistical ancestor of the next-token prediction in 01-01.

The three compose. The standard classical text-classification pipeline is: preprocess per 02-05 → build a bag-of-words matrix over unigrams and bigrams → apply TF-IDF weighting → fit a linear classifier. Each piece is independently swappable, and that modularity is part of why the approach survived.

L1 — The intuition

Bag-of-words is a tally sheet. List every word the corpus uses, then for each document put a mark in each word's box every time you see it. Two documents are similar if their tally sheets look similar. Crude, and it works surprisingly well for topic-level tasks, because topic is largely a matter of which words appear — a document containing mortgage, escrow and amortisation is about property finance regardless of the order those words came in.

TF-IDF is the observation that not all marks are worth the same. Every document has marks in the the box, so that box tells you nothing about which document you are looking at. Almost no document has a mark in the escrow box, so that box is enormously informative. Weight each box by how rare it is across the corpus and the informative boxes start to dominate the comparison. That is the entire idea, and it is one of the most durable ideas in information retrieval.

N-grams are the patch for the bag's worst property. not good and good should not look the same, but with unigrams they differ only by the presence of a not that could have been anywhere in the document. Add adjacent pairs as terms and not good becomes a feature in its own right. You have not recovered grammar — you have recovered a two-token window of adjacency, which for sentiment and phrase detection is often enough.

L2 — The mechanism, with the formulas

Building a document-term matrix. Two passes over the corpus:

1. Fit — walk the corpus, collect every distinct term, assign each a column index. This is the vocabulary. Optionally prune it: drop terms appearing in fewer than min_df documents (noise, typos, one-off identifiers) or more than max_df fraction of documents (corpus-specific stop words). Pruning is where most of the practical quality lives.
2. Transform — for each document, count occurrences of each vocabulary term and write them into the row.

The matrix is stored sparsely — only non-zero cells — because density is typically well under one percent. A corpus of 100,000 documents with a 50,000-term vocabulary is a 5-billion-cell matrix that might hold only a few million non-zeros. Sparse storage is not an optimisation here; it is the difference between feasible and impossible.

TF-IDF, term by term. Term frequency has several conventions and the choice matters:

raw count:        tf(t,d) = count of t in d
normalised:       tf(t,d) = count(t,d) / total terms in d
log-scaled:       tf(t,d) = 1 + log(count(t,d))
boolean:          tf(t,d) = 1 if t in d else 0

Raw counts let long documents dominate simply by being long; length normalisation or log scaling fixes that. Log scaling encodes a real intuition: the tenth occurrence of a term tells you much less than the second did.

Inverse document frequency, in the standard smoothed form:

idf(t) = log( N / df(t) )                    unsmoothed
idf(t) = log( (1 + N) / (1 + df(t)) ) + 1    smoothed, as scikit-learn defaults

where N is the number of documents in the corpus and df(t) is the number of documents containing term t. The 1 + terms prevent division by zero for a term unseen in the fitted corpus, and the trailing + 1 ensures a term appearing in every document gets a non-zero weight rather than being annihilated.

The product is the weight:

tfidf(t,d) = tf(t,d) × idf(t)

Rows are then usually L2-normalised so that every document vector has unit length. This is what makes cosine similarity between two TF-IDF rows reduce to their dot product, which is why 01-04's dot-product-and-cosine machinery applies directly to sparse vectors as well as dense ones.

N-gram extraction. For an n-gram range of (1,2), slide a window of size 1 and then size 2 across the token sequence:

tokens:   ["the","service","was","not","good"]
unigrams: the · service · was · not · good                       5 terms
bigrams:  the service · service was · was not · not good         4 terms
combined: 9 terms from a 5-token document

The vocabulary growth is the cost. A corpus with 50,000 distinct unigrams can easily have several hundred thousand distinct bigrams and millions of distinct trigrams, most occurring once. This is why min_df pruning is effectively mandatory with n-grams: without it the feature space is dominated by singletons that cannot generalise.

Note the interaction with 02-05 that trips people up: stop-word removal must not precede n-gram extraction if you want meaningful bigrams. Remove not and was and the not good bigram never forms; worse, you manufacture bigrams like service good that never occurred in the source text.

L3 — Why IDF works, and what sparsity costs

Why the log in IDF? Because the informativeness of rarity has diminishing returns. Consider a corpus of N = 1,000,000 documents:

term in 500,000 docs → idf = log(1,000,000/500,000) = log(2)      ≈ 0.69
term in  10,000 docs → idf = log(1,000,000/ 10,000) = log(100)    ≈ 4.61
term in     100 docs → idf = log(1,000,000/    100) = log(10,000) ≈ 9.21
term in       1 doc   → idf = log(1,000,000/     1) = log(10^6)   ≈ 13.82

The document frequency drops by factors of 50, 100 and 100 across those steps, but the IDF rises by only 4.6, 4.6 and 4.6. Without the log, a hapax legomenon — a term appearing exactly once — would get a weight a million times larger than a term appearing in half the corpus, and the representation would be entirely determined by typos and unique identifiers. The log is what keeps the scheme from being dominated by noise. This connects IDF to information theory: log(N/df) is the self-information of the event "a randomly chosen document contains t," so IDF weights each term by how surprising its presence is.

What sparsity costs, precisely. Sparse lexical representations have one structural limitation, and every practical weakness follows from it: each vocabulary term is an independent, orthogonal dimension. The consequences:

• No synonymy. car and automobile are different columns with zero relationship. A query for one retrieves nothing for the other. This is the vocabulary mismatch problem and it is the core motivation for dense retrieval in 07-02.
• No polysemy handling. bank as riverbank and bank as financial institution share one column, so their evidence is merged.
• No morphological relationship unless you forced one with the stemming or lemmatization of 02-05. run and running are unrelated columns otherwise.
• No word order beyond the n-gram window. A bigram model sees two-token adjacency and nothing about long-range structure. Negation more than one token from its target — "the food, which I had expected to enjoy, was not in any sense good" — escapes a bigram feature.
• No transfer. The vocabulary is fitted on your corpus. A term absent at fit time is simply dropped at transform time, silently.

Set against that, sparse representations have genuine advantages that dense embeddings do not:

• Exact-match precision. Searching for an error code, a part number, a function name or a legal citation is a case where you want literal matching, and a dense embedding will happily return something semantically adjacent instead. This is why hybrid search exists (07-06).
• Interpretability. Each feature is a term you can read. A linear model's coefficient on a TF-IDF feature is directly inspectable: "the word refund contributes +0.8 to the churn prediction." No dense embedding offers that.
• No training required for the representation. Fitting is counting. There is no GPU, no pretrained model, no embedding-version pinning, no migration when you re-index (12-12).
• Strong small-data behaviour. With a few hundred labelled examples, TF-IDF plus a linear model is frequently competitive with or better than a fine-tuned transformer, and takes seconds to train.

The comparison table this lesson exists to deliver.

Three relationships to hold clearly, because questions probe them:

1. TF-IDF is not an alternative to bag-of-words — it is a weighting of bag-of-words. Same sparse matrix, same columns, same zero pattern; different numbers in the non-zero cells. A question offering "bag-of-words or TF-IDF?" as an either/or is being loose, and the intended contrast is "raw counts versus IDF-weighted counts."
2. N-grams are not an alternative to either. They are a decision about what a term is. You can have a bag-of-bigrams with raw counts, or TF-IDF over unigrams-plus-bigrams. It is an orthogonal axis.
3. Embeddings are a genuinely different kind of thing. Dense, learned, not per-feature interpretable, and able to recognise that two different strings mean the same thing. That last capability is the one sparse methods cannot have, and it is why the course pivots to 03-01 next.

Full arithmetic on a corpus small enough to compute by hand. Every number below is derived by explicit calculation from the formulas in §2 — nothing is quoted.

The corpus (N = 4), already lowercased and tokenized:

d1: "the gpu trains the model"
d2: "the model trains fast"
d3: "the gpu is fast"
d4: "gpu memory limits the model"

Step 1 — build the vocabulary. Distinct terms across all four documents:

the, gpu, trains, model, fast, is, memory, limits        →  8 terms

Step 2 — the bag-of-words matrix. Rows are documents, columns are terms, cells are raw counts.

Immediately visible: the has the largest total count (5) and is the least informative term in the corpus. That is bag-of-words' central defect and exactly what IDF corrects.

Step 3 — document frequencies. How many of the 4 documents contain each term?

df(the)    = 4    (all)
df(gpu)    = 3    (d1, d3, d4)
df(model)  = 3    (d1, d2, d4)
df(trains) = 2    (d1, d2)
df(fast)   = 2    (d2, d3)
df(is)     = 1    (d3)
df(memory) = 1    (d4)
df(limits) = 1    (d4)

Step 4 — IDF, unsmoothed, natural log. idf(t) = ln(N/df(t)) with N = 4:

idf(the)    = ln(4/4) = ln(1.000) = 0.000
idf(gpu)    = ln(4/3) = ln(1.333) = 0.288
idf(model)  = ln(4/3) = ln(1.333) = 0.288
idf(trains) = ln(4/2) = ln(2.000) = 0.693
idf(fast)   = ln(4/2) = ln(2.000) = 0.693
idf(is)     = ln(4/1) = ln(4.000) = 1.386
idf(memory) = ln(4/1) = ln(4.000) = 1.386
idf(limits) = ln(4/1) = ln(4.000) = 1.386

Read the first line, because it is the punchline of the whole scheme. idf(the) = 0. A term appearing in every document gets weight exactly zero and drops out of the representation entirely — TF-IDF discovered the stop word by itself, with no list, no language knowledge and no configuration. That is why 02-05's stop-word removal is partly redundant when you use TF-IDF, and it is a favourite exam observation.

(Note that smoothed IDF — scikit-learn's default — adds 1 to numerator, denominator and result, so the would get a small positive weight rather than exactly zero. The unsmoothed form is used here because it makes the mechanism visible. Both are standard; know that the convention affects the number.)

Step 5 — TF-IDF weights for d1, using raw counts as TF:

d1 = "the gpu trains the model"

the:    tf=2 × idf=0.000  = 0.000
gpu:    tf=1 × idf=0.288  = 0.288
trains: tf=1 × idf=0.693  = 0.693
model:  tf=1 × idf=0.288  = 0.288
(all other terms: tf=0 → 0.000)

So d1's unnormalised vector is [0.000, 0.288, 0.693, 0.288, 0, 0, 0, 0]. the appeared twice — the most of any term — and contributes nothing. trains appeared once and dominates, because it is the rarest term d1 contains.

Step 6 — L2-normalise d1. Compute the vector's length:

||d1|| = sqrt(0.000² + 0.288² + 0.693² + 0.288²)
       = sqrt(0 + 0.082944 + 0.480249 + 0.082944)
       = sqrt(0.646137)
       = 0.8038

Divide each component:

the    = 0.000 / 0.8038 = 0.000
gpu    = 0.288 / 0.8038 = 0.358
trains = 0.693 / 0.8038 = 0.862
model  = 0.288 / 0.8038 = 0.358

d1 normalised: [0.000, 0.358, 0.862, 0.358, 0, 0, 0, 0]. Check: 0.358² + 0.862² + 0.358² = 0.128 + 0.743 + 0.128 = 0.999 ≈ 1. Unit length confirmed (the residual is rounding).

Step 7 — do the same for d2 and compare.

d2 = "the model trains fast"

the:    tf=1 × 0.000 = 0.000
model:  tf=1 × 0.288 = 0.288
trains: tf=1 × 0.693 = 0.693
fast:   tf=1 × 0.693 = 0.693

||d2|| = sqrt(0 + 0.082944 + 0.480249 + 0.480249) = sqrt(1.043442) = 1.0215

normalised: model=0.282, trains=0.678, fast=0.678

Step 8 — cosine similarity between d1 and d2. Because both are L2-normalised, cosine similarity is just the dot product (01-04). Only model and trains are non-zero in both:

cos(d1,d2) = (0.358 × 0.282) + (0.862 × 0.678)
           = 0.10096 + 0.58444
           = 0.685

Step 9 — compare with what raw bag-of-words would have said. Take the raw count rows and cosine them:

d1 raw = [2,1,1,1,0,0,0,0]     ||d1|| = sqrt(4+1+1+1) = sqrt(7)  = 2.6458
d2 raw = [1,0,1,1,1,0,0,0]     ||d2|| = sqrt(1+0+1+1+1) = sqrt(4) = 2.0000

dot = (2×1) + (1×0) + (1×1) + (1×1) + (0×1) = 2 + 0 + 1 + 1 + 0 = 4

cos = 4 / (2.6458 × 2.0000) = 4 / 5.2916 = 0.756

The comparison is the lesson. Raw bag-of-words scores the pair at 0.756; TF-IDF scores it at 0.685. Raw counts rated the pair more similar, and a full quarter of the raw dot product — 2 of 4 — came from the term the, which carries no information at all. TF-IDF removed that contribution and left a similarity based only on the terms that actually distinguish these documents. That is the mechanism, computed rather than asserted.

Step 10 — one more contrast, to see IDF rank terms. Which term best identifies d4 (gpu memory limits the model)?

the:    tf=1 × 0.000 = 0.000   ← in every document; useless
gpu:    tf=1 × 0.288 = 0.288   ← in 3 of 4; weak
model:  tf=1 × 0.288 = 0.288   ← in 3 of 4; weak
memory: tf=1 × 1.386 = 1.386   ← unique to d4; strongest
limits: tf=1 × 1.386 = 1.386   ← unique to d4; strongest

memory and limits are d4's fingerprint. Every term appeared exactly once, so raw counts would have rated all five terms equally. IDF is doing all the work of distinguishing them. This is why TF-IDF remains a serious baseline for retrieval: it automatically identifies which terms make a document findable.

The sharpest demonstration of what n-grams buy is a sentence where unigrams get the answer exactly backwards.

Two documents:

d1: "the service was good"
d2: "the service was not good"

Unigram bag-of-words. Vocabulary: the, service, was, good, not.

Cosine similarity between these rows:

dot   = 1 + 1 + 1 + 1 + 0 = 4
||d1|| = sqrt(4) = 2.0000
||d2|| = sqrt(5) = 2.2361

cos = 4 / (2.0000 × 2.2361) = 4 / 4.4721 = 0.894

89.4% similar — for two sentences with opposite meanings. A sentiment classifier trained on unigrams must learn to treat the isolated presence of not as a negative signal wherever in the document it occurs, which is a weak and error-prone proxy. And if a stop-word list removed not per 02-05, the two rows become identical and the sentences are indistinguishable. That is the concrete argument against stop-word removal for sentiment tasks, expressed as arithmetic.

Now add bigrams — ngram_range = (1,2). New terms from each document:

d1 bigrams: "the service", "service was", "was good"
d2 bigrams: "the service", "service was", "was not", "not good"

Combined vocabulary: 5 unigrams + 5 distinct bigrams = 10 terms.

shared terms: the, service, was, good, the·service, service·was  →  6
dot   = 6
||d1|| = sqrt(7) = 2.6458      (7 non-zero terms)
||d2|| = sqrt(9) = 3.0000      (9 non-zero terms)

cos = 6 / (2.6458 × 3.0000) = 6 / 7.9373 = 0.756

Similarity drops from 0.894 to 0.756. More importantly, d2 now has a feature — not good — that d1 lacks entirely and that means exactly the thing the classifier needs to detect. A linear model can put a large negative coefficient on not good directly. That is the value of n-grams, and it explains why sentiment pipelines classically used unigrams plus bigrams as a default.

Count the cost. Two five-word sentences went from 5 vocabulary terms to 10. At corpus scale that multiplication is brutal: a vocabulary of 50,000 unigrams might yield several hundred thousand bigrams and millions of trigrams, most of them appearing once. Hence:

Decision table — when to reach for sparse lexical methods, and when not to:

The honest summary: sparse lexical methods lost the headline competition and won a permanent niche. They are the correct first move on a new problem, the correct final move when interpretability is a requirement, and half of the correct answer in any serious retrieval system.

These techniques sit under NCA-GENL objectives 2.1 and 2.3 — extracting insights from large datasets and conducting data analysis — and under objectives 1.6 and 1.10 on Python natural-language and ML packages, since CountVectorizer and TfidfVectorizer are named in the blueprint's own text-preprocessing content list. Bag-of-words versus TF-IDF appears explicitly as a must-know contrast in the classical-NLP lesson of the blueprint's Core ML module, alongside n-grams, POS tagging and named-entity recognition.

The reason this is more heavily tested than a "generative AI" exam might suggest is documented in the field calibration: NLP and classical text processing punch above their blueprint weight on this exam. Published candidate reports specifically name lemmatization, WordNet versus word2vec, lexical diversity versus syntactic complexity, and spaCy — all classical items easy to skip when studying LLMs. That is field data rather than official documentation, so treat it as a study-allocation signal, not a promise. But the direction is consistent and the cost of ignoring it is high, because these are cheap points: the concepts are simple and the distinctions are crisp.

The same reports say questions are general-level. You will not be asked to derive the smoothed IDF formula. You will be asked which technique does what, and which to pick in a described scenario.

Question phrasings to expect:

• "What is the main limitation of a bag-of-words representation?" — It discards word order. (A close second, also correct in context: it cannot represent synonymy.)
• "What does TF-IDF do that raw term counts do not?" — Downweights terms common across the corpus and upweights terms rare across it, so distinguishing terms dominate.
• "What is the IDF of a term appearing in every document?" — Zero under the unsmoothed formula. This is the "TF-IDF finds stop words automatically" observation.
• "Which technique lets a bag-of-words model distinguish not good from good?" — N-grams, specifically bigrams.
• "Which scikit-learn class produces TF-IDF features?" — TfidfVectorizer. CountVectorizer produces raw counts.
• "What is an n-gram?" — A contiguous sequence of n tokens.
• "When would TF-IDF outperform dense embeddings?" — Exact-term matching, interpretability requirements, very small labelled datasets, no GPU.
• "Why can TF-IDF not match a query for automobile against a document saying car?" — Each term is an independent orthogonal dimension; sparse lexical representations have no notion of semantic similarity.

Distractor families:

Bag-of-words stores raw term counts; TF-IDF stores counts multiplied by a rarity weight. Same matrix shape, same columns, same sparsity pattern — different numbers in the cells.

The consequence is which terms dominate a similarity comparison. Under bag-of-words, the terms with the largest counts dominate, and in natural language those are function words that carry no topical information. Under TF-IDF, terms that are frequent in this document but rare across the corpus dominate, which is exactly the profile of a term that distinguishes this document from the others.

The four-document worked example in §4 shows it numerically: the contributed 2 of the 4 raw-count dot product between d1 and d2 — half the apparent similarity coming from a term present in every document — and contributed exactly zero after IDF weighting.

Because a term appearing in every document cannot help you tell documents apart, and IDF encodes that mathematically rather than by convention. idf(t) = log(N/df(t)), so when df(t) = N the ratio is 1, the log is 0, and the weight is 0. The term vanishes.

The elegant consequence is that TF-IDF discovers stop words automatically, per corpus, with no list. In a corpus of legal contracts, hereinafter and party might be effectively stop words; in a corpus of medical notes, patient and presented might be. A fixed English stop-word list would catch neither. IDF catches both, because it measures actual document frequency in the actual corpus.

This is why the received advice to always remove stop words before TF-IDF is largely redundant, and occasionally harmful — removal is destructive and breaks n-grams, while IDF weighting achieves most of the same benefit non-destructively.

A contiguous sequence of n items from a text — usually tokens, sometimes characters. Unigram: one token. Bigram: two adjacent tokens. Trigram: three.

N-grams appear in this course in two distinct roles, and it is worth keeping them apart:

1. As features, which is this lesson's use: add bigrams to a bag-of-words vocabulary so the representation can see local adjacency and distinguish not good from good.
2. As a language model, which is the historical antecedent of everything in Module 1: an n-gram language model estimates the probability of the next word from the previous n−1 words, by counting. That is next-token prediction (01-01) done by counting rather than by a neural network — and its fatal weakness, a fixed window with no generalisation across similar contexts, is exactly the weakness embeddings and attention were invented to fix.

The trade-off in both roles is the same: larger n captures more context and produces exponentially more, sparser, less generalisable parameters or features.

Five situations where TF-IDF is the better choice, not a compromise:

1. Exact matching is the requirement. Error codes, part numbers, function names, legal citations, chemical formulas. A dense embedding will return something semantically nearby, which is precisely wrong when the user typed an identifier.
2. Interpretability is a requirement. Each TF-IDF feature is a term, so a linear model's coefficients are directly readable. When a stakeholder or a regulator asks why, "the presence of the word refund contributed +0.8" is an answer. Dense embeddings have none.
3. Very small labelled datasets. With a few hundred examples, TF-IDF plus a regularised linear model is often at or near the ceiling, and it does not overfit the way a fine-tuned transformer will.
4. No GPU, or a hard latency floor. Vectorising and scoring is a sparse dot product. There is no model to load and no forward pass.
5. As the honest baseline. Before claiming a transformer helped, you need a number to beat. Skipping this step is one of the most common ways an LLM project convinces itself of a gain it did not achieve — which is why 07-01 teaches sparse retrieval before dense retrieval.

And the inverse, when embeddings clearly win: users paraphrase, synonymy matters, the corpus is multilingual, or the task requires understanding rather than matching. Most serious retrieval systems end up using both, which is the subject of hybrid search in 07-06.

No, and this is the limitation the rest of the course is built to escape. Every vocabulary term is an independent dimension, orthogonal to every other. car and automobile are two columns with zero relationship: a document containing only car and a query containing only automobile have a cosine similarity of exactly zero on those terms.

The classical patches all have real costs. Manual synonym expansion needs a curated thesaurus and constant maintenance. Query expansion by adding synonyms at search time inflates the query and hurts precision. Latent semantic analysis — factorising the document-term matrix — captures some co-occurrence structure but is not contextual and does not transfer. WordNet-based expansion works for the relationships someone hand-encoded and no others.

None of them solve it properly, because the problem is structural: a representation built from term identity cannot know about meaning it was never told about. The fix is a representation whose dimensions are learned from how words are used, so that words used in similar contexts land in similar places. That is an embedding, and that is where the course goes next.

1. Bag-of-words discards word order entirely. the dog bit the man and the man bit the dog are the same vector. That is its defining limitation and the most-asked fact about it.
2. TF-IDF is a weighting of bag-of-words, not an alternative to it. Same sparse matrix and columns; each cell becomes tf × log(N/df).
3. A term appearing in every document has IDF exactly zero. TF-IDF finds stop words automatically, per corpus, with no list — which makes explicit stop-word removal largely redundant and occasionally harmful.
4. The log in IDF is what keeps hapaxes from dominating. Without it, a term appearing once would outweigh a term appearing in half the corpus by a factor of N.
5. N-grams are an orthogonal axis — a choice about what counts as a term, combinable with any weighting. Bigrams recover the local adjacency that makes not good a feature; the worked example drops d1/d2 similarity from 0.894 to 0.756 and gives the classifier something to grip.
6. The feature space explodes with n. Trigrams without min_df pruning produce millions of single-occurrence features and no gain. Prune, and validate that the higher n actually helps.
7. Fit the vectorizer on training data only, and serialise it with the model. Fitting on everything leaks document frequencies from held-out data; refitting at inference silently repoints the model's coefficients.
8. Sparse methods win on exact match, interpretability, small data and CPU-only budgets. They are the honest baseline, and skipping them is how projects claim gains they did not make.
9. Sparse methods cannot represent synonymy. Every term is an orthogonal dimension, so car and automobile are unrelated. No amount of weighting or n-gram engineering fixes this, because the problem is structural.
10. Know the class names: CountVectorizer for raw counts, TfidfVectorizer for weighted features, ngram_range for the n-gram axis. The blueprint names them explicitly.

This module has taken you from "a model cannot read text" to a complete, working, sparse lexical representation with weighting and local order — and it ends by naming the one thing that representation structurally cannot do. car and automobile are two orthogonal columns. physician and doctor are two orthogonal columns. Every paraphrase your users will ever type is invisible, and no amount of IDF weighting or n-gram engineering changes that, because the representation is built from term identity and term identity does not carry meaning.

The fix is to stop building dimensions from terms and start learning them from usage: a representation where each document or token is a short dense vector of a few hundred learned floats, positioned so that things used in similar contexts land near each other. Then car and automobile are close by construction, and similarity becomes a geometric question your 01-04 dot product answers directly.

Next: 03-01 opens Module 3 by defining exactly what a text embedding is — a learned dense vector, not a count — how the learning positions related words together, what the dimensions do and do not mean, and why this is the representation every RAG system, every semantic search index and every transformer's first layer is built on. The token IDs from 02-01 have been waiting for their vectors since the first page of this module; that is where they get them.