The Vocabulary-Size Tradeoff and Why It Caps Perplexity Comparisons

Identity statement: vocabulary size is the number of distinct entries a tokenizer's fixed vocabulary contains, and it is a tuning decision that trades sequence length against parameter count and training-signal density — never a parameter you can simply increase for a strictly better outcome.

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states the framing directly: vocabulary size is "a tuning decision, not 'bigger is better.'" The two directions of the tradeoff are named explicitly and are worth holding as a matched pair rather than memorizing only one side. A larger vocabulary produces shorter token sequences — fewer tokens per document, because more of a document's content matches a single vocabulary entry rather than needing to be spelled out from several smaller pieces — but it comes at the cost of larger embedding and output-softmax tables, since more parameters are needed to represent more vocabulary entries. A smaller vocabulary runs the tradeoff in reverse: longer sequences, because more of any given document has to be spelled out from smaller fragments, but a correspondingly smaller parameter footprint in exactly the two places vocabulary size actually touches.

Neither direction is categorically better, which is exactly why [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) closes this subsection with "tune to the task, languages, and hardware budget" rather than naming a single correct vocabulary size. A team serving a memory-constrained edge device weighs this tradeoff differently than a team training a frontier-scale model with abundant GPU memory but a hard latency budget on long documents — the tradeoff's shape is fixed, but where along it a given deployment should sit is not.

L1 — Intuition

A vocabulary entry, once trained, represents a specific chunk of text — a whole common word, a fragment, sometimes an entire multi-word phrase in a large enough vocabulary. The more distinct chunks a vocabulary has room to represent as single entries, the fewer chunks any given document needs to be broken into, because more of that document's actual content matches something the vocabulary already has a dedicated entry for.

L2 — Mechanism

Sequence length reduction is the direct, mechanical consequence of vocabulary growth: as a vocabulary gets larger, it accumulates entries for progressively longer and less common fragments — first the most frequent whole words, then less frequent words, then multi-word or specialized fragments in a sufficiently large vocabulary — so a fixed piece of text increasingly gets covered by fewer, longer matches rather than many short ones. This is a real, valuable benefit independent of anything else: fewer tokens per document means more actual content fits inside a fixed context window, fewer tokens are billed per request in a usage-metered API, and — because self-attention's compute cost grows with the square of sequence length — a shorter sequence produces a disproportionately larger compute savings than the raw token-count reduction alone would suggest.

L3 — The exam-relevant edge case: the benefit is real but it is only one side of the ledger

The trap worth naming here is treating "shorter sequences" as the entire story and stopping the analysis there, which is precisely the shape of the false claim this lesson exists to correct. Shorter sequences are a genuine, unambiguous benefit of a larger vocabulary — nothing about section 3's cost undoes that benefit or makes it illusory. What section 3 does is show that the benefit is not free: it is bought with a specific, quantifiable cost elsewhere in the model, and a correct answer to "is a bigger vocabulary better" has to weigh both sides rather than stopping after describing only the win.

L1 — Intuition

Every entry in a tokenizer's vocabulary needs its own row in the model's embedding matrix — the lookup table that converts a token's integer ID into the vector representation the rest of the model actually computes with — and, in most architectures, a corresponding column in the output layer that produces a probability over the entire vocabulary at every generation step. More vocabulary entries means more rows and more columns, and those rows and columns are real, trainable parameters that occupy real GPU memory.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names this cost directly: "larger embedding and output-softmax tables (more parameters, more memory)." The relationship is linear and mechanically simple: if a model's hidden dimension is d, the embedding matrix has shape vocabulary-size times d, so its parameter count scales linearly with vocabulary size for a fixed d. The output projection — the layer that maps the model's final hidden state to a score for every vocabulary entry before the softmax turns those scores into a probability distribution — has the same shape in most architectures, whether it is a separate matrix or (in architectures that tie weights) the same matrix reused. Either way, the total parameter cost that scales with vocabulary size is proportional to vocabulary size times hidden dimension, counted once or twice depending on whether the two layers share weights.

That "output-softmax" half of the cost carries a second, less obvious consequence beyond raw parameter count: computing a probability distribution over the entire vocabulary at every single decoding step is itself a computation whose cost scales with vocabulary size, independent of the parameter-storage cost. A larger vocabulary does not just cost more memory to store — it costs more compute, at every generated token, to produce the final probability distribution the model samples from.

L3 — The exam-relevant edge case: the cost is not a rounding error at model scale

The reason this cost is worth taking seriously rather than treating as a minor footnote is that vocabulary-facing parameters are not a small slice of a modern model's total parameter budget, particularly as hidden dimensions have grown alongside vocabularies across model generations. A vocabulary increase that looks modest in relative terms — doubling from, say, 50,000 to 100,000 entries — doubles the exact parameter cost computed in section 4's arithmetic, and at large hidden dimensions that doubling is measured in hundreds of millions of parameters, not a handful. Treating vocabulary-facing parameter growth as negligible compared to the rest of a large model's parameter count is exactly the kind of assumption the worked example below is built to correct with real numbers.

Take a model with hidden dimension d = 4,096, a realistic mid-to-large size for a modern LLM, and compute the embedding-table parameter cost — vocabulary size times hidden dimension — at three different vocabulary sizes. Treat the output projection as tied to the embedding matrix for this illustration, so the same arithmetic applies to both without double-counting; an untied output layer would add this same figure again.

V = 32,000 (a common smaller-vocabulary choice):
  32,000 x 4,096 = 131,072,000 params  ~  131 M

V = 50,000 (a common mid-range choice):
  50,000 x 4,096 = 204,800,000 params  ~  205 M

V = 128,000 (a large, modern multilingual-leaning choice):
  128,000 x 4,096 = 524,288,000 params  ~  524 M

Going from a 32,000-entry vocabulary to a 128,000-entry vocabulary — a 4x increase in vocabulary size — produces almost exactly a 4x increase in embedding-table parameters, from about 131 million to about 524 million, which is the linear relationship section 3 described made concrete. That roughly 393-million-parameter difference is not trivial at any model scale; it is comparable to the entire parameter count of some historically significant smaller language models, spent entirely on vocabulary lookup rather than on the layers that do the model's actual reasoning.

Now put the other side of the ledger next to it. Suppose a fixed 10,000-token document encodes, on average, to roughly 2,500 tokens under the 32,000-entry vocabulary but only roughly 1,850 tokens under the 128,000-entry vocabulary — illustrative numbers for this constructed example, not measured from any specific released tokenizer, but consistent with the general direction a larger vocabulary produces.

32,000-entry vocabulary: ~2,500 tokens for the same document
128,000-entry vocabulary: ~1,850 tokens for the same document

Sequence-length reduction: ~26% fewer tokens
Attention compute is roughly quadratic in sequence length, so a ~26%
shorter sequence corresponds to roughly (1 - 0.74^2) ~ 45% less attention
compute for that document at the larger vocabulary size.

The two numbers do not resolve to a single "better" answer — that is the entire point of calling this a tradeoff rather than a decision with one correct answer. [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) names "task, languages, and hardware budget" as the deciding factors for exactly this kind of choice. A ~393-million-parameter cost buys a ~26% shorter sequence and a substantially larger attention-compute saving on long documents; whether that trade is worth making depends on whether the deployment's actual constraint is parameter memory or per-request compute on long inputs.

L1 — Intuition

Perplexity is a per-token measure of how surprised a language model is by held-out text, and per-token measures are only meaningful when "a token" means the same thing on both sides of a comparison. Two models that tokenize the identical piece of text into different numbers of tokens are not being scored on the same underlying units, even when they are evaluated on the exact same raw text.

L2 — Mechanism

[GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states this downstream effect directly: "tokenization directly affects perplexity, so perplexity scores are only comparable across models that share the same tokenizer." The causal chain runs exactly through the vocabulary-size material this lesson has already covered: a different vocabulary size (or a different merge rule, per M3-03) changes how many tokens a given piece of text encodes to, which changes the denominator perplexity's per-token averaging divides by, which changes the resulting number in a way that has nothing to do with which model actually predicts held-out text better.

Concretely, perplexity is computed from a model's average per-token negative log-likelihood over a held-out sequence, then exponentiated. If Model A's tokenizer encodes a passage into 500 tokens and Model B's tokenizer — with a different vocabulary — encodes the exact same passage into 650 tokens, the two models are computing an average over a different number of terms, covering different-sized chunks of text per term. A model whose tokenizer happens to produce longer average tokens (fewer, chunkier pieces per document) is, all else equal, predicting a "harder" per-step task in one sense (each token carries more information to guess correctly) and an "easier" one in another (fewer decision points per document) — and these effects do not cancel out cleanly across different vocabularies, which is exactly why the two models' perplexity numbers are not directly comparable even on identical underlying text.

L3 — The exam-relevant edge case: same tokenizer is a hard requirement, not a nice-to-have

The detail that separates a correct understanding of this rule from an incomplete one is that "same tokenizer" is not a soft preference that makes comparisons somewhat more reliable — it is close to a hard requirement for the comparison to mean anything at all. Two models with even modestly different vocabulary sizes, trained by the same merge-rule family, on similar data, can still produce meaningfully different token counts on the same evaluation text, which is enough to invalidate a head-to-head perplexity comparison between them. M6-01 covers perplexity's own definition and where it does and does not apply in full; this lesson's job is narrower and more specific: establishing exactly why the tokenizer, and by extension the vocabulary-size decision this lesson covers, is the mechanism that caps whether any two perplexity numbers can be honestly placed side by side in the first place.

⭐ THE EARNED INSIGHT: > Perplexity is not comparable across tokenizers not because of some arbitrary evaluation-protocol rule, but because tokenizer choice changes the very units perplexity is averaged over — a shorter or longer average token, produced by a different vocabulary size or merge rule, changes what a "per-token" score even means, and two numbers computed over different units are not the same measurement no matter how similar the reported figures look. The fix is not a correction factor; it is refusing to compare at all unless both models share the exact same tokenizer.

Take a constructed, illustrative scenario built to make the comparability failure concrete, not a report of any real benchmark result. Two language models are evaluated on the identical held-out passage of natural-language text.

Model A (vocabulary size 32,000):
  Passage encodes to 1,000 tokens
  Reported perplexity: 18.4

Model B (vocabulary size 128,000):
  Same passage encodes to 740 tokens
  Reported perplexity: 22.1

Read naively, Model A's lower perplexity number looks like the better result — lower is better, after all, and 18.4 is lower than 22.1. But the two models are not scoring the same 1,000 decision points; Model B is being scored over only 740 tokens for the identical text, because its larger vocabulary encoded the same content into fewer, larger chunks. Model B's per-token task is, in a real sense, a harder one on average — each of its tokens is a larger, more information-dense unit to predict correctly than Model A's smaller, more granular tokens are — and a fair comparison would need to account for that difference in what a "token" represents before concluding anything about which model actually understands the passage better.

What would actually need to hold for these two numbers to be comparable:
  Both models tokenize the SAME passage into the SAME number of tokens,
  using the SAME vocabulary — which by definition means using the
  same tokenizer, not merely tokenizers of a similar general design.

This is a constructed illustration — the specific 18.4 and 22.1 figures, and the 1,000-versus-740 token counts, are chosen to be plausible rather than measured from any real evaluation run — but the mechanism it demonstrates is exactly the rule [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) states: perplexity numbers computed under different tokenizers are not measuring the same thing, and reporting them side by side as though a smaller number always means a better model is a category error, not a close call decided by a small margin.

The framing so far — sequence length against parameter count — is the two-item version of the tradeoff the source material states directly, and it is sufficient for the exam-level reasoning this domain asks for. It is worth naming a third factor that compounds the parameter cost side, because it explains why the embedding-table cost in section 4 is not merely a storage inconvenience but an actual training-quality concern: a fixed training corpus, split across more vocabulary entries, gives each individual rare entry fewer occurrences to learn from. A 50,000-entry vocabulary trained on a corpus of a given size sees each of its rare entries some number of times; the same corpus split across a 128,000-entry vocabulary spreads that same total volume of training text across many more distinct entries, so the entries near the rare end of the frequency distribution are seen correspondingly less often.

The practical consequence is that very large vocabularies tend to accumulate a tail of entries whose embeddings are poorly estimated — not because anything went wrong in training, but because those specific rows simply never saw enough occurrences in context to be trained as thoroughly as a frequent word's embedding was. This is a second, independent reason a larger vocabulary is not a costless choice, layered on top of the direct parameter-count cost section 3 already covers: it is not just that a larger vocabulary needs more parameters, it is that some of those parameters end up less reliably trained than they would be in a smaller vocabulary drawing on the same fixed corpus. Treat this as an additive consideration to the two-factor tradeoff the source material states directly, not a replacement for it — the exam's own framing is the sequence-length-versus-parameter-count tradeoff, and this training-signal-density point is the reasoning that explains why the parameter-count side of that tradeoff is not merely a memory-budget line item.

Objective 3.4 covers this material directly, and the source material's own trap statement is explicit: "'A bigger vocabulary always improves accuracy.' False — it's a sequence-length vs model-size tradeoff." [GROUND TRUTH] (Sources/ncp-genl/domain-3-data-preparation.md) frames the professional-level expectation for this domain as reasoning about tradeoffs rather than reciting definitions, and vocabulary size is the clearest single example of a tradeoff this domain tests, because both directions of the tradeoff — the benefit and the cost — are individually true statements that only become a wrong answer when one is presented without the other.

Expect this material in a small number of recurring shapes: a direct claim-evaluation item stating "a larger vocabulary always improves model accuracy" or a similarly absolute claim, where the correct response identifies the missing cost side of the tradeoff; a scenario naming a specific constraint — a memory-limited deployment target, a long-document use case, a multilingual corpus — and asking which vocabulary-size direction fits it, testing whether "tune to task, languages, and hardware budget" is applied correctly rather than defaulting to one size regardless of the stated constraint; and a cross-domain item connecting vocabulary size to perplexity comparability, testing whether the tokenizer-dependence chain from section 5 is understood as a direct consequence of vocabulary-size and merge-rule choices, not an independent fact about perplexity in isolation.

What the distractors typically look like

The standing traps here are: presenting a bigger vocabulary as a strictly better choice with no accompanying cost, which is the domain's own named false claim; describing the embedding/softmax cost of a larger vocabulary as negligible or a rounding error, when the parameter arithmetic in section 4 shows it scales linearly and can reach hundreds of millions of parameters at realistic hidden dimensions; and comparing two models' perplexity scores directly without checking whether they share a tokenizer, treating the comparison as valid by default rather than as something that requires verifying a specific precondition first.

Because the parameter and compute cost of a larger vocabulary is real and continues to scale even after "fits in memory" is satisfied — a vocabulary that technically fits still spends real training-time compute on a wider output softmax at every single generated token, and real embedding-table parameters that could otherwise be spent on the model's actual reasoning layers. Beyond raw memory fit, M3-05's exploratory-analysis material connects to a related cost this lesson has not yet covered directly: spreading a fixed training corpus across more vocabulary entries means each individual rare entry is seen fewer times during training, thinning the signal available to train that entry's embedding well. A vocabulary chosen purely to maximize sequence-length savings, without weighing these accumulating costs, is optimizing one side of the tradeoff while ignoring the other — exactly the failure mode this lesson's material is built to prevent.

No, and this is the precise shape of the false claim the source material names. A larger vocabulary changes how text is packaged into tokens and how many parameters the model spends on vocabulary lookup — it does not, on its own, change the model's capacity to reason, and it is not a substitute for more training data, more training compute, or a better architecture. A model with a larger vocabulary can encode the same content in fewer tokens, which can translate into real practical benefits (more content per context window, less compute on long documents), but "shorter sequences" and "smarter model" are not the same claim, and conflating them is exactly how "bigger vocabulary always improves accuracy" gets treated as though it were self-evidently true rather than the specific overstatement this lesson's material flags directly.

Not on its own. Sharing a merge-rule family (both BPE, for instance) is necessary but not sufficient for a fair perplexity comparison, because vocabulary size is an independent axis from merge rule, and section 5 established that it is specifically vocabulary size — how many entries the tokenizer has, which changes token counts for the same text — that drives the comparability problem, not just which merge rule built the vocabulary. Two BPE-trained tokenizers at 32,000 and 128,000 entries will still encode the same passage into meaningfully different numbers of tokens, which is exactly the scenario section 6's worked example walks through. "Same tokenizer" means the identical trained vocabulary — same size, same merge history, same resulting entries — not merely the same family of training algorithm. Anything short of that identical vocabulary reintroduces the token-count mismatch that makes a side-by-side perplexity comparison invalid.

• Vocabulary size is a tuning decision, never a free win. A larger vocabulary shortens sequences but linearly enlarges the embedding and output-softmax tables; a smaller vocabulary reverses both effects.
• The embedding-table cost is linear and real — vocabulary size times hidden dimension, which reaches hundreds of millions of parameters at realistic model scales, not a rounding error.
• "Bigger vocabulary always improves accuracy" is the specific false claim this material names. The correct framing is a sequence-length-versus-model-size tradeoff, tuned to task, languages, and hardware budget.
• Perplexity is only comparable across models sharing the exact same tokenizer, because tokenizer choice — vocabulary size and merge rule alike — changes the per-token unit perplexity is averaged over.
• Vocabulary size and merge rule (M3-03) are two independent tokenizer-training decisions, not one combined choice, and both feed into the same downstream perplexity-comparability constraint.

The vocabulary-size tradeoff tells you what a tokenizer decision costs and buys once training has already committed to it — but it says nothing about how you would have noticed, before committing, whether your underlying dataset actually needed a different tokenizer, a different split discipline, or a different label balance in the first place.

Next: M3-05 closes this module with exactly that earlier-stage question — the five-step exploratory data analysis checklist that catches imbalance, leakage, and truncation risk before fine-tuning ever begins, rather than after a tokenizer or vocabulary-size decision has already been made and locked in.