Perplexity in LLM Evaluation: What It Measures and Where It Does Not Apply

Perplexity (PPL) is the exponential of a causal language model's average negative log-likelihood (NLL) per token on a piece of text — equivalently, the exponentiation of its cross-entropy against that text. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) Formally, for a token sequence w_1, ..., w_N scored by a model that produces a conditional probability for each token given everything before it:

NLL_i  = -log P(w_i | w_1, ..., w_{i-1})
H      = (1/N) * sum(NLL_i)          # mean per-token NLL = cross-entropy
PPL    = exp(H)

Two readings make the number legible. The information-theoretic reading treats H as the average number of nats of "surprise" the model experienced per token; perplexity converts that log-space quantity back into a linear scale. The branching-factor reading is more useful for exam purposes: perplexity is the size of the uniform choice set that would have produced the same average surprise. A perplexity of 6 means the model was, on average, as uncertain at each position as if it had been picking uniformly among 6 equally likely next tokens. A perplexity of 1 is a model that assigned probability 1 to every actual token — perfect, and on any real text a sign of memorization, not skill.

Lower is better, without exception, within a valid comparison. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) A model that spreads less surprise across the tokens it is shown is a model whose learned distribution more closely tracks the true distribution of the text. This one-directional reading — smaller number, better fit — is the first half of this domain's most tested trap-pair; the second half is that the direction does not save you if the comparison itself is invalid, which sections 2 and 3 build out in full.

Why perplexity needs a token-by-token probability to exist at all

The formula above is not an arbitrary convention; it is a direct restatement of the chain rule of probability. A causal, decoder-only model factors the probability of an entire sequence as the product of each token's conditional probability given only the tokens before it: P(w_1, ..., w_N) = P(w_1) * P(w_2|w_1) * P(w_3|w_1,w_2) * .... Taking the negative log of that product turns it into a sum, and dividing by N turns the sum into an average — which is exactly H above. Every step of that derivation depends on the model actually producing a left-to-right conditional probability at every position. That dependency is the hinge the entire next section swings on.

L1 — Intuition: perplexity needs a sequence probability, and a masked model never produces one

A causal decoder generates one token at a time, left to right, and each of those steps is a genuine conditional probability over "what comes next given everything so far." Multiply those conditionals together (in log space, sum them) and you get a coherent probability for the whole sequence — which is the only quantity perplexity is defined to exponentiate. A masked language model like BERT does not work this way. Its training objective hides a subset of tokens and asks the model to reconstruct each hidden token using context from both directions — the words before it and the words after it. There is no ordering in which you can multiply those per-position guesses together and get a valid joint probability for the sentence, because each guess was conditioned on information (later tokens) that a genuine left-to-right factorization is not allowed to use. Perplexity has no home to live in for a model built this way.

L2 — Mechanism: what breaks specifically, and what a masked model reports instead

Concretely, a masked model's forward pass over an unmasked sentence does not naturally produce anything of the shape P(w_i | w_1 ... w_{i-1}). What it produces, per masked position, is P(w_i | context with w_i removed, both directions) — a fundamentally different conditional, one that peeks at the future by construction. You cannot chain these together into P(w_1, ..., w_N) the way the causal chain rule requires, because the chain rule's multiplication step assumes each factor only used information available up to that point in the ordering, and a bidirectional guess violates that assumption at every single position, not just at the edges. Researchers have defined a workaround called pseudo-perplexity — mask each token one at a time, holding all the others fixed, score that single masked position, and average the results across the sequence — but this is an engineered substitute that produces a legitimately different number, not a rescued version of the same metric. A pseudo-perplexity and a causal-model perplexity are not on the same scale and were never meant to be compared to one another.

L3 — The trap this generates, and why it recurs so reliably

⚠️ UNVERIFIED this specific framing is my own construction rather than a source-cited fact, but the shape of the trap is: an exam item describes a masked encoder model (frequently naming BERT or a BERT-family model explicitly) and then asks which metric to use for "evaluating how well it predicts text," with perplexity offered as one option. The correct behavior is to recognize the architecture first — encoder-only, bidirectional, MLM-trained — and rule perplexity out on structural grounds before even considering whether the rest of the item's framing makes sense. M1-03's architecture-family matching (encoder → MLM/bidirectional, decoder → CLM/causal) is the exact prerequisite this rests on: if you cannot recognize that a described model is encoder-only, you cannot recognize that perplexity is off the table for it. The distractor is effective precisely because perplexity is such a familiar, generically "evaluation-sounding" term that test-takers reach for it as a default answer to any "how good is this language model" question, when its applicability is gated by architecture, not by general usefulness.

L1 — Intuition: perplexity is a per-token average, and tokens are not a fixed unit

Perplexity divides total surprise by the number of tokens the text was split into. That denominator is not a property of the text — it is a property of the tokenizer. Two tokenizers can split the identical string into different numbers of pieces, and because perplexity is an average over however many pieces resulted, changing the tokenizer changes the denominator of a ratio whose numerator (the model's actual beliefs about the string) may not have moved at all.

L2 — Mechanism: the same joint probability, two different tokenizers, two different perplexities

Suppose a model's genuine belief about a string, expressed as the joint probability it assigns to that exact string, does not change. If tokenizer A splits it into 5 tokens and tokenizer B splits the identical string into 7 tokens (because B has a smaller vocabulary and therefore produces more, shorter subword pieces), the same total surprise gets divided by 5 in one case and by 7 in the other. Averaging identical total surprise over more pieces produces a smaller per-token average, which exponentiates into a smaller perplexity — a model can look "better" purely because its tokenizer produces more, shorter tokens, with the model's actual predictive quality held perfectly constant. Scores are only comparable under the same tokenizer. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) This is not a minor caveat; it is the reason perplexity is unsuitable as a cross-vendor leaderboard metric the moment two systems use different tokenizers, which in practice is almost always the case once you compare across model families rather than checkpoints of the same model.

L3 — Sliding-window scoring: getting an honest number out of a fixed-context model

A related, purely-computational wrinkle affects any model with a bounded context window being scored on text longer than that window. If you chop a long document into disjoint, non-overlapping chunks and score each chunk independently, every chunk's first several tokens are scored with little or no preceding context, which inflates their measured surprise and therefore inflates the whole document's average perplexity relative to what the model would show with full context. The fix is strided (sliding-window) scoring: move the scoring window forward by a stride smaller than the window's full length, so that most tokens are scored with a long run of preceding context behind them, at the computational cost of re-processing the overlap. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) The magnitude of the effect is not small: GPT-2 large's measured perplexity on WikiText-2 improves from roughly 19.4 with no overlap (disjoint chunking) to roughly 16.4 with a stride of 512 tokens — the same weights, the same test set, a materially different reported number purely from the scoring procedure. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) Anyone reporting a perplexity figure without stating the stride used has left out a variable that measurably moves the number, which is exactly why a rigorous evaluation report states its stride the same way it states its tokenizer.

Constructed scenario. The probabilities below are invented for arithmetic clarity; they are not measurements from any real checkpoint.

A causal model scores the four-token continuation ["Latency", " dropped", " sharply", " today"] and assigns these probabilities to the tokens that actually appeared:

Token        P(actual token | context)     NLL = -ln(P)
"Latency"           0.20                     1.6094
" dropped"          0.08                     2.5257
" sharply"          0.04                     3.2189
" today"            0.30                     1.2040

Step 1 — sum the NLLs:
  1.6094 + 2.5257 + 3.2189 + 1.2040 = 8.5580

Step 2 — divide by token count (N = 4):
  H = 8.5580 / 4 = 2.1395 nats/token

Step 3 — exponentiate:
  PPL = exp(2.1395) = 8.494

This model's perplexity on this four-token span is about 8.49 — as uncertain, on average, as if it were choosing uniformly among roughly 8 to 9 equally likely continuations at each step.

Now hold the model's underlying beliefs fixed and change only the tokenizer. Suppose a second tokenizer splits " sharply" into two subword pieces, " sharp" and "ly", with the model (re-scored under this tokenizer's vocabulary) assigning them probabilities 0.22 and 0.55 respectively:

Re-tokenized sequence is now 5 tokens:
"Latency" (0.20) | " dropped" (0.08) | " sharp" (0.22) | "ly" (0.55) | " today" (0.30)

NLLs:  1.6094, 2.5257, 1.5141, 0.5978, 1.2040
Sum:   1.6094 + 2.5257 + 1.5141 + 0.5978 + 1.2040 = 7.4510

H   = 7.4510 / 5 = 1.4902
PPL = exp(1.4902) = 4.437

Perplexity fell from 8.49 to 4.44 — nearly halved — purely by re-tokenizing one word into two pieces, with no change to any underlying capability the model has. The joint probability the model assigns to the string barely moved (the original path multiplies to 0.20 * 0.08 * 0.04 * 0.30 = 1.92e-4; the retokenized path multiplies to 0.20 * 0.08 * 0.22 * 0.55 * 0.30 = 5.808e-4, both minuscule and in the same order of magnitude once you account for the extra split), but the denominator that perplexity divides by grew from 4 to 5, and that alone drove most of the visible improvement. This is the single most important operational fact about perplexity for a deployment engineer comparing two candidate models: if their tokenizers differ, the perplexity numbers are not directly comparable, full stop, regardless of which model is actually the better fit to the data.

Constructed scenario, illustrating the mechanism behind the GPT-2/WikiText-2 figures cited above without reusing their specific numbers. Suppose a model has a 4-token context window (deliberately tiny, for arithmetic clarity) and is scored on an 8-token document: ["A","B","C","D","E","F","G","H"], where each letter stands for one token.

Disjoint chunking splits the document into two non-overlapping 4-token windows: [A,B,C,D] and [E,F,G,H]. Every token in the second window is scored with access only to whatever came before it within that window — E is scored with zero preceding context, as if it were the first word of a new document, even though D genuinely preceded it. Suppose the model assigns these probabilities to the tokens it is asked to predict (position 1 of each window has no context, so scoring conventionally starts at position 2 within a window in a strict windowed scheme, but for illustration assume every position is scored and a context-starved position gets a materially worse probability):

Window 1 [A,B,C,D]:  NLLs = 2.0, 1.5, 1.2, 1.0   → sum = 5.7
Window 2 [E,F,G,H]:  NLLs = 2.3 (starved), 1.4, 1.1, 0.9   → sum = 5.7

Total NLL = 11.4,  N = 8,  H = 11.4 / 8 = 1.425
PPL (disjoint) = exp(1.425) = 4.158

Strided scoring with stride 2 instead slides the 4-token window forward by 2 tokens at a time: [A,B,C,D], then [C,D,E,F], then [E,F,G,H], and at each step after the first only the new, rightmost tokens are counted toward the total (the leftmost tokens of each later window were already scored with full context in an earlier window, so they are not re-counted). E is now scored inside the window [C,D,E,F], with C and D as real preceding context rather than none at all, and its probability improves accordingly — say from 2.3 down to 1.6, with F, G, and H improving by similar amounts for the same reason:

Window 1 [A,B,C,D], all 4 tokens counted (first window, nothing to skip):
  A=2.0, B=1.5, C=1.2, D=1.0                    → sum = 5.7

Window 2 [C,D,E,F], only new tokens E,F counted (C,D already counted above):
  E=1.6, F=1.3                                  → sum = 2.9

Window 3 [E,F,G,H], only new tokens G,H counted (E,F already counted above):
  G=1.05, H=0.9                                 → sum = 1.95

Total NLL = 5.7 + 2.9 + 1.95 = 10.55,  N = 8,  H = 10.55 / 8 = 1.319
PPL (strided) = exp(1.319) = 3.739

Perplexity fell from 4.158 (disjoint) to 3.739 (strided) — about a 10% drop — on the identical model and the identical document, purely because the strided scheme gave the second-half tokens real preceding context instead of scoring them cold. This is the same direction and the same mechanism behind the GPT-2 large/WikiText-2 figures cited earlier (roughly 19.4 down to roughly 16.4 at stride 512), just at a scale small enough to trace by hand. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) Neither number is "wrong" — both are legitimate perplexity computations under their stated procedure — but they are not the same measurement, and reporting one without saying which procedure produced it hides a 10%-or-larger swing behind a single unqualified number.

The row worth re-reading is the last one: a fluent, low-perplexity continuation can still be completely unfaithful to a retrieved context, because perplexity is a property of the model's own distribution over language, not a check against any external source of truth. That distinction becomes the entire subject of M6-05.

Perplexity's cheapness — no labels, no reference answers, just a model and some held-out text — is exactly why it belongs earlier in a model's lifecycle than the metrics in this module's later lessons. A team comparing a quantized checkpoint against its full-precision original, or a distilled student against its teacher, can compute perplexity on a frozen held-out corpus in minutes and get an immediate signal about whether the compression damaged the model's core language-modeling ability. That is a legitimate and common use, and it is why perplexity shows up naturally alongside the quantization and distillation techniques covered earlier in this cert's Model Optimization domain. What it cannot certify is anything about the compressed model's downstream task performance, instruction-following, or safety behavior — a quantized model can hold its pretraining-corpus perplexity almost perfectly flat while still degrading noticeably on a held-out instruction-following benchmark, because perplexity was never designed to see that kind of regression. Treat a stable perplexity after compression as a necessary sanity check, not sufficient evidence that nothing important broke.

The rule that survives all nine rows: perplexity is a training-side and monitoring-side instrument, evaluated against itself over time or against a same-family, same-tokenizer peer — never a general-purpose product-quality score.

Evaluation is a 7% domain on the NCP-GENL blueprint, objectives 6.1 through 6.4, and it is described in its own source material as small but dense with trap-pair questions — perplexity's direction and applicability are named explicitly as one of those pairs, alongside BLEU-versus-ROUGE. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) Because the domain is small, each objective tends to be tested by one or two sharply-written items rather than broad survey coverage, which raises the value of getting the boundary conditions — not just the definition — exactly right.

How the question tends to be phrased

Expect a short scenario naming a specific model architecture and asking which evaluation approach fits, with perplexity offered as a plausible-sounding but wrong answer whenever the named architecture is encoder-only or masked. A second common shape states two systems' perplexity scores and asks what can be concluded, where the correct answer hinges on whether the two systems share a tokenizer — if the item does not state that they do, the safe reading is that the comparison is not established as valid. A third shape asks directly which direction is better ("lower" is always the keyed answer within a valid comparison), sometimes paired with a distractor that reverses BLEU/ROUGE's own precision/recall orientation in the same item to test whether a candidate confuses two entirely separate trap-pairs.

What the distractors typically look like

The domain's standing distractor is offering perplexity as the evaluation method for a masked or encoder-only model. [GROUND TRUTH] (Sources/ncp-genl/domain-6-evaluation.md) A second recurring distractor states or implies that higher perplexity is better, inverting the one fact that should never be inverted. A third, subtler distractor presents two perplexity numbers from different tokenizers as though the comparison were automatically valid, omitting the tokenizer detail entirely and expecting the reader to notice its absence rather than its presence — a "trap by omission" rather than a false statement.

Not in its ordinary form. A masked model's training objective does not produce the left-to-right sequence probability perplexity requires, so plain perplexity has no valid definition for it. What exists as a substitute is pseudo-perplexity, computed by masking one token at a time, holding the rest of the sequence fixed, scoring that single position's reconstruction probability, and averaging across all positions in the sequence. Pseudo-perplexity is a real, usable diagnostic for masked models, but it is a different quantity on a different scale from causal-model perplexity, and treating the two numbers as comparable — for instance, claiming a BERT variant has "lower perplexity" than a GPT variant using pseudo-perplexity on one side and true perplexity on the other — repeats the exact tokenizer-style comparison error this lesson has been warning against, just at the level of the whole metric rather than at the level of the tokenizer.

Because perplexity rewards predicting the single most statistically likely continuation of ordinary text, and instruction-following, refusal, and formatting behaviors are frequently less likely under a model's raw pretraining distribution than the generic continuation would have been. A model fine-tuned to follow instructions, decline unsafe requests, or produce structured output is being steered away from what plain next-token statistics would have predicted, and that steering shows up as a perplexity increase on generic held-out text even as the model becomes measurably more useful in practice. This is a real property of the metric, not a contradiction of "lower is better" — the "lower is better" rule holds within one fixed evaluation setup, but the evaluation setup itself (a corpus of generic web text) stopped being the thing anyone actually cared about the moment the model was aligned toward instruction-following instead of raw text prediction.

• Perplexity is exp(average per-token negative log-likelihood), computed from a causal model's left-to-right conditional probabilities.
• Lower is better — but only inside a valid comparison: same tokenizer, same held-out corpus, ideally the same stride policy.
• Perplexity is undefined for masked language models like BERT because their bidirectional objective produces no left-to-right sequence probability to exponentiate; pseudo-perplexity is a different, non-comparable substitute.
• Sliding-window scoring materially changes the reported number for fixed-context models on long text — GPT-2 large's WikiText-2 perplexity moves from roughly 19.4 to roughly 16.4 between no-overlap and stride-512 scoring.
• Perplexity needs no labels, which makes it cheap and useful for compression sanity checks, and simultaneously makes it blind to correctness, helpfulness, and groundedness.
• A model can look "better" purely from a smaller-vocabulary tokenizer producing more, shorter tokens — the worked example nearly halved perplexity with no change in the model's actual beliefs.

⭐ THE EARNED INSIGHT

Perplexity answers exactly one question — how well did the model's own probabilities match this text under this tokenizer — and every trap built around it is really the same trap wearing a different architecture or a different vocabulary: someone treating a number that is only valid inside narrow, stated conditions as though it were valid everywhere those conditions go unchecked.

Perplexity's boundary is architectural and tokenizer-based; the next question is what happens when a metric's boundary is about orientation instead — precision versus recall, the two axes generation metrics get built on. M6-02 covers BLEU, ROUGE, and METEOR, and its own standing trap-pair is exactly that: which of the two is precision-oriented and which is recall-oriented, and what METEOR adds once you need more than raw n-gram overlap.