Perplexity Explained: What It Measures, What It Misses, and Why Lower Is Not Always Better

Perplexity is the exponentiated average per-token cross-entropy loss of a language model on a text. It is a measure of predictive uncertainty, not of output quality. Two equivalent readings:

• Information-theoretic reading. Cross-entropy measures, in bits or nats, how many units of information you need to encode the text under the model's probability distribution. Perplexity is that quantity converted from log space back into "number of choices". A model that assigns high probability to the tokens that actually occur has low cross-entropy and therefore low perplexity.
• Branching-factor reading. Perplexity is the effective vocabulary size the model was hesitating over at each position. Perplexity 1 means the model predicted every token with certainty. Perplexity equal to the vocabulary size means the model learned nothing and is guessing uniformly. Real language models on real text land somewhere in the single or low double digits per token, depending on tokenizer, domain, and model size.

Formally, for a sequence of tokens w_1 … w_N and a model that assigns each token a conditional probability given its predecessors:

cross-entropy H = -(1/N) * Σ_{i=1..N} log P(w_i | w_1 … w_{i-1})
perplexity  PPL = exp(H)                     # if log is natural log
perplexity  PPL = 2^H                        # if log is base 2

Both forms appear in the literature; they give the same number as long as the base of the log and the base of the exponent match. 01-05 covers cross-entropy itself in detail — perplexity adds exactly one operation on top of it, the exponentiation, and that operation exists purely to make the number human-readable.

Three properties follow immediately and are worth stating because each one generates a specific class of mistake:

1. Perplexity is defined only for models that assign probabilities to tokens. It is therefore defined for autoregressive (decoder-only) language models like GPT-style models, and it is not naturally defined for encoder-only models like BERT, whose masked-language-model objective is not a left-to-right probability over the whole sequence. You can define pseudo-perplexity for a masked model, but it is a different quantity and not comparable to autoregressive perplexity.
2. Perplexity is per-token, so it depends on the tokenizer. Change the tokenizer and you change the number of tokens N that the same text produces, which changes the per-token average even if the model's probability of the text as a whole is identical.
3. Perplexity is unsupervised. It needs no reference answer, no human label, no rubric — just a model and some text. That is its great practical advantage and the source of its great limitation: a metric that requires no notion of "correct" cannot measure correctness.

L1 — Intuition: how surprised was the model?

Imagine reading a sentence one word at a time and, before each word, writing down your guess distribution. If the sentence is "the cat sat on the ___", you would put most of your probability on "mat", "floor", "chair" — a handful of options. Your effective branching factor there is maybe 5. If the sentence is "my favourite number is ___", you are choosing among effectively hundreds. Average that branching factor over a whole document, and you have perplexity. A better language model is one that is less surprised by real text, i.e. narrows the branching factor at each position.

L2 — Mechanism: the exact computation

For each token position, the model produces a distribution over its vocabulary. You look up the probability it assigned to the token that actually appeared, take the log, sum, negate, divide by the token count, and exponentiate.

for each token i in the evaluation text:
    p_i   = model probability of the actual token at position i, given all prior tokens
    nll_i = -log(p_i)                      # negative log likelihood, in nats
H   = sum(nll_i) / N                       # mean NLL per token = cross-entropy
PPL = exp(H)

Four implementation details change the number materially, and every one of them is a place where two teams reporting "perplexity" are reporting different quantities:

Because of the tokenizer dependence in particular, perplexity is not a cross-model leaderboard metric. Two models with different tokenizers can be ranked in opposite orders by perplexity and by any downstream task metric, and neither ranking is a lie — they are measuring per-token surprise on different tokenizations of the same string. This is the single most important operational fact about perplexity and the one an exam question is most likely to probe indirectly.

L3 — Depth: why exponentiate at all, and what "effective branching factor" really means

Suppose a model assigns uniform probability 1/k to each of k candidate tokens at every position and the true token is always among them. Then nll_i = -log(1/k) = log k at every position, so H = log k and PPL = exp(log k) = k. The exponentiation exists exactly so that this case reads out as k rather than as log k. Perplexity is, by construction, "the size of the uniform choice set that would produce this much average surprise".

This also explains the shape of perplexity's sensitivity. Because it is exponential in the loss, small loss improvements look large in perplexity terms at low loss and small at high loss. Going from cross-entropy 3.0 to 2.9 changes perplexity from 20.09 to 18.17 — about 9.5%. Going from 1.0 to 0.9 changes it from 2.718 to 2.460 — also about 9.5%. The relative change in perplexity is exp(ΔH), constant for a constant loss delta. But the absolute perplexity drop is nearly two whole points in the first case and a quarter of a point in the second, which is why perplexity charts on a well-trained model look flat while the loss is still genuinely improving. Read loss curves in loss space, and quote perplexity only as a headline. 12-03 covers what a loss curve's shape tells you diagnostically.

One further depth point: perplexity has a hard floor of 1 and an interpretation problem near it. Perplexity below about 1.5 on natural text usually means one of three things has gone wrong: the evaluation text was in the training data (memorisation, not capability), the text is highly repetitive or templated, or the scoring loop is leaking future tokens. A suspiciously low perplexity is a data-hygiene alarm, not a victory. This is exactly the contamination logic that 10-01 applies to public benchmarks.

The critical distinction to hold: perplexity is reference-free and quality-blind; every other metric in that table needs a reference or a label and is therefore capable of scoring correctness. Perplexity measures the model's opinion of the text; BLEU, ROUGE, exact match and BERTScore measure the text's agreement with a human-supplied target. 09-06 handles the BLEU-versus-ROUGE contrast in full, and 09-05 gives the full metric-selection decision procedure.

Two special cases are worth pinning down because they show up as distractors:

• A lower-perplexity model can produce worse output. Perplexity rewards predicting the most likely continuation. Instruction-following, refusal, and formatting behaviours are often less likely under the raw pretraining distribution, so a model fine-tuned to follow instructions frequently has higher perplexity on generic web text than its base model while being far more useful. Alignment training and perplexity pull in different directions; 11-06 explains why.
• Perplexity on generated text is not perplexity on held-out text. Scoring a model's own output with itself always gives a low number — the model likes its own choices. Perplexity is only informative on text the model did not produce and did not train on.

We construct a small, explicitly illustrative example. These probabilities are invented for arithmetic, not measured from any real model.

A model scores the token sequence ["The", " cat", " sat", " on", " it"] and assigns these probabilities to the actual tokens:

Step 1 — sum the negative log-likelihoods.

2.302585 + 2.995732 + 1.386294 + 0.693147 + 3.912023 = 11.289781

Step 2 — divide by the token count to get cross-entropy per token.

H = 11.289781 / 5 = 2.257956 nats/token

Step 3 — exponentiate.

PPL = exp(2.257956) = 9.564

So this model's perplexity on this sentence is about 9.56: on average, it was as uncertain as if it were choosing uniformly among roughly 9.6 tokens at each step.

Step 4 — see where the surprise came from. The mean hides the story. Position 5 ( it, p = 0.02) contributes 3.912 of the 11.290 total — 34.6% of all the surprise from 20% of the tokens. Position 4 ( on, p = 0.50) contributes 6.1%. Perplexity is an average, and averages of exponentially-weighted surprise are dominated by their worst positions. The per-token NLL list is more diagnostic than the perplexity. A single unexpected token — a rare proper noun, a domain term, a typo — can dominate a short text's perplexity entirely.

Step 5 — check the effect of one changed probability. Suppose an improved model raises P(" it") from 0.02 to 0.10 and leaves everything else identical. Then nll_5 falls from 3.912023 to 2.302585, the sum becomes 9.680343, H becomes 1.936069, and:

PPL = exp(1.936069) = 6.932

Perplexity drops from 9.56 to 6.93 — a 27.5% improvement — from a single token's probability rising by 0.08. That extreme leverage is a real property, and it is why perplexity on short texts is volatile and why perplexity should be computed over a substantial corpus, not a sentence.

Step 6 — demonstrate the tokenizer trap. Now suppose a different tokenizer splits cat into two tokens, c and at, with probabilities 0.30 and 0.60. Everything else is unchanged. The sequence is now six tokens:

nll:  2.302585 (The) + 1.203973 (" c") + 0.510826 (at) + 1.386294 (sat)
      + 0.693147 (on) + 3.912023 (it)   =  10.008848
H   = 10.008848 / 6 = 1.668141
PPL = exp(1.668141) = 5.302

Perplexity fell from 9.56 to 5.30 with no change to the model's beliefs about the sentence — the joint probability assigned to the string is nearly the same (0.10 × 0.30 × 0.60 × 0.25 × 0.50 × 0.02 = 4.5e-6 versus 0.10 × 0.05 × 0.25 × 0.50 × 0.02 = 1.25e-5, in fact slightly lower). All that changed is the denominator. This is the entire argument for why cross-model perplexity comparison across tokenizers is invalid, and it is worth being able to reproduce from memory. Tokenizer differences are covered in 02-04.

The compressed rule: perplexity is a training-side and monitoring-side instrument. It is not a product-quality metric. If the question is "is the output good?", perplexity is not the answer, and on a multiple-choice exam an option offering perplexity for a quality question is almost always a distractor.

Perplexity sits at the intersection of two blueprint domains. In Core Machine Learning and AI Knowledge (30% of the exam) it belongs to objective 1.5, familiarity with ML fundamentals, and it is the natural companion of cross-entropy as the training objective. In Experimentation (22%) it is one of the statistical performance metrics named by the duplicated objective pair 2.2 / 3.2 — "compare models using statistical performance metrics, such as loss functions or proportion of explained variance". Perplexity is a monotone transform of a loss function, so it is squarely inside that wording.

The objective-numbering defect, stated plainly. The official study guide prints Experimentation's objectives as 3.1–3.5, and those five lines are a verbatim duplicate of Data Analysis's 2.1–2.5. Read literally, they describe data mining, data analysis, chart creation and trend identification — not model evaluation. The Experimentation section's own scope statement is explicit that the domain covers "AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback (RLHF)", and its suggested-reading list names machine-translation evaluation, GLUE, cross-validation, hallucinations and RAG evaluation. Published candidate reports independently confirm BLEU and hallucination content on the exam. So when you see a metrics question, answer it from the derived scope; do not expect the printed 3.x text to help you. This lesson is the one lesson in this module that would still be justified by the printed objectives, because "loss functions" appears verbatim in 2.2/3.2.

Expected question phrasings:

• "What does perplexity measure?" → How well a language model predicts a sample of text; equivalently the exponentiated average cross-entropy per token. Distractors: "the fluency of generated text", "the factual accuracy of outputs", "the diversity of the vocabulary used".
• "A model's perplexity on a corpus drops from 24 to 12. What can you conclude?" → It predicts that corpus roughly twice as sharply. You cannot conclude the outputs are better, more truthful, or preferred by users.
• "Two models are compared by perplexity but use different tokenizers. Why is the comparison unsound?" → Perplexity is per-token, so different tokenizations change the denominator; the numbers are on different scales.
• "Which metric requires no reference or labelled data?" → Perplexity. This is the property that distinguishes it from BLEU, ROUGE, exact match and BERTScore.
• "Which metric would you use to evaluate a summarisation system?" → ROUGE (recall-oriented), not perplexity. Perplexity appearing here is a classic distractor.
• "Is a lower perplexity always better?" → No. It is better within one tokenizer and one corpus. An instruction-tuned model may show higher perplexity on generic text and be the better product.

Distractor families. Four recur. (1) Perplexity-as-quality — offered for summarisation, translation, or helpfulness questions. (2) Perplexity-for-classification — offered where accuracy, precision/recall or F1 belong. (3) Cross-tokenizer comparison presented as valid. (4) Perplexity described as a measure of output diversity — the word "perplexity" sounds like it should mean "variety", and it does not; decoding-time diversity is controlled by temperature and top-p (04-05), which do not change perplexity at all.

No — it is better only within a fixed tokenizer, a fixed evaluation corpus, and ideally a fixed model family. Inside those brackets, a lower perplexity genuinely means the model assigns more probability mass to real text, which usually correlates with better language modelling. Outside them, three failure cases break the rule. Different tokenizers change the per-token denominator and can reverse the ranking. Different corpora have different intrinsic predictability, so the comparison measures the text, not the model. And alignment training deliberately shifts a model away from the most-probable continuation towards the most-useful one, which can raise perplexity on generic text while improving every metric users care about. There is also a degenerate direction: a model that has memorised the evaluation text will show very low perplexity and no generalisation at all.

Not reliably, and this is the most consequential thing perplexity cannot do. A hallucination is a fluent, confident, false statement. Perplexity measures fluency-under-the-model, which is precisely the property a hallucination has. A fabricated citation in a plausible format is high-probability text; the model is not surprised by it at all. Low perplexity is therefore fully compatible with a completely fabricated answer.

There is a weaker, real signal in the neighbourhood: token-level confidence and entropy at generation time sometimes correlate with factual uncertainty, so an unusually flat next-token distribution at the position where a name or a number is emitted can be a warning sign, and self-consistency across several sampled generations is a stronger one. But these are heuristics, not measures, and they miss confidently-wrong outputs — the dangerous kind. Detecting hallucination requires grounding the claim against a source, which is what faithfulness metrics in 09-07 do and what the mitigation ladder in 09-12 is built around.

There is no absolute answer, and any specific number quoted without its tokenizer and corpus is uninterpretable. Perplexity is only meaningful as a relative quantity: this checkpoint versus that checkpoint, on this frozen held-out set, with this tokenizer. What you can say structurally is that perplexity is bounded below by 1 (perfect prediction) and above by the effective vocabulary size (uniform guessing), that a well-trained modern language model on general text sits in the single-to-low-double digits, that perplexity below roughly 1.5 on natural text should be investigated as possible contamination, and that perplexity in the hundreds or thousands on your own domain text means the model has effectively never seen your domain. Rather than chase a target value, record your baseline on a frozen set and watch the direction of travel. A useful practical trio: baseline perplexity on general held-out text, perplexity on your own domain corpus, and the ratio between them. That ratio is a domain-fit indicator, and it moves when your traffic drifts — which makes it a genuinely good production monitor.

Because perplexity is defined over a left-to-right factorisation of the sequence probability, and a masked language model does not produce one. An autoregressive decoder-only model gives you P(w_i | w_1 … w_{i-1}) at every position by construction, so the chain rule assembles a well-defined probability for the whole sequence. BERT's masked-language-model objective instead predicts a masked token from both directions of context, so the per-position probabilities are conditioned on future tokens and do not multiply into a valid sequence probability. Researchers define a pseudo-perplexity by masking each token in turn and averaging, which is a legitimate quantity but is neither the same measure nor comparable to autoregressive perplexity. The practical consequence for the exam: perplexity belongs to the decoder-only, generative side of the architecture map in 04-03, and encoder-only models are evaluated with classification metrics instead.

1. Perplexity = exp(mean per-token cross-entropy). One operation on top of the training loss, performed purely to make the number readable.
2. Read it as an effective branching factor. PPL 8 ≈ as uncertain as a uniform choice among 8 tokens.
3. It needs no labels. That makes it cheap and makes it correctness-blind. Both facts follow from the same property.
4. It is per-token, so it is tokenizer-dependent. The worked example dropped PPL from 9.56 to 5.30 by re-tokenizing, with no change in the model's beliefs.
5. Never compare perplexity across tokenizers or across corpora. Both comparisons are invalid, and both are common exam distractors.
6. Lower is not always better. Instruction tuning frequently raises perplexity on generic text while improving the product.
7. Perplexity cannot detect hallucination. A hallucination is fluent by definition, which is exactly what perplexity rewards.
8. A suspiciously low perplexity is a contamination alarm, not an achievement.
9. It is defined for autoregressive models, not natively for masked encoder-only models like BERT.
10. Use it for training monitoring, checkpoint selection within a family, and domain-drift detection. Use a reference-based or rubric-based metric for anything about output quality.

Perplexity told you how surprised the model was, and told you nothing about whether the answer was any good. Every metric that can answer that question needs a target — a reference, a label, or a judgement — and all three ultimately trace back to a human deciding what "good" means and writing it down. That decision is far less reliable than it looks: give the same output to two careful people with the same instructions and they will disagree more often than either expects, and the size of that disagreement puts a hard ceiling on every number you compute downstream, including the reward model inside RLHF.

Next: 09-03 covers human evaluation — how to write a rubric that two people apply the same way, how to measure their agreement with Cohen's kappa and Krippendorff's alpha, and what to do when agreement comes back at 0.4.