BPE vs WordPiece vs SentencePiece: which model uses which

All three are algorithms for fitting a subword vocabulary to a corpus, plus a matching segmentation procedure for applying it. Each produces a frozen vocabulary of the kind 02-02 described, and each guarantees no out-of-vocabulary failures. Here they are in one paragraph each.

Byte-pair encoding (BPE) starts with a base alphabet — characters, or in byte-level BPE the 256 byte values — and repeatedly finds the most frequent adjacent pair of symbols in the corpus, merges it into a new single symbol, and records the merge. Repeat until the vocabulary reaches its target size. The output is an ordered list of merge rules; segmentation replays those merges on new text in the same order. The selection criterion is pure frequency. GPT-family models use BPE, in its byte-level variant.

WordPiece works the same way structurally — grow a vocabulary by merging pairs — but changes the selection criterion. Instead of merging the most frequent pair, it merges the pair that most increases the likelihood of the training corpus under a unigram language model over the vocabulary. Informally: it merges the pair whose joint frequency is highest relative to the product of its parts' frequencies, so a pair that co-occurs more than chance would predict is preferred over a pair that is merely common because both halves are common. BERT-family models use WordPiece, and its continuation marker is ##.

SentencePiece is different in kind, and this is the part most often misunderstood. It is best described as a tokenizer framework or implementation rather than a single merge algorithm: it can be trained with a BPE objective or with a unigram language-model objective. Its defining property is that it operates on raw text as an undifferentiated character stream with no whitespace pre-tokenization, encoding spaces as an ordinary visible symbol (conventionally ▁, U+2581 lower one-eighth block) inside the vocabulary. Because it never assumes that spaces delimit words, it works identically on English and on languages written without spaces. T5 uses SentencePiece, commonly with the unigram objective.

L1 — The intuition: three different answers to "which pair should we merge next?"

Picture a corpus reduced to individual characters. You have a budget of, say, 30,000 vocabulary slots and you must decide what to spend them on. Every algorithm here spends them by combining pieces into bigger pieces; they disagree about which combination to buy next.

• BPE buys the most common pair. Simple, greedy, frequency-driven. If t+h is the most frequent adjacent pair in English text, th becomes a symbol.
• WordPiece buys the pair that best explains the corpus — the pair whose parts appear together far more often than their individual frequencies would predict. It resists merging pairs that are common only because both halves are ubiquitous.
• SentencePiece with the unigram objective works backwards: it starts from a large pool of candidate pieces and prunes the ones whose removal hurts the corpus likelihood least, until the vocabulary is small enough. Subtractive rather than additive.

And orthogonally to all of that, SentencePiece makes a second choice nobody else makes: it never splits on spaces first. Spaces are just another character in the stream, encoded as a visible symbol. That is the difference that matters for languages without whitespace.

L2 — The mechanism, algorithm by algorithm

Byte-pair encoding, in detail. BPE was originally a data-compression algorithm, repurposed for tokenization. Training:

1. Initialise the vocabulary with the base alphabet — every character, or all 256 byte values in the byte-level variant.
2. Represent the corpus as sequences of those base symbols.
3. Count every adjacent symbol pair across the corpus. Find the most frequent.
4. Merge it: add the concatenated pair as a new vocabulary entry, replace every occurrence in the corpus, and append the merge to an ordered list.
5. Repeat from step 3 until the vocabulary hits the target size.

Segmentation at inference: split the text into base symbols and replay the recorded merges in their original order, applying each wherever it matches. The merge order is part of the model — the same set of merges applied in a different order would produce different splits, which is why the merge list is ordered rather than a set.

Two properties matter. First, BPE is deterministic and fast. Second, in its byte-level form the base alphabet is the 256 byte values, so any input whatsoever is representable, including malformed Unicode and binary. That is the guarantee 02-01 described, and it is why byte-level BPE became the default for large generative models.

WordPiece, in detail. Training follows the same additive shape but changes step 3's criterion. Rather than raw pair frequency, WordPiece scores each candidate pair by how much merging it would increase the corpus likelihood under a unigram model. The score is commonly written as a ratio:

score(a, b) = freq(ab) / ( freq(a) × freq(b) )

Read that ratio carefully, because it is the whole idea. A pair like e+ might have an enormous joint frequency, but both parts are individually so common that the ratio is unremarkable — the pair is common by coincidence of its parts. A pair like q+u in English has a lower joint frequency, but q almost never appears without u, so the ratio is very high and the merge is genuinely informative. WordPiece prefers the second. The effect is a vocabulary biased toward pieces that carry distinctive information rather than pieces that are merely frequent.

Segmentation at inference is typically greedy longest-match against the vocabulary, working left to right within each pre-token, exactly as the worked example in 02-02 demonstrated. Non-initial pieces carry the ## prefix, so playing might come out as play + ##ing. That marker is not decoration: it makes the segmentation unambiguously reversible, since a decoder can tell that ##ing continues the previous token rather than starting a new word.

SentencePiece, in detail. SentencePiece's two contributions are independent and both matter.

Contribution one — no whitespace pre-tokenization. Every other tokenizer discussed so far splits on whitespace before doing subword work. SentencePiece does not. It takes the raw string, replaces each space with the visible meta-symbol ▁, and treats the whole thing as one character stream. Consequences:

• Vocabulary entries can span what English speakers would call a word boundary, since nothing forbids it.
• The tokenization is losslessly reversible by construction: concatenate the pieces and replace ▁ with a space, and you have the original string back exactly, including its spacing. There is no ambiguity to resolve and no detokenization heuristic to get wrong.
• Languages without spaces between words work identically, because the algorithm never depended on spaces meaning anything. This is the headline benefit and the reason SentencePiece exists.
• The tokenizer is language-agnostic and requires no external word segmenter. For Chinese or Japanese, a whitespace-pre-tokenizing approach would need a separate language-specific segmenter as a preprocessing step; SentencePiece needs nothing.

Contribution two — a choice of training objectives. SentencePiece implements BPE and it implements a unigram language model objective. The unigram procedure is subtractive:

1. Seed a large candidate vocabulary — far bigger than the target — from frequent substrings.
2. Fit a unigram probability for each piece by expectation-maximisation over the corpus.
3. Compute, for each piece, how much the corpus likelihood would drop if that piece were removed.
4. Prune the pieces whose removal costs least.
5. Repeat until the vocabulary reaches the target size.

Because every piece carries a probability, unigram segmentation is a probabilistic search for the highest-likelihood segmentation of the input, rather than a greedy left-to-right match. This also enables subword regularisation: sampling alternative segmentations of the same text during training as a form of data augmentation, which a purely deterministic greedy tokenizer cannot offer.

This dual nature is the source of the single most common confusion about SentencePiece, and it is worth stating explicitly: "SentencePiece" and "unigram" are not synonyms. SentencePiece is the framework; unigram is one objective it can be trained with; BPE is another. A question that offers "SentencePiece is an alternative to BPE" as an option is offering something that is, strictly, a category error — though in practice the intended contrast is usually "SentencePiece-with-unigram versus BPE," which is a coherent comparison.

L3 — Why the differences are smaller than they look, except for one

Trained on the same corpus to the same vocabulary size, BPE and WordPiece produce substantially overlapping vocabularies. Both are frequency-driven additive merge procedures; the likelihood criterion shifts which pairs win at the margin, not the overall character of the result. Common words get their own entries under both; rare words split into fragments under both; neither produces morphologically principled splits. If you swapped one for the other in a model's pretraining, you would get a slightly different token count and a model of broadly similar quality.

The whitespace assumption is the difference that is not cosmetic. A tokenizer that pre-splits on whitespace has, baked into it, the assumption that the language delimits words with spaces. For English, German, Spanish and most European languages that assumption holds well enough. For Chinese, Japanese and Thai it is simply false: there are no spaces between words, so whitespace pre-tokenization either does nothing (leaving the subword algorithm to operate on entire sentences) or requires bolting on a language-specific word segmenter, which is a separate model with its own errors and its own maintenance burden.

SentencePiece dissolves the problem by never making the assumption. That is why it is the standard choice for multilingual models, and it is the most defensible substantive claim you can make in a comparison question about these three.

The second non-cosmetic difference is round-trip fidelity. SentencePiece's ▁ convention makes detokenization exact and mechanical. Marker-prefix schemes like WordPiece's ## are also reversible, but reconstruction depends on interpreting the marker correctly and on knowing the tokenizer's normalisation and spacing conventions. Byte-level BPE achieves exactness a third way, by attaching the leading space to the following token. All three can round-trip; SentencePiece's route is the one with the fewest conventions to get wrong.

This table is the highest-value asset on the page. Learn the first two rows cold.

The memorization card

Because the blueprint flags this as a card-flag lesson — a not-derivable fact to be drilled cold — here is the minimum you must be able to produce in under three seconds:

GPT → BPE BERT → WordPiece T5 → SentencePiece

And the marker recognition, which is the second-most-asked form:

##ing → you are looking at WordPiece output (BERT) ▁the → you are looking at SentencePiece output (T5) the with the space inside the token → byte-level BPE (GPT)

Two mnemonics that hold up. First, alphabetical alignment: BERT → WordPiece is the odd one out, and the other two are ordered — BPE for the family whose name starts earliest in the generative lineage (GPT), SentencePiece for T5, the latest of the three architectures in the comparison. Second, semantic: SentencePiece works on whole sentences rather than words, which is exactly its defining property and its name. That one is worth internalising because it makes the "handles languages without spaces" answer fall out of the name itself.

Take the word unhappiness. Below is how each algorithm would plausibly segment it, with the reasoning shown. These are constructed illustrations demonstrating each algorithm's characteristic behaviour and marker convention — they are not measured outputs from any particular released tokenizer, whose vocabularies differ by version.

Byte-level BPE (GPT-style). Base symbols are bytes; the merge list is replayed in order. Frequency-driven merges will have built up common English fragments, so the merges fire roughly like this:

u n h a p p i n e s s                 ← base symbols
un h app i ness                       ← frequent pairs merged
un happ i ness                        ← further merges
un + happiness       →   ["un", "happiness"]      2 tokens

If the word had appeared with a preceding space, the space would ride along inside the first token: [" un", "happiness"]. No continuation marker; the space convention does the work.

WordPiece (BERT-style). Greedy longest-match against a vocabulary whose merges were chosen by the likelihood ratio, with ## on non-initial pieces:

["un", "##happ", "##iness"]            3 tokens

The ## prefixes are the signature. A decoder reading this knows to concatenate without inserting spaces, so un + happ + iness reassembles exactly.

SentencePiece with unigram (T5-style). The space before the word is part of the stream, encoded as ▁, and the segmentation is the highest-likelihood one rather than a greedy left-to-right match:

["▁un", "happiness"]                   2 tokens

The ▁ on the first piece marks it as word-initial. Concatenate the pieces and replace ▁ with a space and you have unhappiness back exactly, with no convention to interpret.

Side by side:

Notice what is not different: none of the three produced the morphologically correct un + happy + ness. All three produced statistically convenient pieces. That reinforces 02-02's point that tokens are statistical units, and it is why a question offering "WordPiece produces morphologically correct splits" as a distinguishing feature is offering a wrong answer.

Notice also that the token counts differ — 2, 3, 2 — for the same word. This is the mechanism behind 02-03's insistence that a token count is a property of the (text, tokenizer, version) triple.

This is where the three genuinely diverge, so it is worth walking through explicitly. Consider a short Japanese phrase — written, as Japanese is, with no spaces between words. Call it 私はモデルを訓練する ("I train a model"), ten characters, zero spaces.

A whitespace-pre-tokenizing tokenizer (BPE or WordPiece as normally configured) first splits on whitespace. There is none. So the entire phrase arrives at the subword stage as a single pre-token, and the subword algorithm must segment ten characters with a vocabulary that — if fitted on a predominantly English corpus — contains almost no multi-character Japanese entries. The likely result is a fall-back to per-character or, under byte-level BPE, per-byte pieces. Each of these characters occupies three bytes in UTF-8, so a byte-level tokenizer without Japanese entries could spend up to three tokens per character:

10 characters × up to 3 UTF-8 bytes = up to 30 tokens for a 10-character phrase

Compare that with English prose at roughly four characters per token and the asymmetry is stark. This is the mechanism behind 02-03's finding that non-English text costs more tokens for the same meaning — and it is a tokenizer design consequence, not an inherent property of the language.

SentencePiece never split on whitespace in the first place, so the absence of spaces changes nothing about its procedure. If it was trained on a corpus including Japanese, its vocabulary contains multi-character Japanese pieces, and it segments the phrase into a handful of meaningful units. If it was trained only on English, it too falls back — the framework does not conjure knowledge it was not trained on. The advantage is that SentencePiece can be trained on such a corpus without requiring a language-specific word segmenter as a preprocessing step. That is the accurate, defensible version of the claim, and it is stronger than the loose "SentencePiece handles Japanese better," which is only true given appropriate training data.

Decision table — which tokenizer for which situation:

Tokenizer algorithms sit under NCA-GENL objective 1.6 — familiarity with the capabilities of Python natural language packages — and they are one of the confusable sets this course's own self-sufficiency contract names for explicit distractor training, alongside WordNet vs word2vec, BLEU vs ROUGE, TensorRT vs TensorRT-LLM, and stemming vs lemmatization. Published candidate reports place tokenization in the highest-frequency tier of exam topics. That tiering is field calibration rather than official documentation, so read it as a study-time allocation signal — but here it converges with the blueprint's own instruction, which flags this lesson as a card and states plainly that the mapping is not derivable.

That combination is why this lesson is drilled rather than reasoned. Most of this course teaches you to derive answers from mechanisms. This one cannot be derived: nothing about a decoder-only architecture implies byte-pair encoding, and nothing about masked language modelling implies WordPiece. They are historical facts about specific model families, and the exam can ask them directly.

Question phrasings to expect:

• "Which tokenization algorithm does BERT use?" — WordPiece. The trap distractor is BPE, because BPE is the most famous of the three.
• "Which model family uses SentencePiece?" — T5. Also correct for many multilingual and encoder-decoder models, but T5 is the keyed association.
• "What distinguishes SentencePiece from BPE and WordPiece?" — It does not require whitespace pre-tokenization, treating text as a raw stream, which makes it language-agnostic.
• "BPE selects which pair to merge based on what?" — Frequency of adjacent pairs.
• "What does the ## prefix indicate in a token list?" — A WordPiece continuation piece: a non-initial fragment of a word.
• "A team needs a tokenizer for a corpus containing Chinese and Thai. Which do you recommend?" — SentencePiece, because those languages do not delimit words with spaces.
• "Which of these tokenizers can produce an out-of-vocabulary token?" — None of the three in normal configuration; byte-level BPE cannot even in principle.

Distractor families, which are unusually well-defined for this topic:

Byte-pair encoding, in its byte-level variant. The base alphabet is the 256 possible byte values rather than characters, which gives byte-level BPE a property worth stating separately: no input can ever be out-of-vocabulary, because every possible byte sequence decomposes into bytes the tokenizer knows. Malformed Unicode, emoji, binary blobs, text in scripts the vocabulary has never seen — all encode successfully, just at more tokens per unit of meaning.

The other visible convention is that the leading space belongs to the following token. the and the are different vocabulary entries. This is why 02-02 warned about tokenizing fragments and summing: a leading space that gets absorbed differently when strings are joined changes the count.

WordPiece, and its signature is the ## prefix on non-initial pieces. If you print a token list and see ["play", "##ing"], you are looking at WordPiece output. BERT-derived models — DistilBERT, ELECTRA and similar — inherit it.

Because WordPiece selects merges by likelihood improvement rather than raw frequency, its vocabulary skews slightly toward pieces that carry distinctive information — the q+u case from §2 rather than the e+space case. In practice, trained on the same corpus to the same size, its vocabulary overlaps heavily with a BPE vocabulary. The criterion is the exam-relevant difference; the practical difference is modest.

SentencePiece, commonly with the unigram language-model objective. This pairing is the third leg of the memorization card, and it generalises usefully: encoder-decoder and multilingual models converged on SentencePiece for the reason §2 gave, namely that its refusal to assume whitespace word boundaries makes it language-agnostic without bolting on per-language segmenters.

The visible signature is ▁ (U+2581) marking word-initial pieces. Seeing ["▁un", "happiness"] in a token list identifies SentencePiece immediately.

SentencePiece is the framework; unigram is one training objective it supports, and BPE is another. They are at different levels of the stack, which is why treating them as competitors produces confused answers.

The clean way to hold it:

So "SentencePiece with BPE" and "SentencePiece with unigram" are both real, commonly deployed configurations. When an exam question contrasts SentencePiece with BPE, the intended contrast is almost always the whitespace-handling difference, which is the substantive one.

Marginally, for the merge criterion; substantially, for language coverage.

Swapping BPE for WordPiece on an English corpus at the same vocabulary size is a marginal change: overlapping vocabularies, similar token counts, similar downstream quality. It is not where model quality comes from.

What does matter, and matters a lot:

• Vocabulary size, with all three arms of the 02-02 trade-off in play — sequence length, embedding parameters, per-row training signal.
• Training-corpus language mix, which determines whose text encodes efficiently and whose does not. A model whose tokenizer saw little of a language will spend many tokens per unit of meaning in it, consuming context and money.
• Whether whitespace pre-tokenization is assumed at all, which is a hard constraint for space-free languages rather than a tuning preference.
• Byte-level fallback, which decides whether unusual input degrades gracefully or fails.

So the honest answer to a "which algorithm is best?" question is that the algorithm is rarely the binding constraint — but that SentencePiece is genuinely the right answer when whitespace cannot be assumed, and byte-level BPE is genuinely the right answer when arbitrary bytes must be representable.

1. The card, cold: GPT → BPE. BERT → WordPiece. T5 → SentencePiece. The blueprint flags this as not derivable, so it is drilled, not reasoned. The most likely wrong answer is "BERT uses BPE."
2. The merge criteria differ in a stateable way. BPE merges the most frequent adjacent pair. WordPiece merges the pair that most improves corpus likelihood, favouring pairs that co-occur more than chance. Unigram prunes the least-useful pieces from an oversized candidate set.
3. SentencePiece's defining property is not its merge rule — it is the absence of whitespace pre-tokenization. It treats raw text as a stream and encodes spaces as ▁. That is what makes it language-agnostic and the right choice for Chinese, Japanese and Thai.
4. SentencePiece is a framework, not an algorithm. It can be trained with BPE or unigram objectives. Its true peer at the objective layer is unigram, not BPE.
5. You can identify each on sight. ##ing is WordPiece. ▁the is SentencePiece. A space living inside the token is byte-level BPE.
6. None of them produces morphological splits. All three produce statistically convenient pieces. Expecting linguistic correctness is a category error, and it is a distractor.
7. None of them can produce out-of-vocabulary failures in normal configuration, and byte-level BPE cannot even in principle.
8. For a pretrained model there is no choice to make. The tokenizer is a matched pair with the embedding matrix; substituting it fails silently rather than loudly, which makes it more dangerous, not less.
9. Vocabulary size and language coverage matter far more than the merge criterion. Do not spend engineering effort on BPE-versus-WordPiece for English work; spend it on 02-02's trade-off and on measuring your own corpus per 02-03.

Every algorithm in this lesson cuts words into statistically convenient pieces with no regard for grammar. That leaves an obvious gap: sometimes you genuinely do want the linguistic root — you want running, ran and runs to collapse to run so that a search index or a bag-of-words classifier treats them as one term. Subword tokenization does not do that and was never trying to.

Classical NLP has two different tools for it, they are not interchangeable, and the distinction between them is one of the most heavily reported items on this exam. One is a crude rule-based truncation that can produce non-words. The other is a dictionary-backed lookup that produces a valid root and needs to know the part of speech to do its job.

Next: 02-05 draws that line precisely — stemming versus lemmatization, when each is correct, why stop-word removal was standard practice in classical pipelines and is usually wrong for LLMs, and the canonical order of the preprocessing pipeline that the exam asks about directly.