Token Embeddings vs Sentence and Document Embeddings

Three granularities, one mechanism, different units.

Token embeddings. A transformer's internal representation. The input text is tokenized (02-02), each token id is looked up in the embedding matrix to get an initial vector, and then each layer rewrites those vectors using self-attention over the whole sequence (04-01). What comes out of the final layer is a sequence of vectors — one per token, each of width d (the hidden size from 01-03). For an input of 40 tokens with hidden size 768, the output is a matrix of shape (40, 768). Every one of those 768-wide rows is contextual: the vector for bank differs depending on whether river or transfer appeared nearby.

Sentence embeddings. One vector for a whole sentence or short passage, shape (768,) for the same model. You get there by pooling the token vectors — collapsing the (40, 768) matrix down to a single (768,) vector — and, in any model worth using for retrieval, by having trained the model so that this pooled vector is meaningful when compared to other pooled vectors.

Document embeddings. The same operation applied to a longer span: a paragraph, a section, a page, or in principle a whole document. The mechanism is identical to sentence embedding; what changes is that the span may exceed the model's maximum input length, at which point something has to give — truncation, chunking, or aggregation. In practice, "document embedding" in a RAG system almost always means "an embedding of a chunk of a document", because chunking is what makes the span fit and keeps the vector specific enough to be useful.

The critical asymmetry: a model that produces good token embeddings does not automatically produce good sentence embeddings. Pooling is trivially easy to do and the result always looks fine — right shape, right magnitudes, cosine similarities in a plausible range. Whether those pooled vectors are actually comparable to each other depends on whether the model was trained to make them so. That single sentence is the reason this lesson exists.

L1 — The intuition: collapsing a matrix to a vector

You have 40 vectors and you need one. Something must summarise. The three summarisation strategies you will meet, in order of how often you will see them:

• Mean pooling — average the token vectors component by component. Every token contributes equally.
• CLS pooling — take the vector at a single designated position. BERT-family models prepend a special [CLS] token, and its final-layer vector is conventionally read as a whole-sequence summary.
• Max pooling — take the component-wise maximum across tokens. Rarer for text retrieval.

All three turn (n_tokens, d) into (d,). All three are one line of code. None of them, on their own, makes the result good.

L2 — The mechanism, and why attention masking matters

Mean pooling has one implementation detail that causes real bugs: padding. When you embed a batch of texts of different lengths, short texts are padded out to the batch's longest sequence with a pad token. If you average across all positions including the pad positions, short texts get their real signal diluted by meaningless pad vectors, and — worse — the amount of dilution depends on what else happened to be in the batch. The same sentence embedded in two different batches then produces two different vectors.

The fix is attention-mask-weighted mean pooling: sum only the real-token vectors and divide by the count of real tokens. Every serious embedding library does this. It is worth knowing about because it explains a specific mystifying symptom — "identical text gives different vectors on different runs" — that has nothing to do with the model and everything to do with pooling.

CLS pooling has a different dependency. The [CLS] position is only a meaningful summary if the model was trained with an objective that pushed a summary into it. In original BERT pretraining, [CLS] was trained for next-sentence prediction, which is a weak signal for semantic similarity; that is why simply grabbing [CLS] from a raw pretrained BERT and comparing those vectors by cosine is a well-known disappointment. In a model fine-tuned for sentence embeddings with a pooling head, whichever pooling the fine-tuning used is the pooling you must use at inference — mismatching them silently degrades quality.

L3 — Why fine-tuning on pairs is what actually makes span vectors comparable

Here is the part that separates people who understand embeddings from people who use them.

A base encoder is pretrained on masked-language modelling: predict hidden tokens from their context. That objective makes token representations excellent, because token prediction is what it optimises. It never once asks the model "is this whole sentence similar in meaning to that whole sentence?" So there is no reason for the geometry of pooled vectors to be well-behaved, and empirically it is not: pooled base-encoder vectors tend to be crowded into a narrow cone of the space, so almost every pair of unrelated texts returns a high-looking cosine and the differences between related and unrelated pairs are small.

Purpose-built sentence-embedding models fix this with contrastive fine-tuning on pairs. You assemble large collections of pairs that should be close — question and its answer, a sentence and its paraphrase, a query and the passage that satisfies it, a sentence and its translation — and train with an objective that pulls each positive pair together while pushing it away from negatives (typically the other items in the same batch, "in-batch negatives"). Now the pooled vector is directly optimised for the thing you will use it for: cosine comparison of whole spans.

Two consequences follow, and both are exam-relevant selection criteria taken up in 03-03:

1. Symmetric vs asymmetric objectives. If the training pairs were two similar sentences (paraphrase, duplicate question), the model is tuned for symmetric similarity — comparing like with like. If the pairs were short query against long passage, it is tuned for asymmetric search — the two sides have different shapes and different roles. A symmetric model used for query-to-passage retrieval underperforms, and vice versa.
2. Prefixes and instructions. Some models are trained expecting a marker on each side, such as a query prefix and a passage prefix, or a short instruction string. If the model's documentation specifies one and you omit it, you get a working-looking system that quietly retrieves worse. This is an unglamorous, very common defect.

So the honest mechanism summary is: pooling gives you a span vector; pair-based fine-tuning makes span vectors comparable. Doing the first without the second is the mistake.

The specificity–coverage trade-off across granularity

There is a genuine tension and it is worth naming explicitly, because it is the reasoning that drives chunk-size decisions in 06-02.

That last row is the phenomenon worth remembering: the longer the span, the more the embedding regresses toward a bland centroid. A vector has a fixed number of dimensions no matter how much text you feed it, so there is a hard limit on how much distinguishable information one vector can hold. A vector for a whole employee handbook cannot simultaneously be near "how do I claim mileage" and near "what is the parental leave policy" and far from everything else. Embedding is lossy compression, and compressing 40 pages into 768 numbers loses the things you were going to search for.

The arithmetic below is a constructed illustration, not output from a real model. Real hidden sizes are hundreds of dimensions; four is used here so you can follow every step.

Suppose an encoder processes the sentence "invoice is overdue" and returns these final-layer token vectors, with [CLS] prepended:

[CLS]    = [0.10, 0.20, 0.10, 0.10]
invoice  = [0.90, 0.10, 0.20, 0.00]
is       = [0.10, 0.10, 0.10, 0.10]
overdue  = [0.70, 0.20, 0.60, 0.10]

Step 1 — mean pooling over the real tokens

Average the three content tokens plus [CLS], component by component (4 vectors):

dim 0: (0.10 + 0.90 + 0.10 + 0.70) / 4 = 1.80 / 4 = 0.450
dim 1: (0.20 + 0.10 + 0.10 + 0.20) / 4 = 0.60 / 4 = 0.150
dim 2: (0.10 + 0.20 + 0.10 + 0.60) / 4 = 1.00 / 4 = 0.250
dim 3: (0.10 + 0.00 + 0.10 + 0.10) / 4 = 0.30 / 4 = 0.075

mean-pooled = [0.450, 0.150, 0.250, 0.075]

Step 2 — CLS pooling on the same input

CLS-pooled = [0.10, 0.20, 0.10, 0.10]

Two different vectors from the same forward pass. Compare them:

dot     = (0.450*0.10) + (0.150*0.20) + (0.250*0.10) + (0.075*0.10)
        = 0.0450 + 0.0300 + 0.0250 + 0.0075 = 0.1075
|mean|  = sqrt(0.2025 + 0.0225 + 0.0625 + 0.005625) = sqrt(0.293125) = 0.5414
|CLS|   = sqrt(0.01 + 0.04 + 0.01 + 0.01) = sqrt(0.07) = 0.2646
cosine  = 0.1075 / (0.5414 * 0.2646) = 0.1075 / 0.1433 = 0.75

A cosine of 0.75 between two summaries of the same sentence from the same model. They are not interchangeable. Pool one side of your system with mean and the other with CLS and you have introduced a systematic error that no test will report as an error — only as worse results.

Step 3 — what padding does to mean pooling

Now embed the same sentence in a batch where the longest item is 8 tokens, so 4 pad positions are appended. Suppose pad positions return near-zero vectors, [0.00, 0.00, 0.00, 0.00], and you naively average over all 8 positions:

dim 0: 1.80 / 8 = 0.225
dim 1: 0.60 / 8 = 0.075
dim 2: 1.00 / 8 = 0.125
dim 3: 0.30 / 8 = 0.0375

unmasked mean = [0.225, 0.075, 0.125, 0.0375]

That is exactly half of the correct vector — every component scaled by 4/8. Note what happens next: cosine similarity ignores magnitude, so cosine against another vector is unchanged by uniform scaling, and this bug can hide indefinitely in a cosine-based system. But it does not hide if the index uses raw inner product or Euclidean distance, where a halved vector ranks differently; nor does it hide when pad vectors are not exactly zero, in which case the direction shifts too and the amount of shift depends on the batch. This is why "use the attention mask" is not a stylistic preference.

Step 4 — dilution as the span grows

Add a second, unrelated sentence to the same span — "the otter swims" with constructed token vectors averaging to [0.05, 0.85, 0.10, 0.40] — and mean-pool the combined span. Averaging the two sentence-level means:

combined = [(0.450+0.05)/2, (0.150+0.85)/2, (0.250+0.10)/2, (0.075+0.40)/2]
         = [0.250, 0.500, 0.175, 0.2375]

Now compare a billing query whose constructed vector is [0.90, 0.10, 0.20, 0.00]:

vs the invoice-only span:
dot = (0.90*0.450)+(0.10*0.150)+(0.20*0.250)+(0.00*0.075) = 0.405+0.015+0.05+0 = 0.470
|query| = sqrt(0.81+0.01+0.04+0) = sqrt(0.86) = 0.9274
cos = 0.470 / (0.9274 * 0.5414) = 0.470 / 0.5021 = 0.94

vs the combined span:
dot = (0.90*0.250)+(0.10*0.500)+(0.20*0.175)+(0.00*0.2375) = 0.225+0.05+0.035+0 = 0.310
|combined| = sqrt(0.0625+0.25+0.030625+0.05640625) = sqrt(0.39953) = 0.6321
cos = 0.310 / (0.9274 * 0.6321) = 0.310 / 0.5862 = 0.53

Adding one irrelevant sentence dropped the match from 0.94 to 0.53 in this constructed example. The relevant sentence is still fully present in the chunk — nothing was deleted — but the vector moved, because a vector is an average and averages move. Scale that to a chunk containing thirty sentences on eight topics and you have the mechanism behind "our retrieval returns chunks that technically contain the answer but score below the noise". Chunk size is not a storage decision; it is a signal-to-noise decision, and 06-02 treats it as one.

The bi-encoder vs cross-encoder line

This is worth pinning down now because it explains why retrieval pipelines have two stages.

The reason embeddings dominate retrieval despite being less accurate than cross-encoders is entirely about precomputation: you can embed a million chunks overnight and then answer a query with one forward pass plus a nearest-neighbour lookup. A cross-encoder would need a million forward passes per query. 07-06 and 07-07 build the two-stage pipeline that gets both properties.

Objective 1.8 — "Select and use models to create text embeddings" is the direct claim, and the select half of that verb is precisely a granularity decision: you cannot select an embedding model without knowing whether you need one vector per token or one per passage. Objective 1.4 — "Curate and embed content datasets for RAGs" makes the same demand from the data side, because "embed a content dataset" means choosing what unit gets a vector. Objective 1.6 brings in vector databases, which store one vector per record and therefore force the question.

The blueprint design note for this lesson calls this "the single most consequential confusion in applied retrieval", and that is a claim about consequence rather than frequency: the confusion is silent. No component raises an error when you index a whole document as one vector, or when you pool a base encoder that was never trained for it. The system runs, returns results, and is worse than it should be with no signal saying why.

Question phrasings to expect

• Architecture-to-task matching. "Which model type would you use for named-entity recognition?" — an encoder producing token-level representations. "Which for retrieving relevant passages?" — a sentence-embedding / bi-encoder model. This is the same matching skill 04-03 drills for BERT vs GPT vs T5.
• Granularity in a RAG scenario. "A team indexes each 50-page PDF as a single vector and retrieval quality is poor. What is the most likely cause?" The answer names the span being too long — chunk the documents.
• Bi-encoder vs cross-encoder. "Why is a cross-encoder unsuitable for first-stage retrieval over a million documents?" Because it cannot precompute document representations; each query requires a pass per candidate.
• Pooling. A question may simply ask what mean pooling is, or note that CLS and mean pooling give different vectors. Depth needed is identification, not implementation.
• "Sentence embeddings are just averaged word embeddings" — true or false? False as a description of good sentence-embedding models: they are pooled and fine-tuned on pairs. Averaging is the naive approach they outperform.
• Vector-database record design. "What does one row in a vector index correspond to in a RAG system?" One chunk and its embedding, plus metadata — not one document, and not one token.

Distractor families

That is one way to build one, and it is the weak way. Averaging static word vectors (word2vec, GloVe) gives you a span vector with two structural losses: no context, so bank contributes the same vector in every sentence, and no word order, so "the dog bit the man" and "the man bit the dog" produce identical vectors. Averaging a contextual encoder's token vectors fixes the first problem — attention has already mixed context into each token vector — but not the ordering problem entirely, and it leaves the comparability problem untouched.

A modern sentence-embedding model does pool, often by mean pooling, but the pooling is only the last step. What makes it work is that the whole network was fine-tuned with a contrastive objective on pairs, so the pooled output is directly optimised for cosine comparison. The right mental model is: pooling is the plumbing, pair-based training is the product. If a question asks whether sentence embeddings are "just averaged word embeddings", the intended answer is no, with that reason.

Chunks of roughly paragraph scale are the usual answer, and the reasoning matters more than any specific number.

You are trading two failure modes against each other. Chunks that are too small retrieve the correct sentence stripped of the context needed to answer with it — the generator sees "this does not apply in the second case" with no indication of what the second case is. Chunks that are too large dilute the vector until the relevant sentence stops driving the geometry, exactly as the §4 arithmetic showed. Between those, there is a band where the chunk is a self-contained thought.

Three practical qualifications. First, respect document structure where it exists: splitting on headings, list items, or paragraph boundaries beats splitting every N characters, because a chunk that ends mid-sentence is a chunk whose embedding is partly noise. Second, overlap between adjacent chunks cheaply reduces the chance that the one sentence you needed sits astride a boundary. Third, the unit you embed and the unit you return to the model need not be identical — you can embed a small precise chunk and return it together with its neighbours or its parent section, which gets specificity in retrieval and context in generation. 06-02 and 07-08 develop all three. Anyone who gives you a universal chunk size without asking what your documents look like is guessing; the correct move is to test two or three settings against your own eval set, which is what 03-04 teaches you to build.

A bi-encoder embeds the query and the passage independently and compares the two vectors; a cross-encoder feeds the query and passage into a model together and outputs a single relevance score. Everything else follows from that.

Because a bi-encoder handles each side alone, the corpus side can be computed in advance and stored in a vector index — the query then costs one embedding call plus a nearest-neighbour search, which is why bi-encoders scale to millions of documents. Because a cross-encoder needs both texts in the same forward pass, nothing can be precomputed; scoring one query against a million passages means a million forward passes.

The pay-off for that cost is accuracy: in a cross-encoder, every query token can attend to every passage token, so it can detect the fine-grained matching that two independently produced vectors cannot express. Hence the standard two-stage design — a bi-encoder retrieves a shortlist of maybe 50 candidates from everything, and a cross-encoder reranks those 50 (07-07). When you see "embedding model", think bi-encoder; when you see "reranker", think cross-encoder.

Mechanically yes — both are d-dimensional vectors from the same model, so the arithmetic runs. Meaningfully no. The two live at different granularities and, in a pair-tuned model, the pooled output has been shaped by an objective the individual token vectors were never subject to. A cosine between a token vector and a pooled span vector is a number without an interpretation.

The same caution applies to the more tempting version of this mistake: comparing a pooled vector from one model against a pooled vector from another. Identical dimensionality does not imply a shared space, as 03-01 established. Any comparison must be between vectors from the same model, the same version, and the same pooling and prefix conventions — which is the operational reason 12-12 treats an embedding-model change as a full corpus migration rather than a configuration tweak.

• Granularity is the whole distinction. Token embeddings are one vector per token, shape (n, d). Sentence and document embeddings are one vector per span, shape (d,). Same model family, different unit, different jobs.
• Pooling converts token vectors to a span vector — mean, CLS, or max. Mean pooling must respect the attention mask, or padding dilutes short texts by a batch-dependent amount.
• Pooling alone is not enough. A base encoder pretrained on masked-language modelling was never asked to make span vectors comparable, and its pooled vectors crowd together. Purpose-built sentence-embedding models add contrastive fine-tuning on pairs, and that is what makes cosine comparison work.
• The query side and the corpus side must use identical treatment — same model, same version, same pooling, same prefixes. Mismatches degrade silently and are never reported as errors.
• Symmetric and asymmetric objectives are different products. Sentence-to-sentence tuning suits dedup and clustering; short-query-to-long-passage tuning suits search. Picking the wrong one is a real quality loss.
• Longer spans dilute. The constructed worked example dropped a match from 0.94 to 0.53 by adding a single unrelated sentence. A vector's width is fixed regardless of input length, so a whole-document vector regresses toward a bland centroid.
• One row in a vector index is one chunk, with its embedding and its metadata — not one document and not one token.
• Token-level tasks need token-level outputs. NER, POS tagging, and extractive QA all label positions, and pooling has thrown positions away.
• Bi-encoder for retrieval, cross-encoder for reranking. The bi-encoder wins on precomputation and scale; the cross-encoder wins on accuracy over a shortlist.
• Exam depth is identity plus matching: which granularity for which task, why whole-document vectors fail, and why a cross-encoder cannot do first-stage retrieval.

You now know what an embedding is and which granularity your task needs. What you cannot yet do is walk into a decision meeting, look at a list of candidate embedding models, and defend a choice — because the criteria are not obvious from a model's name and several of them only become visible when you go looking. Maximum sequence length silently truncates your chunks. Dimensionality multiplies straight into index memory and query latency. The symmetric-versus-asymmetric split decides whether short queries work at all. And a model trained on general web text may simply not know your domain's vocabulary.

Next: 03-03 turns those four into an explicit selection procedure with the memory arithmetic worked out — max sequence length against your chunk size, dimension against your index budget, training objective against your query shape, and domain vocabulary against your corpus. Then 03-04 shows you how to check by hand whether the model you chose actually works on your data, before any metric exists to tell you.