Extracting Embeddings from Encoder and Decoder Models, and Comparing Them with Cosine Similarity

Identity statement: embedding extraction is the act of taking one or more of a transformer's internal hidden-state vectors — the same per-token representations that exist at every layer regardless of what the model was trained to output — and combining them into a single fixed-length vector that represents an entire input sequence, a span, or a single token, for use in a downstream comparison or retrieval task.

Every transformer layer, in every architecture family, produces one vector per input token at every layer — this is just what M1-01 and M1-02 already described attention and the feedforward sub-layers as computing. Nothing about producing those per-token vectors is specific to a particular training objective; MLM and CLM are both just different ways of training the weights that produce these vectors, not different claims about whether the vectors themselves exist. The practical question extraction has to answer is which layer's hidden states to use (commonly the final layer, though intermediate layers are sometimes preferred for specific tasks) and how to collapse a whole sequence's worth of per-token vectors into one vector, since most downstream comparisons need a single fixed-length representation per input rather than one vector per token.

[GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "You can pull embeddings from both encoder and decoder models — e.g., the final hidden state (or a pooled representation) for a sequence." The two named options in that sentence — "the final hidden state" and "a pooled representation" — are not describing two different architectures; they are two different pooling strategies available to either family, and which one is more natural depends on how each family's training objective shapes its own final-layer representations.

L1 — Intuition

Think of a transformer's final layer as producing one labeled index card per token — each card summarizing what that token "means" given the entire context the model was allowed to see while producing it. An encoder produces its cards using full bidirectional context for every token, so a card near the middle of the sequence already reflects information from both earlier and later tokens. A decoder produces its cards causally, so a card at position i only reflects tokens 1 through i — the very last card in the sequence is special, because it is the only one that has seen the entire input, exactly the way a person reading left to right only fully understands a sentence once they reach its last word. Turning a whole stack of index cards into one embedding means choosing how to combine them: average every card together, or just take the last one, or a handful of other strategies covered below.

L2 — Mechanism

Encoder pooling most commonly uses one of two strategies. Mean pooling averages the final-layer hidden state vectors across every token position in the sequence, producing one vector whose every dimension is the average of that dimension's value across all tokens — a strategy that treats every token's contribution as roughly equal and tends to produce a broad, sequence-wide summary. CLS-token pooling, used by BERT-style models specifically, reads out the hidden state at a special token prepended to every input during training (conventionally called [CLS]) — because that token attends bidirectionally over the entire sequence in every layer, its final-layer representation is trained, via whatever downstream objective was fine-tuned on it, to summarize the whole sequence in one vector by design, rather than by post-hoc averaging.

Decoder pooling works differently because a causal model's tokens do not have equal access to context — only the last token in the sequence has attended, even indirectly, to every other token. Last-token pooling exploits this directly: read out the final-layer hidden state at the last token position, since it is the one position in a causal model whose representation has necessarily incorporated the entire preceding sequence through the accumulated effect of causal self-attention at every layer. Mean pooling is also usable on a decoder's hidden states, and is sometimes preferred when the goal is a representation less dominated by whatever the model happened to attend to most heavily right at the sequence's end — but last-token pooling has the specific advantage of being exactly the representation the model's own causal training objective was continuously optimizing to be maximally informative, since predicting the next token is a function of precisely that final hidden state.

L3 — The exam-relevant edge case: why "decoders can't produce embeddings" is specifically wrong

The trap named directly in the source material deserves the precise mechanical reason it is wrong, not just the assertion that it is. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Embeddings are not limited to encoder models — decoders produce usable representations too." The misconception likely arises from a true but narrower fact: a decoder's causal mask means any single token position before the last one has an incomplete view of the sequence, so pooling from an early or middle position in a decoder would indeed produce a representation missing later context — and if you only ever considered pooling from an arbitrary middle token, you might wrongly generalize that decoders cannot produce full-sequence embeddings at all. But the last token specifically does not have this limitation: by the time causal self-attention reaches the final position, it has attended, layer after layer, to every token before it, and that final hidden state is exactly as complete a summary of the sequence as an encoder's pooled representation is, just built through causal accumulation rather than through simultaneous bidirectional attention. This is precisely why "extract embeddings from decoder models" is not a workaround or an unusual trick — it is a standard technique with a specific correct pooling choice (last-token, or a mean pool), and treating decoders as embedding-incapable confuses a limitation of one pooling choice (an early or middle token) with a limitation of the entire architecture family.

Constructed scenario, illustrative only — real hidden states are learned, not hand-set. Take a 3-token sequence and suppose, hypothetically, its final-layer hidden states (dimension 4, for simplicity) are the following — imagine these came from a decoder's causal forward pass, where each token's vector reflects only itself and the tokens before it:

Token 1 ("The"):    h1 = [0.10, 0.20, 0.05, 0.00]   -- has seen only itself
Token 2 ("cat"):    h2 = [0.30, 0.10, 0.40, 0.10]   -- has seen tokens 1-2
Token 3 ("sat"):    h3 = [0.20, 0.50, 0.30, 0.60]   -- has seen tokens 1-3 (the FULL sequence)

Mean pooling (average across all 3 token vectors, dimension by dimension):
  dim 1: (0.10 + 0.30 + 0.20) / 3 = 0.20
  dim 2: (0.20 + 0.10 + 0.50) / 3 = 0.267
  dim 3: (0.05 + 0.40 + 0.30) / 3 = 0.25
  dim 4: (0.00 + 0.10 + 0.60) / 3 = 0.233
  mean-pooled embedding = [0.20, 0.267, 0.25, 0.233]

Last-token pooling (just read out h3, the final position):
  last-token embedding = [0.20, 0.50, 0.30, 0.60]

The two pooled vectors are meaningfully different, and the difference is not noise — it is exactly the mechanical distinction section 2 described. Mean pooling blends in token 1's vector, which by construction has seen only itself and carries the least sequence-level context of the three; last-token pooling uses only h3, which is the single position in this causal sequence that has actually attended to the whole thing. Neither choice is universally "more correct" — mean pooling can smooth out an unusually dominant final token's idiosyncrasies, while last-token pooling directly matches what the decoder's own causal training objective was optimizing — but the two are different vectors, computed by different rules, and a system that mixes pooling strategies inconsistently across the embeddings it compares will produce meaningless similarity scores, for reasons section 5 makes precise.

This is the requirement the source material states as its own standing caution, and it is easy to underestimate because it sounds almost too obvious to need stating: [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Query and document embeddings must come from the same model / vector space to be comparable." The reason is not a convention or a best practice — it is a mathematical necessity. An embedding vector's individual dimensions have no inherent, fixed meaning; they are whatever a specific model's specific training run happened to arrange them into. Two different models — even two models trained on similar data for a similar objective — will, in general, place semantically similar concepts at different coordinates, in different arrangements, because nothing forces two independently trained models' internal geometries to align. Comparing a vector from model A against a vector from model B is comparing two numbers that happen to occupy the same array position but carry no shared meaning at that position — the comparison will produce a number, but that number reflects the two models' unrelated internal geometries colliding, not any genuine semantic relationship between the underlying texts.

This has a direct, practical consequence for retrieval systems specifically, because retrieval is the setting where "compare a query embedding against many document embeddings" is the entire mechanism. If a document corpus was embedded with one model's checkpoint, and a later system update embeds new queries with a different checkpoint — even a newer, ostensibly better version of what looks like "the same model family" — the comparison between old document embeddings and new query embeddings is not meaningful, because a retrained or fine-tuned checkpoint is, for this purpose, a different model with a different vector space, however similar its name or lineage. The general term for the fix — recomputing an entire corpus's embeddings after a model change — is out of scope for this lesson's architecture focus, but the underlying reason it is necessary is exactly the vector-space requirement stated here: comparability requires shared geometry, and shared geometry requires the same model (or a model deliberately trained to align its space with another's), not merely a similar one.

Identity statement: cosine similarity between two vectors is the cosine of the angle between them, computed as their dot product divided by the product of their magnitudes (Euclidean norms): cosine_similarity(a, b) = (a · b) / (||a|| × ||b||). [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Compare embeddings with cosine similarity (angle-based, magnitude-insensitive) for retrieval, clustering, and deduplication."

The dividing-by-magnitudes step is what makes this metric specifically about direction rather than length. Two vectors pointing in exactly the same direction have a cosine similarity of 1, regardless of whether one of them is twice as long as the other — dividing by both norms exactly cancels out any difference in scale, leaving only the angular relationship. Two vectors at a right angle to each other score 0, and two vectors pointing in opposite directions score −1. This magnitude-insensitivity is not an incidental property; it is the entire reason cosine similarity, rather than raw Euclidean distance, is the standard choice for comparing text embeddings. A longer document tends to accumulate a hidden-state vector with a larger overall magnitude than a short query, purely as an artifact of how much text contributed to producing it, with no necessary relationship to whether the document's meaning is close to the query's meaning — Euclidean distance would penalize that magnitude difference as if it were a semantic difference, while cosine similarity ignores it entirely and asks only whether the two vectors point in a similar direction in the space.

Constructed scenario, illustrative only. Take two embedding vectors, deliberately given different magnitudes to make the magnitude-insensitivity claim concrete rather than asserted:

Vector A (a "query" embedding):     a = [3, 4]
Vector B (a "document" embedding):  b = [6, 8]     -- exactly 2x vector A, same direction
Vector C (a different document):    c = [4, -3]    -- same magnitude as A, different direction

Step 1 -- dot product of A and B:
  a . b = (3*6) + (4*8) = 18 + 32 = 50

Step 2 -- magnitudes (Euclidean norms):
  ||a|| = sqrt(3^2 + 4^2) = sqrt(9+16) = sqrt(25) = 5
  ||b|| = sqrt(6^2 + 8^2) = sqrt(36+64) = sqrt(100) = 10
  ||c|| = sqrt(4^2 + (-3)^2) = sqrt(16+9) = sqrt(25) = 5

Step 3 -- cosine similarity, A vs B:
  cos(a,b) = 50 / (5 * 10) = 50 / 50 = 1.0     -- perfectly aligned, despite B being 2x longer

Step 4 -- dot product and cosine similarity, A vs C:
  a . c = (3*4) + (4*-3) = 12 - 12 = 0
  cos(a,c) = 0 / (5 * 5) = 0.0                  -- orthogonal, despite IDENTICAL magnitude to A

Read the contrast between the two comparisons as the entire point of the metric. Vector B has twice the magnitude of vector A, and yet cosine similarity scores them at a perfect 1.0, because they point in exactly the same direction — a magnitude difference this large produced zero effect on the score. Vector C, by deliberate contrast, has the exact same magnitude as vector A (both norm 5) and yet scores a 0.0, because the two point in orthogonal directions — identical length bought it nothing, because cosine similarity was never measuring length in the first place. A Euclidean-distance-based comparison would have told the opposite story: A and B, despite pointing the same direction, sit far apart in raw distance purely because B is longer; A and C, despite pointing in unrelated directions, might sit deceptively close in raw distance if their magnitudes happen to align. That inversion is exactly why cosine similarity, not raw distance, is the standard tool for comparing embeddings whose magnitude can vary for reasons — text length, pooling strategy, numerical scale of a particular model's outputs — that have nothing to do with meaning.

⭐ THE EARNED INSIGHT An embedding is never comparable in isolation — its meaning is entirely relative to the specific model, pooling choice, and vector space that produced it. "Decoders can't produce embeddings" and "just compare these two vectors" are the same category of mistake wearing different clothes: both treat a vector as if it carried absolute, portable meaning, when every embedding is only ever meaningful relative to the exact function that generated it, applied consistently on both sides of the comparison.

Constructed scenario, illustrative only. Suppose a small support-ticket retrieval system embeds a corpus of 3 past tickets using a decoder-only model with last-token pooling, and later a new engineer, unaware of that choice, embeds an incoming query using the same model but with mean pooling instead — an easy mistake, since both are valid pooling strategies for a decoder in isolation, and nothing about the code failing loudly would flag the inconsistency.

Ticket 1 embedding (last-token pooled):   t1 = [0.9, 0.1, 0.0]
Ticket 2 embedding (last-token pooled):   t2 = [0.1, 0.9, 0.0]
Ticket 3 embedding (last-token pooled):   t3 = [0.0, 0.1, 0.9]

Query embedding, pooled INCONSISTENTLY (mean pooling instead of last-token):
  query_meanpooled = [0.5, 0.5, 0.3]

cosine(query_meanpooled, t1) = (0.5*0.9 + 0.5*0.1 + 0.3*0.0) / (||query|| * ||t1||)
                              = (0.45 + 0.05 + 0.0) / (0.699 * 0.906)
                              ~= 0.50 / 0.633 ~= 0.79

cosine(query_meanpooled, t2) = (0.5*0.1 + 0.5*0.9 + 0.3*0.0) / (0.699 * 0.906)
                              ~= 0.50 / 0.633 ~= 0.79

cosine(query_meanpooled, t3) = (0.5*0.0 + 0.5*0.1 + 0.3*0.9) / (0.699 * 0.906)
                              = (0.0 + 0.05 + 0.27) / 0.633 ~= 0.32 / 0.633 ~= 0.51

Ticket 1 and ticket 2 come out at an identical similarity score against the mean-pooled query, which is a strong signal that something upstream is broken rather than that the two tickets are genuinely equally relevant — mean pooling and last-token pooling are two different functions of the same underlying hidden states, and comparing a vector produced by one function against vectors produced by the other is not the same failure as comparing two different models' spaces, but it produces the same category of symptom: a similarity score that reflects a mechanical mismatch rather than a semantic relationship. The fix is not a different similarity metric — cosine similarity is doing exactly what it is supposed to do here, faithfully reporting the angle between the vectors it was given. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md) is the source of the underlying rule this failure violates. The fix is upstream: re-embed the query with the same pooling strategy — last-token — that produced the ticket embeddings, so that every vector in the comparison was produced by the same function applied consistently, matching the model-level "same vector space" requirement from section 5.

[GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): embeddings are covered under objective 1.3, "(c) Embeddings," within Domain 1 (LLM Architecture) — 6% of the blueprint by weight, but foundational to retrieval and RAG material tested more heavily elsewhere. The domain's own scope note frames this as a hands-on, not merely conceptual, objective: "Objectives explicitly ask you to 'write code to extract embeddings.'" This is a professional-level exam, and this objective in particular rewards having actually implemented pooling logic, not only being able to define what an embedding is.

Expect a source-of-embeddings item testing the standing misconception directly: "From which models can you extract usable embeddings?" with "Both encoder and decoder models" as the keyed answer against distractors like "Encoder-only models only" — [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md) states this exact self-check question. Expect a comparability item describing a scenario where embeddings from two different sources are compared, testing whether you flag the shared-vector-space requirement as the missing prerequisite. Expect a metric item asking what cosine similarity measures or why it is preferred over Euclidean distance for text embeddings, with "angle, magnitude-insensitive" as the keyed property.

What the distractors typically look like

The standard traps in this lesson's style are: restricting embedding extraction to encoder models only, when decoder extraction via last-token pooling is equally valid and explicitly named in the objective; describing embeddings as storing readable text rather than numeric vectors — [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md) states this directly: "embeddings store numeric vectors, not text"; and describing cosine similarity as sensitive to vector length or as interchangeable with Euclidean distance, when its defining property is exactly the opposite.

Not reliably, and this is a direct extension of the shared-vector-space requirement rather than a separate rule. Fine-tuning updates the model's weights, which changes the geometry every subsequent embedding is produced in — even a light fine-tuning pass can shift where concepts land relative to the original checkpoint's arrangement, because the training signal that produced the fine-tuned weights had no obligation to preserve the original space's coordinates. The safest practice is to treat a fine-tuned checkpoint as a distinct model for embedding-comparability purposes and re-embed anything that needs to be compared against its outputs, exactly as you would after switching to an entirely different model.

Because the raw dot product alone is not magnitude-insensitive — two long vectors pointing in only vaguely similar directions can produce a larger raw dot product than two short vectors pointing in a nearly identical direction, purely because of their length, which would make "raw dot product" a poor similarity measure for vectors whose lengths vary for content-unrelated reasons like document length. Cosine similarity is specifically the dot product divided by both vectors' magnitudes, which cancels out length and leaves only the angular relationship — this is precisely why the metric is described as measuring the angle between two vectors, not simply their raw dot product.

It affects performance, not just bookkeeping consistency, though the consistency requirement from section 8 still applies regardless of which strategy performs best for a given task. Because mean pooling and last-token (or CLS-token) pooling are genuinely different functions of the same underlying hidden states, they can emphasize different information — mean pooling tends to produce a representation influenced by every token roughly equally, which can be an advantage when a sequence's meaning is spread evenly across it, while last-token or CLS-token pooling concentrates on whichever single position the model's own training most heavily optimized to be informative, which can be an advantage when that position genuinely does summarize the sequence well. Neither is universally superior; the practical answer is empirical — measure retrieval or classification quality under both strategies for a given model and task — but the two are not interchangeable defaults, and switching between them mid-deployment without re-embedding everything consistently reproduces exactly the mismatch worked through in section 8.

• Embeddings can be extracted from both encoder and decoder models — objective 1.3 requires both, and "decoders can't produce embeddings" is a named exam misconception, not a fact.
• Pooling strategy should match the architecture: CLS-token or mean pooling for encoders; last-token or mean pooling for decoders, because only the last position in a causal sequence has attended to the whole thing.
• Query and document embeddings must come from the same model or vector space to be comparable — different checkpoints, even closely related ones, produce geometrically unrelated spaces.
• Cosine similarity measures the angle between two vectors, not their length — dividing the dot product by both magnitudes cancels out scale differences entirely.
• Embeddings store numeric vectors, not text — a common surface-level misreading the exam tests directly.
• This is a hands-on objective: the exam expects you to have implemented embedding extraction and comparison, not only to define the terms.

This lesson covered how to pull a fixed-length numeric representation out of a model and compare two such representations meaningfully — a task with no generation step at all. The module's final lesson turns to the opposite end of a decoder's job: producing new tokens, one at a time, from the probability distribution the model outputs at every generation step. M1-05 closes out Module 1 by working through greedy decoding, beam search, and the temperature, top-k, and top-p sampling knobs that turn that distribution into an actual sequence of generated tokens — and the standing distinction this lesson's cosine-similarity precision should have primed you for: reshaping a distribution and truncating it are two different operations, not two names for the same thing.