Encoder-Only vs. Decoder-Only vs. Encoder-Decoder: Matching Architecture to Task

Identity statement: an architecture family is defined by two coupled choices — which direction attention is allowed to look, and what training objective that direction of attention makes trainable. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Encoder-only (BERT). Sees the whole context in both directions; trained by masking tokens and predicting them (MLM). ... Decoder-only (GPT). Masked/causal attention only sees tokens to the left; trained to predict the next token (CLM). ... Encoder-decoder (T5). The encoder reads the input; the decoder generates output conditioned on it."

Encoder-only models apply no restriction to the attention mask at all — every token's Query is free to attend to every other token's Key, both before and after it in the sequence, which is why this is called bidirectional attention. That full visibility is exactly what makes masked language modeling (MLM) trainable: during training, some fraction of input tokens are replaced with a special mask token, and the model is trained to predict the original token at each masked position using context from both directions simultaneously — genuinely using words that come after the blank, not just before it, since nothing in the attention mask prevents it. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): BERT is "Great for producing embeddings and for classification." A model trained this way ends up producing representations well suited to understanding a fixed piece of text — the whole input is visible when the representation is built — but the training objective never teaches it to produce a next token in sequence, one token at a time, which is why encoder-only models are not the tool of choice for open-ended generation.

Decoder-only models impose a causal mask on attention: token i's Query is only allowed to attend to Keys at positions ≤ i — itself and everything before it, never anything after. This is a direct restriction on the same QKᵀ mechanism M1-01 derived; the mask simply sets the attention score for any "future" position to negative infinity before softmax, so those positions receive exactly zero weight after softmax, regardless of what their raw content-based similarity would otherwise have been. That restriction is precisely what makes causal language modeling (CLM) — predicting the next token given everything so far, and only everything so far — both trainable and, crucially, consistent with how the model will actually be used at inference time: generating token 501 has never seen tokens 502 onward, during training or during generation, so there is no mismatch between how the model learned and how it is deployed. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): GPT-family decoder-only models are "the generation workhorse."

Encoder-decoder models run a full bidirectional encoder over the input sequence first, producing a set of context-aware representations for every input token, and then run a causal decoder that generates the output sequence token by token — with one addition beyond a plain decoder: at each decoder step, in addition to attending causally over its own previously generated tokens, the decoder also attends (without a causal restriction) over the encoder's output representations, a mechanism often called cross-attention. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "The encoder reads the input; the decoder generates output conditioned on it. Natural fit for seq2seq tasks." This is the architecture that lets a model read an entire source sentence bidirectionally before committing to any part of the translation, while still generating the translation autoregressively, one token at a time, exactly as a decoder-only model would.

L1 — Intuition

Imagine three different ways of reading and responding to a sentence with a word blanked out: "The chef ___ the soup before serving it." An encoder-style reader can look at the whole sentence, blank included, and use both "the chef" before the blank and "before serving it" after the blank to guess "tasted" or "seasoned" — full context, both directions, one guess. A decoder-style reader, generating word by word left to right, would never see "before serving it" at the time it has to choose the word after "chef" — it only ever has "The chef" available, and must predict the next word from that alone, exactly the situation autoregressive generation is always in. An encoder-decoder reader gets both: it can read the full source sentence bidirectionally first (as an encoder would), and then generate its response one word at a time, using everything it read plus everything it has generated so far (as a decoder would) — reading with full context, producing with the necessary left-to-right discipline generation requires.

L2 — Mechanism

The mechanical difference between all three families lives entirely in the attention mask and, for the encoder-decoder case, in the addition of a cross-attention step — nothing about the underlying scaled dot-product attention formula from M1-01 changes. An encoder's self-attention mask is entirely unrestricted: every position can attend to every position, full stop. A decoder's self-attention mask is lower-triangular in the sequence-position sense — position i can attend to positions 1 through i, and every entry above that diagonal in the attention-score matrix is masked to negative infinity before softmax, guaranteeing a zero attention weight there regardless of content. An encoder-decoder stacks an unrestricted encoder self-attention block, a causal decoder self-attention block, and a third, unrestricted cross-attention block where the decoder's Queries are compared against the encoder's output Keys and Values rather than the decoder's own — letting the decoder pull information from anywhere in the source sequence while still generating its own output causally.

Training objective follows directly from what the mask permits. MLM is only trainable where bidirectional context exists to predict a masked token from, which is only the encoder's unrestricted mask. CLM is trainable under a causal mask specifically because "predict the next token from everything so far" is exactly what a causal mask leaves visible — nothing more, nothing less. An encoder-decoder's training objective is typically framed as a general text-to-text mapping — input sequence in, output sequence out — trained by having the decoder predict each output token causally, conditioned on the encoder's bidirectional read of the input via cross-attention.

L3 — The exam-relevant edge case: why an encoder cannot simply "run causally instead"

The tempting but wrong intuition is that an encoder-only model could be made to generate text by just applying a causal mask to it at inference time, since the underlying attention mechanism is identical across all three families. This fails for a specific, trainable-objective reason, not an architectural impossibility: an encoder-only model's weights were trained under an unrestricted mask, where every attention weight the model learned assumes it can see both directions — the model's Wq, Wk, and Wv were tuned during training to produce useful representations under that assumption, for every layer, for the entire training run. Applying a causal mask to those same weights at inference time changes what each layer sees without retraining the weights to expect it, and the resulting representations are not what the model learned to produce; they are a mismatch between training-time assumptions and inference-time constraints. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md) states the pairing rule this edge case supports directly: "Match architecture to objective — encoder → MLM/bidirectional; decoder → CLM/causal. A question pairing 'BERT' with 'next-token generation,' or 'GPT' with 'masked language modeling,' is wrong." The fix is not a masking trick applied after training; it is choosing the architecture whose training objective already matches the task, or retraining a model under the objective the task actually needs.

[GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md) states this exact three-row comparison as the domain's own reference table, matching architecture, example, attention direction, objective, and best-fit use case row for row.

Constructed scenario, illustrative only. Take the sentence "The chef tasted the soup before serving it," and trace what each architecture family can and cannot do with it.

Task A — fill in a blank: "The chef ___ the soup before serving it."

Encoder-only (bidirectional, MLM-style):
  Attention mask for the blank position: UNRESTRICTED
  Can attend to: "The", "chef" (before)  AND  "the", "soup", "before", "serving", "it" (after)
  Prediction uses BOTH directions -> plausible guess: "tasted" or "seasoned"

Decoder-only (causal, CLM-style), asked to predict what comes after "The chef":
  Attention mask at this position: CAUSAL (only positions <= current)
  Can attend to: "The", "chef"  ONLY -- nothing after this point exists yet at generation time
  Prediction uses ONLY the left context -> plausible guess: "was" or "quickly" (whatever
  continuation the causal-only context makes likely -- it never gets to see "the soup" at
  this step, because at true generation time those words do not exist yet)

Task B — translate the sentence to French, word by word, as an encoder-decoder would:

Step 1 (Encoder pass, bidirectional, runs ONCE over the full source sentence):
  Every source token's representation is built using the WHOLE source sentence,
  both directions -- "chef" attends to "soup", "before", "serving", "it" too.

Step 2 (Decoder pass, causal, runs ONE STEP AT A TIME):
  Decoder step 1: generate "Le"     -- causal self-attn sees nothing yet; cross-attn sees ALL of step 1's encoder output
  Decoder step 2: generate "chef"   -- causal self-attn sees "Le"; cross-attn again sees ALL of step 1's encoder output
  Decoder step 3: generate "a"      -- causal self-attn sees "Le chef"; cross-attn again sees ALL encoder output
  ... continues one token at a time, decoder self-attention growing causally,
      cross-attention always reaching back to the SAME full bidirectional encoder output.

The pattern to extract: the encoder-decoder gets the best of both worlds specifically because the two masking regimes are applied to two different computations — the encoder's one-time, whole-sentence, bidirectional read, and the decoder's step-by-step, causally growing, autoregressive write — bridged by cross-attention rather than forcing one masking regime to serve both jobs. Notice, too, what stays constant across every decoder step in Task B: the encoder pass runs exactly once, regardless of how many output tokens the decoder eventually produces, and every one of those output steps reaches back into that same, unchanging encoder output through cross-attention — there is no re-encoding of the source sentence partway through generation, which is why an encoder-decoder's encoder cost does not grow with output length the way its decoder cost does.

Constructed scenario, illustrative only. Four tasks, each pointing at a different family for a reason worth stating explicitly rather than by pattern-matching the task name to a familiar model.

Task: sentiment classification of a fixed product review. The entire review is available at once — there is no generation step, only a single yes/no or star-rating judgment over text that already fully exists. An encoder-only model fits natively: its bidirectional attention can use the whole review, and its training objective (MLM, or a classification head fine-tuned on top of encoder representations) never needed to predict a next token in the first place. A decoder-only model can be adapted to this task too, but it is spending an autoregressive-generation-shaped tool on a task that never needed generation, and its causal restriction means later words in the review cannot inform its understanding of earlier words the way an encoder's bidirectional attention naturally allows.

Task: open-ended chatbot response generation. There is no fixed target text to fill in; the model must produce novel token sequences one at a time, conditioned on a conversation history that keeps growing. This is decoder-only's native mode — CLM training is literally "predict the next token given everything so far," and nothing about an encoder-only model's MLM training ever taught it to produce a coherent multi-token continuation from scratch, one step at a time, because MLM never trains that skill at all.

Task: document summarization, source and summary in different lengths. This is a genuine sequence-to-sequence mapping — a long input, a shorter output, and the output needs to be grounded in a full, bidirectional read of the input before any of it is generated. An encoder-decoder fits this natively: encode the source bidirectionally once, then generate the summary causally, with cross-attention reaching back into the full source at every generation step. A decoder-only model can also be trained to summarize (concatenate source and target, train causally over the whole thing), and many production systems do exactly that — but the encoder-decoder's clean separation between a one-time bidirectional read and a step-by-step causal write is the architecture the objective was explicitly designed around.

Task: extracting a dense embedding vector to compare two customer support tickets for similarity. This has no generation step at all — the goal is a fixed-size vector representation, not new tokens. Either an encoder-only or a decoder-only model can produce a usable embedding (a claim M1-04 develops in full), but an encoder-only model's bidirectional training gives every token's final representation access to the whole input when the embedding is pooled, which is typically the more natural fit for a task that has no notion of "before" or "after" within the ticket text — there is no autoregressive generation step whose left-to-right constraint would need to be respected.

⭐ THE EARNED INSIGHT Encoder, decoder, and encoder-decoder are not three different attention mechanisms — they are one mechanism wearing three different masks, and the mask alone determines what training objective is even trainable. A question about which architecture fits a task is never really a question about model size or named brands; it is a question about which mask matches which objective, and the wrong-pairing distractor exists precisely because it is easy to remember "BERT" and "GPT" as brand names while forgetting that the mask, not the name, is what does the actual work.

The three families are not equally common in the current generation of large, widely deployed models, and understanding why is itself an exam-relevant piece of context rather than trivia. ⚠️ UNVERIFIED: the specific market-share proportions of encoder-only, decoder-only, and encoder-decoder models among today's most widely used large language models are not stated in the source material and should not be quoted as a specific figure — what is verifiable, from the mechanism itself, is the structural reason decoder-only architectures scaled so well for general-purpose use.

A decoder-only model's training objective, causal language modeling, has one property the other two families do not share as cleanly: it can be applied to essentially any text at all, with no need for a separately designed masking scheme (as MLM requires) or a paired input-output structure (as seq2seq training requires) — every document in a training corpus is simply "predict the next token, everywhere, causally," which makes CLM trivially scalable to enormous, heterogeneous web-scale corpora without any task-specific data curation. That same causal-only structure also happens to be the exact structure needed for open-ended generation, instruction-following, and multi-turn conversation — three of the most commercially significant capabilities for a large-scale deployed model — without any architectural modification between "how it was trained" and "how it is used." Encoder-only models, by contrast, remain extremely well suited to their native tasks (embeddings, classification) but were never trained to generate open-ended continuations at all, and encoder-decoder models, while natively strong at seq2seq tasks, carry the added architectural complexity of two separate stacks bridged by cross-attention — complexity that pays off for translation and summarization specifically, but that a general-purpose conversational system does not automatically need.

This is not a claim that encoder-only or encoder-decoder architectures are obsolete or inferior — M1-04 will show encoder-only models remain a first-class source of embeddings, and encoder-decoder models remain the natively correct fit for genuine seq2seq tasks. It is a claim about why one family became the default choice for general-purpose, open-ended, conversational deployment specifically: the training objective and the deployment use case are the same shape, with no translation step between them, in a way that is not automatically true for the other two families.

[GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): architecture families are covered under objective 1.2, "(b) Architecture Families," within Domain 1 (LLM Architecture), which carries 6% of the NCP-GENL blueprint — the smallest domain, but, as the domain's own framing states, "the conceptual base for everything else." The source material states the standing trap explicitly rather than leaving it implicit: "Match architecture to objective — encoder → MLM/bidirectional, decoder → CLM/causal. A question pairing BERT with next-token generation, or GPT with masked language modeling, is wrong." That sentence is close enough to exam phrasing that it is worth treating as close to verbatim guidance on what the keyed answer will look like.

Expect a direct pairing item: "Which architecture/objective pairing is correct?" with "BERT — masked language modeling" as the keyed answer against distractors like "GPT — masked language modeling" or "BERT — causal language modeling," each scrambling which family goes with which objective. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md) states this exact item as its own self-check question 2. Expect a task-to-architecture item: a scenario names a task (classification, open-ended generation, translation) and asks which family fits best, testing whether you reason from attention direction and training objective rather than from having memorized which model name goes with which task by rote. Expect a mechanism item about cross-attention specifically, testing whether you understand that an encoder-decoder's decoder is not merely "a decoder-only model with an encoder attached," but a causal self-attention block plus a distinct cross-attention block reaching into the encoder's bidirectional output.

What the distractors typically look like

The standard traps in this lesson's style are: swapping which architecture goes with which training objective (the domain's single most named trap); describing an encoder-only model as capable of autoregressive generation without qualification, when its training objective never taught that skill; and describing an encoder-decoder's decoder half as identical to a decoder-only model, omitting the cross-attention mechanism that lets it condition on the encoder's bidirectional read — a real component missing from the description, not an added mechanism invented as a false claim.

It is also worth noting, briefly, that "encoder-decoder" architectures with cross-attention are not unique to text. NVIDIA's own multimodal-generative-AI material covers a structurally different but conceptually related encoder-decoder design — "U-Net architecture: encoder-decoder structure and skip connections" — for image-generation backbones, where the "decoder" reconstructs a full-resolution image from a compressed bottleneck representation rather than generating a sequence of tokens causally, and "skip connections" (not cross-attention) are the mechanism that carries detail from encoder to decoder. The two are worth keeping distinct rather than assuming one framing transfers to the other: this lesson's encoder-decoder is a token-sequence architecture built from the same masked-attention mechanism as the rest of Domain 1, while a U-Net's encoder-decoder is a convolutional, spatial-resolution architecture solving a different kind of reconstruction problem entirely — the shared vocabulary ("encoder," "decoder," "skip") describes two different mechanisms in two different exam contexts.

Yes, and this is common in practice — a decoder-only model's final-token representation (or a pooled representation across all tokens) can be fed into a classification head, exactly as an encoder's representation would be. The distinction this lesson emphasizes is not "which family is capable of which task" in an absolute sense, but which family's training objective and attention direction natively fits a task's structure. A decoder-only model doing classification is using causal, left-to-right attention for a task that has no inherent left-to-right structure — it works, often well, but it is not exploiting the same full-context advantage an encoder-only model's bidirectional training was built around.

Because the decoder's self-attention block is causally masked specifically so it only ever attends to its own previously generated tokens — mixing the encoder's tokens into that same causal self-attention would either break the causal guarantee (if the encoder's tokens were treated as unmasked, mixed in with causally-masked decoder tokens, in one confused attention operation) or force the encoder's bidirectionally-computed representations to be artificially causally restricted, throwing away the exact advantage of having encoded them bidirectionally in the first place. A separate cross-attention block keeps the two masking regimes cleanly apart: decoder self-attention stays causal over the decoder's own output so far, and cross-attention stays unrestricted over the encoder's already-finished bidirectional representations, letting the decoder draw on the full source at every step without compromising either guarantee.

Nothing prevents building a single model that offers multiple masking modes — some research and production architectures do exactly this, applying a bidirectional (or "prefix") mask over part of the input and a causal mask over the rest, within one set of weights trained to handle both regimes. What this lesson's three-family framing captures is the dominant, cleanly separated design space the exam tests: encoder-only, decoder-only, and encoder-decoder as three distinct, individually trained objectives, each with its own consistent masking regime throughout training. A hybrid-masking model is a real and legitimate design choice, but it is a fourth, more complex point in the same design space, built by combining the same underlying masking mechanism this lesson derived rather than by inventing a new one — and the exam's own scope, per the source material's own three-row table, is these three named families, not every possible hybrid built from the same components.

• The three families differ in attention mask, not in the underlying attention formula — M1-01's softmax(QKᵀ/√dₖ)·V is unchanged; what changes is which positions are allowed to attend to which.
• Encoder-only is bidirectional and trained with MLM — best for understanding tasks: embeddings, classification, NER.
• Decoder-only is causal and trained with CLM — the native architecture for autoregressive generation.
• Encoder-decoder combines both, bridged by cross-attention — a bidirectional one-time read of the input, a causal step-by-step write of the output, with the decoder reaching back into the encoder's output at every generation step.
• The domain's single most named trap is pairing the wrong architecture with the wrong objective — BERT with generation, or GPT with masked prediction, are both wrong for the same underlying reason: the training objective and the attention mask do not match.
• An encoder cannot simply be run causally at inference time to generate text — its weights were trained under an unrestricted mask, and applying a causal mask afterward changes what each layer sees without retraining it to expect that change.

This lesson established what each architecture family can see and what training objective that visibility enables, but stopped short of one specific, testable skill the exam names directly: pulling a usable numeric representation — an embedding — out of a trained model, from either family, and comparing two such embeddings meaningfully. M1-04 picks that up directly: how embeddings are extracted from encoder and decoder models alike, why query and document embeddings must share one vector space to be comparable at all, and why cosine similarity, not raw distance, is the standard comparison for this exam's purposes.