Encoder-only vs decoder-only vs encoder-decoder models (BERT, GPT, T5)

All three are built from the same transformer blocks you met in 04-01: multi-head self-attention, a position-wise feed-forward network, residual connections, layer normalisation, and a position signal from 04-02. They differ in two choices, and everything else follows.

Choice one: what each position is allowed to attend to.

• Encoder-only — every position attends to every position, in both directions. No mask. A token at index 3 sees index 9. This is bidirectional self-attention, and it is why an encoder's representation of a word is informed by the whole sentence, including the part after it.
• Decoder-only — every position attends only to itself and earlier positions. Causal mask applied. Index 3 cannot see index 4. This is what makes next-token prediction a well-posed problem.
• Encoder-decoder — a bidirectional encoder stack processes the input, then a causally masked decoder stack generates the output while also attending to the encoder's output through a third attention type, cross-attention. The decoder's self-attention is masked; its cross-attention over the encoder output is not.

Choice two: what the model was pretrained to predict.

• Masked language modelling (MLM) — hide a fraction of the input tokens and train the model to reconstruct them from both sides. BERT's objective. It requires bidirectional context to make sense, and it produces excellent representations. It does not train the model to continue text.
• Causal language modelling (CLM) — predict the next token from all previous tokens, at every position simultaneously. GPT's objective, and the one 01-01 introduced. It is exactly the task the model will be asked to perform at inference.
• Sequence-to-sequence denoising / span corruption — corrupt the input (drop spans, shuffle, mask) and train the model to emit the original as a separate output sequence. T5's span-corruption and BART's denoising objectives. It trains both reading and writing, of different sequences.

The single most useful compression of the whole lesson: the mask decides whether a model can generate; the objective decides what it is good at; the pair together decides which tasks are even available. A 70-billion-parameter encoder-only model still cannot write you a paragraph, and a small decoder-only model can. Size is orthogonal.

L1 — The intuition, in three sentences

An encoder is a reader. It takes text in, looks at all of it at once, and hands you back a rich understanding of each token and of the whole — from which you can attach a small classifier head, or pool into an embedding.

A decoder is a writer. It takes what exists so far and produces the next thing, over and over. Because it may only look left, it can be run forward into text that does not exist yet.

An encoder-decoder is a translator in the general sense: a reader bolted to a writer, where the writer is allowed to consult the reader's notes at every step. Use it when the output is a different sequence from the input rather than a continuation of it.

L2 — The attention pattern, drawn

A five-token sequence, showing which positions each row may attend to. ● means allowed, · means masked out.

ENCODER — bidirectional self-attention
        k1  k2  k3  k4  k5
  q1     ●   ●   ●   ●   ●
  q2     ●   ●   ●   ●   ●
  q3     ●   ●   ●   ●   ●
  q4     ●   ●   ●   ●   ●
  q5     ●   ●   ●   ●   ●

DECODER — causally masked self-attention
        k1  k2  k3  k4  k5
  q1     ●   ·   ·   ·   ·
  q2     ●   ●   ·   ·   ·
  q3     ●   ●   ●   ·   ·
  q4     ●   ●   ●   ●   ·
  q5     ●   ●   ●   ●   ●

ENCODER-DECODER — decoder cross-attention over encoder output
        (encoder positions e1..e4, all visible at every decode step)
        e1  e2  e3  e4
  q1     ●   ●   ●   ●
  q2     ●   ●   ●   ●
  q3     ●   ●   ●   ●

Three things are visible in that picture and hard to unsee afterwards:

1. The encoder's matrix is full — hence maximum context per token, hence the best representations, hence embeddings and classification.
2. The decoder's matrix is triangular — hence a valid next-token target at every position, hence generation.
3. Cross-attention is full over the encoder's positions even inside a decoder — the writer sees the entire source at every step, while still not being allowed to see its own future.

L2 — Encoder-only, end to end

Pipeline: tokens → embeddings + position → N bidirectional transformer blocks → per-token output vectors → a small task-specific head.

The head is the part people skip and the part that makes the architecture usable:

Notice what is absent: no output vocabulary loop, no next-token sampling, no decode step. There is nowhere for free text to come from. Encoder-only models do not generate. They score, tag, span-select, and embed.

Note also that extractive QA is genuinely available to an encoder — it selects a span from the passage — while abstractive QA, where the answer is written in new words, is not. That distinction is a favourite exam trap and it is exactly the extractive/abstractive divide 06-01 and summarisation work depend on.

L2 — Decoder-only, end to end

Pipeline: tokens → embeddings + position → N causally masked blocks → per-token output vectors → a projection to vocabulary size → a probability distribution over the next token at every position.

Training and inference differ in an important way that trips people up:

• During training, the whole sequence is processed in one parallel pass. The causal mask means position t's prediction only used positions < t, so all positions can be scored simultaneously against their true next tokens. One pass, thousands of supervised predictions. This efficiency is a large part of why decoder-only pretraining scaled.
• During inference, generation is serial. Token t+1 cannot be computed until token t has been chosen, because it becomes part of the input. This is autoregressive generation, 04-04, and it is why output length drives latency in a way input length does not.

Because the pretraining task (predict the next token) is the same shape as almost every downstream request (produce text), decoder-only models turned out to be extraordinarily general: classification becomes "answer with one word," summarisation becomes "write a summary," translation becomes "translate this." That generality — not superior architecture — is why decoder-only models dominate the current generative landscape.

L3 — Encoder-decoder, and what cross-attention adds

Pipeline: source tokens → encoder stack → source representations (computed once). Then, for each output step: target tokens so far → masked self-attention → cross-attention against the source representations → feed-forward → next-token distribution.

Cross-attention is ordinary attention with one asymmetry: the queries come from the decoder, the keys and values come from the encoder. The writer asks "what in the source is relevant to what I am writing right now," and gets a fresh answer at every step and in every layer.

What that buys, concretely:

• A dedicated, fully bidirectional reading of the source. The encoder gets to look at the source in both directions without any masking compromise, which is the ideal condition for understanding it.
• Clean separation of source and target. The model never confuses "text I was given" with "text I am producing" — they live in different stacks with different roles. A decoder-only model handles both in one undifferentiated stream and must infer the boundary from format.
• Source representations computed once. The encoder runs a single time per request regardless of how long the output becomes.

That is the architectural case for why encoder-decoder models are the classical answer to translation and abstractive summarisation: those tasks are transductions — one whole sequence in, a different whole sequence out — and the architecture was designed for exactly that shape.

L3 — Why decoder-only models took over anyway

If encoder-decoder is the better fit for transduction, why is nearly every headline model decoder-only? Three reasons worth being able to state, at the level the exam pitches:

1. The pretraining objective is universal. Next-token prediction can be run on any text at all, with no pairing, no corruption scheme, and no task design. That makes the data supply effectively the whole web.
2. Everything can be phrased as continuation. "Translate the following to French:" turns a transduction into a continuation. One model serves generation, summarisation, translation, and classification alike without task-specific heads or fine-tuning, which is what makes zero-shot and few-shot prompting from 05-01 possible in the first place.
3. Simpler serving. One stack, one cache, one decode loop. 12-05's KV cache applies straightforwardly.

None of that makes encoder-decoder obsolete. It remains the natural design where the mapping is fixed and the output is short relative to a long input, and encoder-only remains the right and cheaper answer for classification and embeddings. The exam tends to reward the classical mapping — BERT for understanding, GPT for generation, T5/BART for translation and summarisation — while a good candidate can also say why a decoder-only model can do all three.

This is the highest-value single asset in the lesson. Learn it cold.

And the pretraining objectives on their own, since they are examined as a separate discrimination:

Read the third and fourth columns together and the whole design space snaps into place: you cannot use bidirectional context and also learn to generate from the same objective, because seeing the future makes predicting it trivial. MLM buys representation quality by giving up generation. CLM buys generation by giving up right-hand context. Seq2seq denoising gets both by putting them in different stacks. That is the entire trade, and it is the deepest thing in this lesson.

Four requirements as an exam would phrase them. Work each one by asking two questions in order: does the output need to be newly written text? and is the output a different sequence from the input, or a continuation of it?

Task A — "Route incoming support tickets into one of twelve categories, with 30,000 labelled historical tickets available."

Output is a label, not text. One sequence in, one label out. No generation needed. → Encoder-only. Fine-tune a BERT-family encoder with a classification head on the 30,000 examples. This will typically be cheaper to serve and faster than prompting a large generative model, and the labelled data is exactly what a supervised head wants. A decoder-only LLM could do it via prompting and might be the pragmatic choice with no labelled data — but with 30,000 labels and a fixed label set, the encoder is the textbook answer and the keyed one.

Task B — "Extract every drug name and dosage mentioned in clinical notes."

This is named-entity recognition: one label per token, spans identified in the source. Output is a selection from the input, not new text. → Encoder-only, token classification head. The bidirectional context matters here in a way you can name: deciding whether "Novo" begins a drug name depends on the token after it.

Task C — "Translate 200,000 product descriptions from English into German, preserving the source meaning exactly."

Output is a different sequence in a different language, same information. Classical transduction. → Encoder-decoder is the textbook match, and translation is the canonical encoder-decoder task. A decoder-only instruction-following model is also a completely reasonable production choice today, and if the exam offers both, read the stem: if it names T5 or BART, or describes "a model trained on paired source and target sequences," the encoder-decoder is intended.

Task D — "Build an internal assistant that answers policy questions in prose, using retrieved company documents."

Output is newly written prose, in an open-ended conversational setting, grounded in retrieved context. → Decoder-only, inside a RAG pipeline. Note what else this needs: an encoder-based embedding model to index and retrieve the documents. Which is the point of the worked set — a real system usually contains more than one architecture, an encoder for retrieval and a decoder for generation, and describing it as "an LLM app" hides that.

The two-question decision procedure, generalised:

Does the deliverable include newly written text?
├── No  → encoder-only  (classify · tag · span-select · embed)
└── Yes
    └── Is the output a transduction of a source sequence
        (translate / summarise / rewrite one given document)?
        ├── Yes, and the mapping is fixed → encoder-decoder is the classical fit
        └── No, it is open-ended continuation or chat → decoder-only

A caution on version-sensitivity. Which specific model is best for a task changes constantly, and the boundary between "classically encoder-decoder" and "done with a prompted decoder-only model in practice" has moved and will keep moving. What does not change is the architectural claim: an unmasked encoder-only stack cannot generate, and a causally masked stack can. Build your exam answers on the architecture claim, not on a leaderboard.

Architecture-to-task matching is named explicitly in the course's drill emphasis and sits in the highest-frequency reported topic tier. It serves objective 1.3 — build LLM use cases such as retrieval-augmented generation, chatbots, and summarisers — because choosing the wrong architecture makes a use case impossible rather than merely suboptimal. It serves objective 1.8 — select and use models to create text embeddings — because embeddings come from encoders, and knowing that is the difference between a working retrieval pipeline and a confused one. The blueprint's suggested-reading list names the BERT paper directly, which puts encoder-only pretraining inside examinable scope.

"Encoder vs decoder" also appears on the standing list of confusable pairs that a multiple-choice exam is expected to exploit, alongside WordNet vs word2vec and BLEU vs ROUGE. Confusable pairs are where a well-constructed distractor lives.

Question phrasings to expect:

• "Which architecture is most appropriate for sentiment classification?" → encoder-only (BERT-family).
• "Which architecture is most appropriate for text generation?" → decoder-only (GPT-family).
• "Which architecture is most appropriate for machine translation or abstractive summarisation?" → encoder-decoder (T5, BART).
• "BERT is pretrained with which objective?" → masked language modelling.
• "GPT is pretrained with which objective?" → causal / next-token language modelling.
• "Why can BERT not be used directly to generate text?" → it is bidirectional with no causal mask and was not trained to predict a next token; it emits representations, not a next-token distribution.
• "Which component allows a decoder to condition on an encoded source sequence?" → cross-attention.
• "A team needs to produce embeddings for a vector database. Which family?" → encoder-only.
• "A scenario describes tagging entities in documents. Which family?" → encoder-only, token classification.

Distractor families, and why each is wrong:

The single most common real-world version of this error, worth naming: a retrieval system built with a decoder-only model as the embedder, chosen for brand familiarity. It is architecturally coherent — you can pool hidden states from anything — and it is usually the wrong tool, because the model was never trained for the similarity objective the retrieval step needs. 03-03 and 07-02 handle the consequences.

Two independent reasons, and a strong answer names both.

Reason one: no causal mask. BERT's self-attention is bidirectional, so every position sees every other position. Ask such a model to predict the token at position 5 while position 5 is in its input and the task is degenerate — the answer is on the page. Generation requires that the model has never seen what it is about to produce, which is precisely what the causal mask from 04-01 guarantees and what BERT deliberately lacks.

Reason two: the objective and the output shape. BERT was trained to reconstruct a scattered subset of masked tokens given both sides of each one. It emits per-token representations intended for a task head. There is no autoregressive decode loop, no notion of "the sequence so far," and no training signal that ever asked it to extend text rightward. You can bolt machinery on and coax token-by-token filling out of an MLM model — research has explored this — but it is not what the architecture is for and it is not the exam's answer.

The clean formulation: bidirectional context and next-token prediction are mutually exclusive within one objective, because seeing the future makes predicting it trivial. BERT chose context. GPT chose prediction. T5 got both by using two stacks.

Cross-attention lets the decoder consult the encoded source at every generation step and in every decoder layer.

Mechanically it is the same scaled dot-product attention from 04-01, with the operands split across stacks: queries are projected from the decoder's current hidden states, keys and values from the encoder's output representations. There is no causal mask on cross-attention — the whole source is legitimately available, since it was given as input and contains no future output to leak.

Why this matters in practice:

• Alignment happens where you can see it. Translation requires knowing which source words a target word corresponds to. Cross-attention is where that correspondence is computed, which is why early neural-translation interpretability work leaned on cross-attention maps. Treat those maps as hypotheses rather than proof, for the same reason 04-01 warned about attention maps generally.
• The source is encoded once. Cross-attention keys and values are computed from the encoder output a single time per request and reused at every decode step. A long input therefore costs one encoder pass, not one per output token.
• It is the structural difference from decoder-only. Strip cross-attention from an encoder-decoder and you have two disconnected stacks. It is the bridge, and its absence is exactly what defines decoder-only.

Three attention types now exist in your vocabulary, and the exam can ask you to distinguish them: encoder self-attention (bidirectional, within the source), decoder self-attention (causally masked, within the output so far), and cross-attention (decoder queries, encoder keys and values, unmasked).

It depends on whether the summary is extracted or written, and this is the discrimination the question is really testing.

Extractive summarisation selects sentences or spans from the source and concatenates them. That is a scoring-and-selection problem: encode the document, score each sentence for importance, take the top ones. Encoder-only is sufficient, the output is guaranteed to be verbatim source text, and provenance is trivially exact.

Abstractive summarisation writes new sentences that may use words absent from the source. That requires generation. The classical answer is encoder-decoder — BART and T5 are canonical summarisation models, and abstractive summarisation is the canonical encoder-decoder task alongside translation. A prompted decoder-only model does this routinely in production too.

The trade-off is real and it is the reason the question is interesting: abstractive output reads better and compresses harder, and it can be unfaithful — it can assert something the source does not support. Extractive output is guaranteed faithful to the wording and often reads badly and compresses poorly. Faithfulness failure in abstractive summarisation is a hallucination-adjacent problem, and it is why summarisation evaluation gets its own treatment; ROUGE's recall orientation in 09-06 and faithfulness measurement in 09-07 both exist because of it.

For the exam: if the stem says "abstractive," generation is required. If it says "extractive," an encoder suffices. If it says neither and names T5 or BART, the intended answer is encoder-decoder.

• Encoder-only model — a transformer with fully bidirectional self-attention that produces representations; BERT is canonical. Classification, NER, extractive QA, embeddings.
• Decoder-only model — a transformer with causally masked self-attention trained on next-token prediction; GPT is canonical. Generation.
• Encoder-decoder model — a bidirectional encoder plus a causally masked decoder joined by cross-attention; T5 and BART are canonical. Translation, abstractive summarisation.
• Bidirectional self-attention — unmasked attention where every position sees every position; maximum context per token, incompatible with next-token training.
• Cross-attention — attention whose queries come from the decoder and whose keys and values come from the encoder output; the bridge in an encoder-decoder.
• Masked language modelling (MLM) — pretraining by hiding a fraction of tokens and reconstructing them from both sides. BERT's objective.
• Causal language modelling (CLM) — pretraining by predicting the next token from the left context at every position. GPT's objective.
• Span corruption / denoising — sequence-to-sequence pretraining where corrupted input is mapped to the original as a separate output sequence. T5 and BART.
• Task head — the small trained layer on top of an encoder that converts representations into labels, spans, or pooled vectors.
• Extractive vs abstractive — output selected verbatim from the source versus newly written; the first can be done by an encoder, the second requires generation.
• Transduction — a task mapping one whole sequence to a different whole sequence, the shape encoder-decoder models were designed for.

1. The mask is the fork. Unmasked bidirectional attention gives you an encoder that reads; causal masking gives you a decoder that writes. Everything else in the taxonomy follows from that one choice.
2. BERT → classification, NER, sentiment, extractive QA, embeddings. GPT → generation. T5 and BART → translation and abstractive summarisation. Memorise this triple; it is asked directly.
3. The objectives line up with the masks. MLM needs bidirectional context and cannot generate. CLM must not have bidirectional context and can. Seq2seq denoising trains reading and writing in separate stacks.
4. Encoder-only models genuinely cannot generate free text, at any size. This is architectural, not a matter of tuning or scale.
5. Cross-attention is the encoder-decoder bridge — decoder queries against encoder keys and values, unmasked, at every step and layer. Its absence is what "decoder-only" means.
6. Size is orthogonal to family. An architecture mismatch is never fixed by a bigger model.
7. Decoder-only won on generality, not on fit. Universal pretraining data plus "everything is a continuation" beat a better-fitting architecture for transduction.
8. Real systems mix families. A RAG pipeline is normally an encoder for retrieval and a decoder for generation; describing it as "an LLM" hides half the design.
9. Extractive versus abstractive decides whether you need generation at all — and abstractive output can be unfaithful, which is why summarisation evaluation is its own problem.

You now know that only a causally masked stack can generate, and that at inference it must proceed one token at a time. What you have not seen is the loop itself: what the model actually emits at each step (a probability distribution over the entire vocabulary, not a word), how one token gets chosen from that distribution, how the chosen token is fed back in, and what makes the whole thing stop. That loop is where latency comes from, where the KV cache becomes necessary, and where every source of non-determinism in an LLM system enters.

Next: 04-04 walks the autoregressive generation loop step by step — prefill and decode, the distribution at each step, feeding the output back as input, and the stopping conditions.