Self-attention explained: why transformer context length costs quadratically

Self-attention is a layer that takes a sequence of token vectors and returns a sequence of token vectors of the same shape, where each output vector is a weighted mixture of all the input vectors — with the weights computed from the inputs themselves. "Self" means the sequence attends to itself rather than to some other sequence. "Attention" means the weights are not fixed; they are computed per token, per position, per input.

The consequence worth memorising: after one self-attention layer, the vector sitting at position 4 is no longer the vector for the word at position 4. It is a blend, dominated by whichever positions position 4 found relevant. In the sentence "the bank raised its rates," the vector at "bank" has absorbed signal from "rates" and become a financial-institution vector. In "the bank was muddy and steep," the same starting vector absorbs "muddy" and becomes a riverside vector. Same input word, different output vector, decided entirely by context. This is the mechanical definition of a contextual embedding, and it is the thing static embeddings from 03-01 structurally cannot do.

Three properties fall out of that design, and each one is exam-relevant:

• Parallelism. Every position's output can be computed at the same time as every other position's, because none of them wait on a previous step. This is what makes transformer training GPU-friendly in a way recurrence never was.
• Constant path length. Any two tokens are one hop apart, whatever the distance between them. Relating token 1 to token 2,000 costs the same number of layers as relating token 1 to token 2.
• Order-blindness. Because the output is a weighted sum over positions, shuffling the input tokens shuffles the outputs identically but changes nothing about the values. A bare attention layer literally cannot tell "dog bites man" from "man bites dog." Positional encoding exists to fix precisely this, and it is treated in §7 here and in full in 04-02.

L1 — The intuition of query, key, and value

Think of attention as a lookup in a soft, differentiable dictionary. Every token emits three things derived from its own vector:

• a query — "here is what I am looking for"
• a key — "here is what I am, advertised for others to match against"
• a value — "here is the content I will contribute if you attend to me"

To compute the output for one token, you take that token's query and compare it against every token's key. Where a query and a key are similar, that source token gets a large weight. Then you sum every token's value, weighted by those similarities. That is it. The query/key/value split exists so that "what I am searching for" and "what I offer" can be different functions of the same vector — a pronoun can look for a noun without itself looking like a noun.

The similarity measure is the dot product, which is the exact operation you built in 01-04. Attention is a dot-product similarity search executed inside the model on every forward pass, which is why so much of this course reduces to that one piece of linear algebra.

L2 — The five steps in order

For a single attention head over a sequence of n tokens:

1. Project. Multiply the input token vectors by three learned weight matrices to get Q, K, and V. Those matrices are parameters — part of the weight count you met in 01-02.
2. Score. Compute the dot product of every query with every key. The result is an n × n matrix of raw scores. Row i holds "how much does token i care about each token."
3. Scale. Divide the scores by the square root of the key dimension. This is why the standard form is called scaled dot-product attention, and the reason is in §9 below.
4. Normalise. Apply softmax along each row, turning raw scores into weights that are non-negative and sum to 1 per row. Now each row is a probability-like distribution over source positions.
5. Mix. Multiply the weight matrix by V. Each output row is the weighted average of all value vectors.

The output has the same shape as the input — the batch, sequence, hidden-size triple from 01-03 is preserved. That shape invariance is what lets you stack dozens of these layers.

L2 — Multi-head attention and why one head is not enough

A single attention pattern has to serve every kind of relationship at once: syntactic agreement, coreference, topical relatedness, positional adjacency. One softmax distribution per token cannot express all of those simultaneously, because attending strongly to the subject means attending weakly to everything else.

Multi-head attention runs several attention operations in parallel, each with its own smaller Q/K/V projections, then concatenates the results and projects them back to the hidden size. Different heads learn to specialise. The exam-level statement is: multi-head attention exists so the model can attend to several different relationships in the same layer, rather than being forced to average them into one. The number of heads and the per-head dimension are configuration choices that vary by model and by model version — do not memorise a specific count as though it were universal.

L3 — Where attention sits inside a transformer block

Attention is not the whole architecture. One transformer block, in order:

input token vectors
   │
   ├─► multi-head self-attention ──┐
   │                               ▼
   └────────────────────────► add (residual) ─► layer norm
                                        │
   ┌────────────────────────────────────┘
   ├─► position-wise feed-forward network ──┐
   │                                        ▼
   └───────────────────────────────► add (residual) ─► layer norm
   │
   ▼
output token vectors (same shape as input)

Two things in that diagram do real work and are routinely omitted from summaries:

• The residual connections. Each sub-layer's output is added to its input rather than replacing it. This is what keeps gradients flowing through a deep stack — the same vanishing-gradient problem you met in 01-06, solved structurally. It also means a token's original identity is never fully overwritten by the attention mixture; the mixture is added on top.
• The feed-forward network. Attention moves information between positions. The feed-forward network transforms information within a position, applied identically and independently to every token. A stack of attention layers with no feed-forward layers would only ever be re-mixing the same content. The division of labour — attention communicates, feed-forward computes — is the cleanest one-line summary of a transformer block, and the feed-forward layers hold a large share of the parameters.

L3 — The causal mask, and how the same mechanism becomes generative

The attention described so far is bidirectional: token 4 attends to tokens 1 through n, including tokens 5 and later. That is exactly what you want for understanding a complete input, and it is what an encoder does.

It is fatal for generation. If a model predicting token 5 can see token 5, training is trivial and useless — the model learns to copy the answer, and at inference time, when the future does not exist, it collapses. The fix is the causal mask (also called the look-ahead mask or autoregressive mask): before the softmax, every score where the key position is later than the query position is set to negative infinity. Softmax turns negative infinity into a weight of zero. Position 4 can now attend only to positions 1 through 4.

This is a masking operation, not a different mechanism. Same projections, same dot products, same softmax. The mask alone is the difference between "reads the whole sentence at once" and "can only look left" — which is the difference between BERT and GPT, the subject of 04-03. If you take one L4-depth detail from this lesson, take this one: causal masking is implemented by adding negative infinity to the disallowed scores before the softmax, so the model's ability to see the future is removed by construction rather than by convention.

This comparison is where the exam most often lives, because "why did attention replace recurrence" is a question with a crisp, memorable answer.

Read the last three rows together and the trade is obvious. Recurrence gets order and linear cost for free but pays with a serial bottleneck and a lossy memory. Attention gets direct access and parallelism but must be told about order and pays quadratically for length. Neither is strictly better; the transformer won because the thing it is bad at (cost) is buyable with hardware, and the thing recurrence was bad at (serialisation) is not.

The distractor family to recognise: exam options that credit attention with "faster inference than an RNN per token" or "lower memory than an RNN" are describing the wrong advantage. Attention's advantages are parallel training and long-distance relationships. Its memory profile at long context is worse, not better.

Take the eight-token sequence the cat sat on the mat by the door. Count the tokens as eight for the illustration; real tokenisation from 02-03 may differ.

Step 1 — the score matrix. Every token produces a query; every token produces a key. The score matrix has one row per query and one column per key: 8 × 8 = 64 scores, per head, per layer.

Step 2 — double the sequence. Sixteen tokens gives 16 × 16 = 256 scores. The sequence doubled; the score count went up fourfold. That is quadratic growth stated concretely, and it is worth saying out loud once: 2× the tokens, 4× the attention scores.

Step 3 — scale the illustration up. Suppose — and this is a constructed example, not a measured configuration of any shipped model — a model with 32 layers and 32 attention heads per layer, processing 2,048 tokens.

scores per head per layer = 2,048 × 2,048        =  4,194,304
heads per layer                                   =         32
scores per layer          = 4,194,304 × 32       = 134,217,728
layers                                            =         32
attention scores per forward pass                = 4,294,967,296

Roughly 4.3 billion score entries for one forward pass over one sequence, in a constructed configuration. Now double the input to 4,096 tokens and re-run the arithmetic: the per-head figure becomes 16,777,216, and the total lands near 17.2 billion — four times as much, for twice the text.

Step 4 — the part people miss. Those score matrices are not just compute; during training they are materialised in memory because the backward pass needs them. The n × n term therefore appears in your memory bill as well as your FLOP bill. This is why a training run that fits comfortably at one sequence length can fail outright at twice that length even though the model has not changed a single parameter — the weights are the same size, and the activations are not.

Step 5 — what this does not mean. Quadratic growth in the attention term does not mean the whole model is quadratic in practice at short sequences. The feed-forward layers and the projections scale linearly with n and are multiplied by large hidden dimensions, so at modest context lengths they can dominate the total. The attention term wins as n grows, because quadratic eventually beats linear. The honest statement is: attention introduces a quadratic term that comes to dominate at long context, not "transformers are quadratic at all lengths."

The decision rule that generalises: when a cost surprise appears as you lengthen inputs, suspect the quadratic attention term before you suspect the model size. Model size is fixed; length is the variable you control, and it is the one with the exponent on it.

One caution that the exam rewards and blog posts ignore: a long list of published methods — sparse attention, sliding-window attention, linear-attention approximations, memory-efficient exact-attention kernels — reduce or reorganise this cost, and modern serving stacks use them. Their availability and their exact behaviour are version-sensitive and vary by model and by library. The correct associate-level position is to know that the quadratic term is the target of a whole optimisation literature, not to claim a specific method is in a specific product.

Transformer architecture sits in the highest-frequency reported topic tier for NCA-GENL, alongside prompt engineering, tokenization, text-generation parameters, and NVIDIA NIM. The blueprint objectives it serves are the ones about building LLM use cases and reading research papers to identify emerging LLM trends — objective 1.3 and objective 1.7 — plus the fundamentals objective 1.5. The exam's own suggested-reading list names the original transformer paper, which means the architecture is not optional background; it is examinable content the blueprint points you at directly.

Calibration matters as much as coverage here. Candidate reports converge on the finding that detailed attention mathematics was overkill and did not appear. What did appear is conceptual and comparative: what attention does, why it displaced recurrent models, what the components are called, and what each one is for. So the depth ceiling is set on purpose. You should be able to answer all of the following cold, and none of them require an equation:

Question phrasings to expect:

• "Which mechanism allows a transformer to relate two distant words without passing through the words between them?" → self-attention.
• "What is the primary advantage of the transformer over the LSTM for training on large corpora?" → parallel processing of the sequence, and direct modelling of long-distance relationships.
• "Why do transformers require positional encoding?" → because self-attention is permutation-invariant and carries no order information on its own.
• "In scaled dot-product attention, what are the three learned projections called?" → query, key, value.
• "What does multi-head attention provide over single-head attention?" → several relationship patterns represented in the same layer, rather than one averaged pattern.
• "A team doubles the input sequence length and inference cost rises far more than double. What explains this?" → the quadratic cost of the attention score matrix.

Distractor families, and why each is wrong:

That last row is the single most productive confusion in this module. The transformer architecture is parallel over the sequence. Autoregressive generation is serial over output tokens, because token 5 does not exist until token 4 has been chosen. Both statements are true, they are about different things, and 04-04 exists to keep them apart.

Positional encoding is a position-dependent signal added to (or otherwise combined with) each token's embedding before attention, so that the attention mechanism can distinguish "first" from "fifth." It exists for one reason and one reason only: self-attention is permutation-invariant, so without it, word order is invisible to the model.

Convince yourself with the mechanism. The output at each position is a weighted sum over positions, and the weights come from dot products between projected token vectors. Nothing in a dot product depends on where its operands sat in the input. Permute the inputs and you permute the outputs identically — the set of tokens determines everything. "Dog bites man" and "man bites dog" contain the same multiset of tokens, so a bare attention stack assigns them the same content, differently ordered. For a bag-of-words task from 02-06 that might be tolerable. For language it is fatal.

The canonical fix in the original transformer is a fixed sinusoidal pattern: a deterministic function of position and dimension, computed rather than learned, added to the token embedding. The alternative, used widely, is a learned positional embedding — a lookup table over positions trained like any other parameter. Later families of methods encode relative position, or inject position into the attention computation itself rather than into the input vectors; rotary position embeddings are the best-known of these. Which scheme a given model uses is a per-model, per-version fact and not something to assert from memory.

What you should hold firmly is the functional comparison:

And hold the exam-facing consequences:

• Positional encoding is added to the token stream, not substituted for it. The vector entering attention encodes identity and position.
• It is the reason a maximum sequence length exists at all in some designs — a learned table has a last row. This is one of two independent reasons a context window is bounded, and the other is memory. 04-06 separates them properly.
• "Positional encoding" and "positional embedding" are used near-interchangeably in the literature; the meaningful distinction is fixed-computed versus learned, not the noun.

For the purposes of this module, positional encoding's job is to make self-attention's mechanism undeniable: once you know the layer is a weighted sum over an unordered set, you cannot forget why order has to be injected. 04-02 takes the topic on its own terms.

Two reasons, and both are structural rather than incremental.

Parallel training. An RNN's hidden state at step t is a function of the state at step t−1, so a sequence of 1,000 tokens is 1,000 dependent steps that cannot be spread across a GPU's parallel units. Self-attention computes all positions simultaneously as matrix multiplications, which is exactly the workload GPUs are built for. When the bottleneck on model quality is how much data you can push through in a given wall-clock budget, an architecture that parallelises over the sequence does not just train faster — it makes training at a previously impossible scale feasible at all. Everything about foundation models follows from that.

Direct long-distance relationships. In an LSTM, information from token 1 reaches token 500 only by surviving 499 gated updates, each of which can attenuate or overwrite it. Gating mitigates this; it does not remove it. Self-attention gives token 500 a direct edge to token 1 with a weight the model chooses. Anaphora, long-range agreement, document-level consistency, and "the instruction was at the top of the prompt" all depend on this.

A useful third framing: attention converts sequence modelling into set modelling plus an explicit position signal. That is a strange trade on its face — you throw away order and then add it back — but it is what buys the parallelism, and the exam's phrasing of the advantage ("non-sequential processing, long-distance relationships") is exactly this trade stated as a benefit.

Because unscaled dot products grow with dimension, and large scores make softmax degenerate.

Mechanically: a dot product sums the products of many component pairs. As the key dimension gets larger, the typical magnitude of that sum gets larger too. Feed very large values into a softmax and it saturates — nearly all the weight collapses onto the single largest score and the rest go to nearly zero. Two bad things follow. The layer becomes a hard argmax lookup instead of a soft mixture, losing the ability to blend several sources. And the gradient through a saturated softmax is close to zero, so the layer trains slowly or not at all.

Dividing by the square root of the key dimension counteracts the growth so scores stay in a range where softmax remains informative. That is the entire story, and it is the appropriate depth: the exam may name "scaled dot-product attention" and expect you to know the scaling exists to keep softmax well-behaved. It will not ask you to prove the variance result.

It prevents a position from attending to any later position, which is what makes a decoder able to generate.

The implementation is one line of intent: before the softmax, set every score whose key index is greater than its query index to negative infinity, so softmax assigns it zero weight. Nothing else about the layer changes.

Three consequences worth carrying forward:

1. It defines the model family. Bidirectional (unmasked) self-attention gives you an encoder — good at understanding a complete input, useless at generating without extra machinery. Causally masked self-attention gives you a decoder — able to produce token by token. This is the fault line 04-03 is built on.
2. It makes the training objective honest. With the mask in place, predicting token t from tokens 1 through t−1 is a real task at every position simultaneously, which is why causal language-model training is so efficient: one forward pass yields a prediction target at every position.
3. It is what makes caching possible. Because position t's attention never depends on anything after t, the keys and values computed for earlier tokens stay valid as generation proceeds and can be reused instead of recomputed. That reuse is the KV cache, and it is the single mechanism that explains most of modern LLM serving behaviour. 12-05 owns it.

Neither framing survives contact with the mechanism, and the honest answer is more useful than either.

What attention computes is a learned, context-dependent similarity between projected representations. Because the projections are learned, "similar" does not mean "similar in meaning" — it means "matching in whatever way this head found useful for reducing prediction loss." Published interpretability work has found heads whose patterns look strikingly like syntactic relations, and it is a fair statement that attention patterns often correlate with linguistic structure. It is not a fair statement that a head is a parser, that the model has grammar rules, or that you can read a reliable explanation of an output off an attention map. Attention weights are frequently over-interpreted; treat a heat map as a hypothesis, not evidence.

For exam purposes: attention learns which tokens to combine; it is not given a grammar, and its weights are not an explanation of the output. Transparency and explainability have their own instruments and their own lesson, 13-06.

• Self-attention — a layer in which every position in a sequence computes a weighted mixture of all positions in the same sequence, with weights derived from the inputs.
• Query, key, value (Q/K/V) — three learned projections of each token vector: what it seeks, how it advertises itself, and what it contributes.
• Scaled dot-product attention — the standard attention form: dot-product scores divided by the square root of the key dimension, softmaxed, then used to weight values.
• Attention score matrix — the n × n array of query-key similarities; the origin of the quadratic cost.
• Multi-head attention — several attention operations in parallel with separate projections, concatenated and projected back, so one layer can express several relationship types.
• Residual connection — adding a sub-layer's input to its output, which preserves gradient flow and the token's original content.
• Position-wise feed-forward network — the per-token transformation that follows attention in every block; attention moves information between positions, this transforms it within one.
• Causal mask — setting future-position scores to negative infinity before the softmax so a position cannot attend to later positions; what makes a decoder autoregressive.
• Permutation invariance — the property that reordering inputs reorders outputs without changing them; why positional encoding is necessary.
• Positional encoding — a position-dependent signal combined with token embeddings so the model can distinguish order; fixed-sinusoidal, learned-absolute, or relative/rotary.
• Quadratic attention cost — the n² growth in attention compute and attention activation memory as sequence length n grows.

1. Self-attention lets every token read every token directly, in one parallel step. That is the whole innovation, and both of its famous advantages — parallel training and long-distance relationships — are restatements of it.
2. The mechanism is a soft dictionary lookup. Queries match against keys by dot product; softmax turns matches into weights; values are averaged by those weights. Q/K/V exist so "what I seek" and "what I offer" can differ.
3. Multi-head attention buys multiple simultaneous relationship patterns, not multiple devices.
4. Attention communicates between positions; the feed-forward network computes within a position. Residual connections and layer norm keep a deep stack trainable. All four parts are in every block.
5. The cost is quadratic in sequence length, in FLOPs and — during training — in stored activations. Double the tokens, quadruple the attention work. This is the mechanical reason context is the most expensive dial you control.
6. A bare attention layer is order-blind. Positional encoding is not an optimisation or a nicety; without it, "dog bites man" and "man bites dog" are the same input.
7. One mask separates the two great model families. Unmasked bidirectional attention understands; causally masked attention generates.
8. Depth ceiling, deliberately. Know what each component does, why it exists, and how it fails. Candidate reports are consistent that the derivations do not appear — spend the time you save on architecture-to-task matching and decoding parameters, which do.

You now know why order must be injected into an order-blind mechanism, and you have seen the three families of schemes that do it. What you have not yet seen is what those schemes look like from the outside: why a fixed sinusoidal pattern can be evaluated at a position the model never trained on, why a learned table cannot, and why that difference is one of the two reasons a model has a maximum sequence length at all.

Next: 04-02 takes positional encoding as its own subject — absolute versus relative, fixed versus learned, and the practical consequence when a prompt runs past the positions a model was trained to represent.