Autoregressive generation: how an LLM produces text one token at a time

Autoregressive means "predicting the next value from previous values of the same series." Applied to text: the next token is predicted from the tokens before it, and the prediction is then treated as one of the tokens before the next prediction. The model's own output becomes its input. That feedback is the entire idea, and the word to attach to it is conditioning: each token is conditioned on every token already present, prompt and generated output alike, with no distinction between them.

Three consequences define the concept, and each is examinable:

• The model emits a distribution, not a word. At every step the output is a score for every entry in the vocabulary — tens of thousands of numbers from 02-02's vocabulary — turned into probabilities by a softmax. Choosing one of them is a separate step with its own rules, which is the subject of 04-05. Keeping the model's output and the selection rule apart is the single cleanest way to think about generation.
• Generation is serial in the output. Token 5 cannot be computed until token 4 has been selected, because token 4 is part of token 5's input. This is a data dependency, not an implementation limitation, and no amount of hardware removes it.
• There is no revision. Once a token is emitted and appended, it is context. The model conditions on it. A model that opens a sentence badly must continue from the bad opening; it cannot back up. This is why a confident-sounding wrong first clause tends to be followed by a coherent elaboration of the wrong thing.

The definition worth reproducing verbatim under time pressure: autoregressive generation produces output one token at a time, each conditioned on all previous tokens, by feeding the model's own output back as input.

L1 — The loop in six lines

1. tokenize the prompt                       →  input token ids
2. run a forward pass                        →  a distribution over the vocabulary
3. select one token from that distribution   →  one token id      (this is decoding)
4. append it to the input                    →  a longer sequence
5. is a stopping condition met?              →  if yes, stop
6. go to 2

Step 3 is where every decoding parameter you have heard of lives — greedy, beam, temperature, top-k, top-p, repetition penalty. Steps 2 and 3 are commonly blurred together as "the model generates text," and separating them is the highest-value clarification in this lesson: the model produces probabilities; the decoding strategy produces text.

L2 — Prefill and decode: two phases with different costs

Real serving stacks split the loop into two phases, and they behave so differently that they are effectively different workloads.

Prefill processes the whole prompt in one parallel forward pass. Every prompt token is handled simultaneously, exactly as in training, because they all already exist. This phase is compute-heavy: many tokens, one pass, large matrix multiplications that keep a GPU busy. The time it takes largely determines time to first token (TTFT).

Decode generates output tokens one at a time. Each step processes a single new token against everything cached so far. This phase is memory-bandwidth-heavy rather than compute-heavy: for each token you must read the model's weights and the accumulated keys and values from memory to do a comparatively tiny amount of arithmetic. The time per step determines inter-token latency, and the number of steps is the number of output tokens.

This table is why "the model is slow" is a useless diagnosis. A slow first token is a prefill problem — usually a long prompt. A slow stream is a decode problem — usually output length, or memory bandwidth contention. 12-10 turns this into proper measurement discipline with TTFT, tokens per second, and p95.

L2 — Why the same tokens are recomputed, and the cache that stops it

Naively, step 2 of the loop re-runs the model over the entire sequence every time. Generating 200 tokens from a 1,000-token prompt would mean a forward pass over 1,001 tokens, then 1,002, then 1,003 — nearly all of it repeating work already done.

The causal mask makes that waste avoidable. Because position t never attends to anything after t, the keys and values computed for the earlier tokens do not change when a new token is appended. They can be stored and reused. That store is the KV cache, and with it each decode step only computes the query, key, and value for the one new token, then attends against the cached keys and values.

Two consequences follow that you should carry into every capacity conversation:

• Decode becomes cheap in FLOPs and expensive in memory. The cache grows with every generated token, for every sequence in the batch, on every layer.
• The KV cache is why LLM inference is described as memory-bound. It is the single mechanism that explains most of modern serving behaviour, and 12-05 owns it in full. 12-07 covers how paged memory management addresses its fragmentation.

For this lesson the point is narrower: the KV cache exists because generation is autoregressive and causally masked. It is not a separate optimisation bolted on; it is the direct consequence of the loop's structure.

L3 — What happens inside one decode step

For a single new token, in order:

1. Embed the newly appended token id and add its position signal, per 04-02.
2. For each layer: project the token to a query, key, and value; append the new key and value to that layer's cache; attend the new query against all cached keys; mix the cached values; run the feed-forward network; apply residuals and layer norm — the block from 04-01.
3. Project to vocabulary. The final hidden vector is multiplied by an output projection to produce one score per vocabulary entry. These raw scores are called logits.
4. Optionally modify the logits. Repetition penalties, logit biases, banned-token masks, and grammar or schema constraints all act here, on the logits, before any probability is computed. This is a fact worth knowing: constrained decoding for JSON output in 05-05 works by zeroing out logits for tokens that would break the schema.
5. Softmax to probabilities, after temperature has scaled the logits if temperature is in play.
6. Select one token id, by whichever decoding strategy is configured.
7. Check stopping conditions. Append and loop, or finish.

Steps 3 through 6 are the pipeline 04-05 dissects. Notice that temperature acts on logits before the softmax while top-k and top-p act on the resulting probabilities — an ordering that explains why they compose the way they do.

L3 — Stopping conditions: four ways generation ends

Generation is a loop, so something must break it. There are four mechanisms, and the exam expects at least the first two by name.

Max tokens is the parameter most often misunderstood, and it appears in the exam's list of text-generation parameters alongside temperature and top-p. It is a budget, not a target. It does not make output longer, it does not make it shorter, and it does not instruct the model to aim for a length. It only says when to give up. If your summaries are truncating, max tokens is too low. If your cost per request is unbounded, max tokens is your only hard defence. And because output length drives decode-phase latency, max tokens is also the parameter that bounds worst-case latency — which makes it a reliability control, not just a formatting one.

That last row matters because beam search is the most common category error in this area. Beam search is not an alternative to autoregressive generation — it is autoregressive generation, run over several partial candidates in parallel with a scoring rule that keeps the best few. Everything in this lesson still applies to it.

The other comparison the exam probes is training versus inference within the same model:

Teacher forcing deserves its name: during training, position t's prediction is conditioned on the true tokens 1 through t−1, not on what the model would itself have produced. That is what makes one pass yield a supervised target at every position, and it is efficient. It also creates a mismatch — at inference the model conditions on its own outputs, mistakes included, so an early error shifts it into territory the training distribution never covered. This is the mechanical basis for why generation can start plausibly and drift, and it is a genuinely satisfying thing to be able to explain.

Prompt: The capital of France is. All numbers below are constructed for illustration — they are not measured outputs from any model, and the vocabulary is fictional and tiny so the arithmetic is readable.

Prefill. Five prompt tokens go through one parallel forward pass. Keys and values for all five positions are computed and cached at every layer. The model emits logits at the final position; the earlier positions' logits are discarded, since we already know what came next there.

Decode step 1. Logits at the last position, softmaxed over an illustrative vocabulary:

token        probability
" Paris"        0.87
" the"          0.04
" located"      0.03
" a"            0.02
" in"           0.02
(all others)    0.02

Greedy selection takes Paris. Note that the distribution covers the entire vocabulary, not five options; the rest simply carry near-zero mass.

Decode step 2. Input is now The capital of France is Paris. One new token is processed against six cached positions.

token        probability
"."             0.61
","             0.18
" and"          0.07
" which"        0.05
(all others)    0.09

Greedy takes ..

Decode step 3. Input is The capital of France is Paris.

token             probability
<end-of-sequence>   0.55
" It"               0.19
" The"             0.09
(all others)        0.17

Greedy takes the end-of-sequence token. Generation stops after three decode steps, because the model predicted its own completion — the desired stopping mechanism.

Now change one thing. Suppose max tokens had been set to 2. Generation would have stopped after Paris. — which happens to read fine. Set it to 1 and you get Paris with no punctuation. Set the prompt to ask for a paragraph with max tokens still at 2 and you get two tokens of a paragraph. The cap is blind to meaning; it counts.

Now count the passes. Three output tokens meant one prefill pass plus three decode passes. Ask the same model for a 500-token answer from the same prompt and you get one prefill pass plus 500 decode passes. The prompt did not change. The work went up roughly 100-fold, and none of the added work can be parallelised away. That is the arithmetic behind the most important operational statement in this lesson: output length, not input length, is what makes generation slow.

Finally, note where the cache grew. After three decode steps, the cache holds keys and values for eight positions per layer instead of five. That growth is per-sequence and per-layer, and it is why concurrency at long output lengths is a memory problem — the setup 12-05 and 12-06 build on.

The decision rule: if the symptom scales with how much text was produced, it is the autoregressive loop. If it scales with what the text says, it is not.

Generation and decoding parameters sit in the highest-frequency reported topic tier for NCA-GENL, and the blueprint's own suggested-reading list names an autoregressive-model reading directly — which puts the concept inside examinable scope rather than in the background. It serves objective 1.3, building LLM use cases such as RAG, chatbots, and summarisers, because every one of those is a generation loop with a budget. It serves objective 1.9 on prompt engineering, because a prompt is the conditioning context the loop consumes. And it serves the Software Development objective 4.4 on identifying the system, hardware, and software components required to meet user needs, because latency and cost characteristics fall directly out of prefill and decode.

The exam is pitched at general level, and reports are consistent that deep mathematics did not appear. What is asked is the mechanism and its consequences.

Question phrasings to expect:

• "How does a decoder-only LLM produce text?" → autoregressively: one token at a time, each conditioned on all previous tokens, with the output fed back as input.
• "What does an LLM output at each generation step?" → a probability distribution over the entire vocabulary, not a word.
• "Why can text generation not be parallelised across output tokens?" → each token depends on the previously selected token, a genuine data dependency.
• "Which parameter limits how much text an LLM will produce in one response?" → max tokens (max new tokens).
• "What normally causes generation to stop on its own?" → the model emits an end-of-sequence token.
• "A request has a long prompt and a short answer. Which latency metric will be worst?" → time to first token, driven by prefill.
• "A request has a short prompt and a long answer. What dominates total latency?" → the decode phase, one pass per output token.
• "Why is the same architecture parallel during training but serial during generation?" → training uses the known ground-truth sequence with a causal mask so all positions score at once; generation has no future tokens to condition on until they are produced.

Distractor families, and why each is wrong:

Because during training the future already exists, and during generation it does not.

In training you have the complete text. The causal mask from 04-01 lets the model compute, in one pass, a prediction at every position that only used the tokens to that position's left. Every one of those predictions has a known correct answer — the actual next token — so a single forward pass over a 2,000-token document yields 2,000 supervised predictions simultaneously. This is teacher forcing, and it is the reason transformer pretraining scales.

In generation there is no complete text. Token t+1's input includes token t, and token t does not exist until it is selected. You cannot start step t+1 before step t finishes. That is a serial dependency of the same kind as any iterative computation, and hardware does not dissolve it.

The practical shape of the answer: training parallelises over positions in a sequence you already have; generation cannot, because it is creating that sequence. Techniques such as speculative decoding attack the problem from a different angle — drafting several tokens cheaply and verifying them in one pass — and speculative decoding appears in the TensorRT-LLM feature set covered in 12-08. Their availability and behaviour are version-sensitive, and they do not change the underlying dependency; they change how often you must pay for it.

Because there is no mechanism in the loop that revisits a decision, and because the emitted token immediately becomes conditioning context.

Follow the state. After selecting token t, the token id is appended to the sequence and its keys and values are written into the cache. Step t+1 attends over that cache. Nothing in the architecture provides a path back: no edit operation, no backtracking, no re-scoring of prior selections. The best the system can do is continue, which means any correction is additional text ("actually, the correct figure is…") rather than a repair.

This explains behaviour that otherwise looks like stubbornness:

• Confident wrong openings persist. If the first clause commits to a wrong answer, the highest-probability continuation is a coherent defence of that answer, because coherence is what the model was trained to produce.
• Reasoning-before-answer prompting works for a mechanical reason. If the answer token comes first, everything after it is conditioned on it and cannot change it. If the reasoning comes first, the answer is conditioned on the reasoning. That is the mechanism behind chain-of-thought's usefulness, and also behind its limits — the emitted reasoning is not a guaranteed faithful trace of any internal process, which 05-03 insists on.
• Beam search partially mitigates this and does not solve it. Keeping several candidate prefixes lets a globally better sequence survive an unpromising first token. But beams are still extended left to right and pruned; nothing is edited.

The exam-level statement: generation is append-only, so an early token constrains every later one, and self-correction is new text rather than revision.

The model outputs a distribution. The decoding strategy chooses a token from it. They are separate components with separate settings, and conflating them causes most of the confusion around temperature and top-p.

Two implications that pay off immediately. First, temperature does not make the model smarter or dumber; it reshapes a distribution the model already produced. If the right answer carries 2% probability, no sampling setting reliably retrieves it — that is a model or prompt problem, not a decoding problem. Second, the same model can behave deterministically or creatively without changing a single weight, purely through selection. That is why decoding parameters are the cheapest experiment in the entire stack and the first thing to vary when output quality is off — under the prompt-testing discipline 05-04 sets out.

One honest caveat about determinism. "The same input gives the same logits" holds in principle, and in practice greedy decoding on a real serving stack is not guaranteed to be bit-identical across runs: batching, kernel selection, and floating-point non-associativity can all perturb low-order bits, and a perturbation that flips two near-tied logits changes the selected token and therefore everything after it. Reproducibility deserves its own treatment and gets one in 09-11. For now: temperature zero reduces variability sharply and does not guarantee identical text.

• Autoregressive generation — producing output one token at a time, each conditioned on all previous tokens, with the model's own output fed back as input.
• Logits — the raw per-vocabulary-token scores the model emits before softmax; where penalties and constraints are applied.
• Prefill — the parallel forward pass over the whole prompt; compute-heavy and the main driver of time to first token.
• Decode — the serial phase producing one output token per pass; memory-bandwidth-heavy and driven by output length.
• Time to first token (TTFT) — the delay before the first output token appears; largely a prefill cost.
• Inter-token latency — the delay between successive output tokens; a decode-step cost.
• KV cache — stored per-layer keys and values for tokens already processed, reusable because of causal masking; the reason decode is memory-bound.
• End-of-sequence token — a learned special token whose prediction signals natural completion.
• Max tokens / max new tokens — a hard caller-set cap on generated tokens; a budget and a truncation, never a target.
• Stop sequence — a caller-supplied string that halts generation when produced.
• Teacher forcing — training a model on the ground-truth previous tokens rather than its own predictions; enables parallel training and creates a train/inference mismatch.
• Decoding strategy — the rule that selects one token from the model's distribution; the separate component 04-05 is entirely about.

1. The loop is: distribution → select → append → repeat. Nothing is planned, nothing is revised, and each token is chosen with no commitment to any token after it.
2. The model emits probabilities over the whole vocabulary; the decoding strategy emits text. Keeping those two separate resolves most confusion about temperature and top-p.
3. Generation is serial in the output because of a real data dependency. Token t+1's input contains token t. Hardware cannot remove that.
4. Prefill and decode are different workloads. Prefill is parallel and compute-bound and sets TTFT; decode is serial and memory-bound and scales with output length.
5. Output length, not input length, drives generation time and cost variance. One prefill pass plus one pass per output token.
6. Four stopping conditions: end-of-sequence token, max tokens, stop sequences, context exhaustion. Max tokens is a cap, not a target, and its absence is a production liability.
7. The KV cache is a direct consequence of causal masking, not an unrelated optimisation — and it is why serving LLMs is a memory problem.
8. Training is parallel over positions because the future is known; generation is not because it is being created. Teacher forcing is what makes the training side work, and the resulting mismatch is why generation can start well and drift.
9. Generation is append-only. An early wrong token becomes conditioning context, which is why reasoning-before-answer prompting has a mechanical basis and why self-correction is new text rather than repair.

You now know the loop and you know that step 3 — selecting one token from a distribution over tens of thousands — has been deliberately left as a black box. That box is where every knob a practitioner actually turns lives: greedy versus beam search, temperature, top-k, top-p and nucleus sampling, repetition penalty. It is also the exact point at which an LLM stops being a deterministic function and starts producing different text for the same prompt, which makes it the origin of every reproducibility complaint you will ever field.

Next: 04-05 opens the box — greedy, beam, temperature, top-k, and top-p, what each one does to the distribution, how they compose, and which one to reach for when output is either too bland or too unhinged.