Temperature, top-k, top-p, and greedy decoding explained

A decoding parameter is a runtime setting that controls how one token is chosen from the probability distribution the model emitted. It is not part of the model. It changes no weights. It costs almost nothing to change, applies per request, and can be different for two calls to the same deployed model in the same second.

The boundary that makes everything else make sense: the model decides what is probable; decoding decides what is chosen. From 04-04, the model's final act each step is to produce logits — one raw score per vocabulary entry — which become probabilities under a softmax. Decoding operates on that output. Therefore:

• Decoding can make output more or less varied, more or less repetitive, more or less predictable, longer or shorter-lived, more or less likely to pick an unusual word.
• Decoding cannot make the model know something it does not know. If the correct answer sits at 1% probability, no temperature setting reliably surfaces it, and sampling that reaches it will reach many wrong tokens at similar probability along the way. A factual failure is a model, prompt, or grounding failure — 09-12 and 07-11 — not a decoding one.
• Decoding cannot change the number of decode steps, except for beam search, which multiplies the work per step by the beam width.

The parameter list the blueprint's own must-know content names: greedy, beam, top-k, top-p/nucleus, temperature, max tokens, and repetition penalty, along with the effect of each on determinism versus diversity. That list is the syllabus for this lesson, and max tokens was covered as a stopping condition in 04-04.

Order matters here, because these operate in a pipeline and their order explains how they compose.

logits (one per vocabulary token)
  │
  ├─ 1. logit-level modifications:  repetition penalty · presence/frequency
  │                                 penalties · logit bias · banned tokens ·
  │                                 grammar or schema constraints
  │
  ├─ 2. temperature:  divide every logit by T
  │
  ├─ 3. softmax:  logits → probabilities summing to 1
  │
  ├─ 4. truncation:  top-k, then top-p (renormalise what remains)
  │
  └─ 5. selection:  argmax (greedy) or a random draw (sampling)

Two things to notice immediately. Temperature acts before the softmax, on logits; top-k and top-p act after it, on probabilities. And greedy is not a separate mode so much as the degenerate case of sampling — take the maximum instead of drawing.

L1 — Greedy decoding: always take the most likely token

At each step, pick the argmax. No randomness is introduced.

What it buys: the highest-probability token every time, near-deterministic output, and reproducibility that is good enough for most testing purposes. What it costs: greedy is locally optimal and says nothing about the sequence as a whole. Taking the best token now can walk into a region where every continuation is poor, and there is no way back — 04-04 established that generation is append-only.

Greedy's characteristic failure is repetition and degeneracy. Once a phrase appears, a model conditioned on that phrase often assigns high probability to repeating it, and greedy takes the high-probability option every time, so it loops. You will see this as the results show that the results show that the results show that. Greedy has no mechanism to escape, because escaping requires choosing something that is not the maximum.

Use greedy when you want one answer, the same answer, and the most likely answer: classification-style responses, extraction, deterministic transformations, and evaluation runs where variance would obscure the signal.

L1 — Temperature: reshape the distribution before you sample

Temperature T divides every logit before the softmax.

• T = 1.0 leaves the distribution as the model produced it.
• T < 1.0 (say 0.2) divides by a small number, magnifying the gaps between logits. The softmax then concentrates mass on the already-likely tokens. The distribution sharpens; output becomes more focused and more predictable.
• T > 1.0 (say 1.5) shrinks the gaps. The softmax spreads mass toward the tail. The distribution flattens; output becomes more varied, more surprising, and — pushed far enough — incoherent, because low-probability tokens include genuinely bad ones.
• T → 0 drives the distribution toward putting all mass on the maximum, which is greedy. This is why APIs commonly document temperature 0 as equivalent to greedy decoding, and why "set temperature to zero for determinism" is standard advice.

A worked illustration with constructed numbers — invented so the arithmetic is visible, not measured from any model. Take three tokens with logits 4.0, 2.0, and 1.0 and look at what temperature does to their probabilities:

logits:              A = 4.0     B = 2.0     C = 1.0

T = 1.0  →  scaled:  4.0, 2.0, 1.0     probabilities ≈  0.84  0.11  0.04
T = 0.5  →  scaled:  8.0, 4.0, 2.0     probabilities ≈  0.98  0.02  0.00
T = 2.0  →  scaled:  2.0, 1.0, 0.5     probabilities ≈  0.63  0.23  0.14

The ranking never changes — A is most likely at every temperature. What changes is how much of the mass A holds. At T = 0.5 the alternatives are nearly extinguished; at T = 2.0 token C has gone from a 4% chance to a 14% chance.

Two consequences worth stating explicitly because they are common exam distractors. Temperature does not reorder tokens. And temperature alone, at any positive value, leaves every token in the vocabulary with non-zero probability — including nonsense. That residual tail is exactly what top-k and top-p exist to cut off.

L2 — Top-k sampling: keep a fixed number of candidates

Sort tokens by probability, keep the top k, discard everything else, renormalise the survivors so they sum to 1, and sample from them.

k = 1 is greedy. k = 50 keeps the fifty most likely tokens regardless of how much probability they hold between them. The rest of the vocabulary is removed from consideration entirely, which is the point — it guarantees the tail cannot be sampled.

Top-k's weakness is that k is fixed while the model's confidence is not. Consider two situations:

• After The capital of France is, the distribution is extremely peaked — maybe one token holds most of the mass. Keeping 50 candidates means keeping 49 tokens that were never plausible, and any of them can now be drawn after renormalisation inflates their share.
• Mid-paragraph in creative prose, the distribution may be genuinely flat, with hundreds of reasonable continuations. Keeping 50 arbitrarily amputates good options.

The same k is too permissive in the first case and too restrictive in the second. That mismatch is precisely the problem top-p was invented to solve, and it is the answer to "why would you use top-p instead of top-k."

L2 — Top-p / nucleus sampling: keep a probability mass

Sort tokens by probability, accumulate from the top until the running total reaches p, keep exactly that set — the nucleus — discard the rest, renormalise, sample.

p = 0.9 means "keep the smallest set of tokens whose probabilities sum to at least 0.9." The number of tokens kept is not fixed; it is whatever the distribution requires.

Run it against the two situations above, with constructed distributions:

Confident step  (after "The capital of France is")
  Paris 0.92 · the 0.02 · located 0.02 · a 0.01 · in 0.01 · …
  top-p = 0.9  →  nucleus = { Paris }                     1 token kept
  top-k = 50   →  50 tokens kept, 49 of them implausible

Uncertain step  (mid-sentence in open prose)
  0.06 · 0.05 · 0.05 · 0.04 · 0.04 · 0.04 · 0.03 · … a long flat tail
  top-p = 0.9  →  nucleus = several dozen tokens          set expands
  top-k = 50   →  exactly 50, cutting off equally good options

This is the whole distinction, and it is the one the exam tests: top-k fixes the count; top-p fixes the mass, so its candidate set adapts to the model's confidence at that step. When the model is sure, top-p behaves almost like greedy. When the model is unsure, it opens up. That adaptivity is why nucleus sampling became the common default for open-ended generation.

Top-p's own weakness: at a flat distribution, a high p can admit a very large set including genuinely poor tokens, which is why p and temperature are usually tuned together, and why many stacks let you set top-k and top-p simultaneously as belt and braces — top-k caps the worst case, top-p adapts within it.

L2 — Repetition, presence, and frequency penalties

These act on logits, before temperature. The mechanism is simple: reduce the score of tokens that have already appeared, so repeating them becomes less likely.

• A repetition penalty typically divides or subtracts from the logits of tokens present in the context. Larger values push harder against repetition.
• A presence penalty applies a flat reduction to any token that has appeared at least once — it discourages reusing a token at all.
• A frequency penalty scales the reduction with how often the token has appeared — it discourages overuse while tolerating a first use.

They fix the loop that greedy decoding produces, and they are blunt. Pushed too hard they suppress tokens that legitimately must repeat: the word "the," a variable name in code, a person's name in a biography, a required key in a JSON schema. A penalty strong enough to eliminate looping is often strong enough to break structured output, which is why 05-05's structured-JSON work generally prefers low or zero penalties plus constrained decoding.

Exact parameter names, ranges, and whether a penalty is multiplicative or additive vary by library and by API version. Do not memorise a numeric default as though it were universal; know the direction of the effect and the failure mode.

L3 — Beam search: score sequences, not tokens

Beam search abandons the assumption that a good sequence is built from locally best tokens. It maintains B partial sequences (the beams). At each step it extends every beam by every plausible next token, scores all the resulting candidates by their cumulative sequence probability, and keeps the best B. When beams complete, they are compared and the best full sequence is returned.

What it buys: a sequence that scores better as a whole. A first token that is only third-most-likely can survive long enough to prove it leads somewhere better. For tasks with a single correct-ish output and a fixed mapping — machine translation above all, and to a lesser extent abstractive summarisation — this measurably matters, which is why beam search is traditionally associated with encoder-decoder models from 04-03.

What it costs, and both costs are exam-relevant:

1. Compute and memory scale with beam width. You are running roughly B sequences instead of one, each with its own KV cache from 12-05. Beam search is the one decoding choice that changes the cost of the generation loop rather than just its output.
2. Beam search makes open-ended text worse, not better. This is counterintuitive and is exactly why it is a good exam item. High-likelihood text is bland text. Searching harder for the most likely sequence finds safe, generic, repetitive output, because in open-ended generation the genuinely likely continuations are the boring ones. Human text is not maximum-likelihood text. So beam search on a chatbot produces flatter output than sampling does.

Beam width B = 1 is greedy decoding. That is a satisfying identity to hold: greedy is beam search with a beam of one, and greedy is also temperature-zero sampling.

L3 — How the parameters compose, and the traps in composing them

• Temperature then truncation. Temperature reshapes; top-k/top-p cut. Raising temperature and using a tight nucleus gives varied choices from a still-sane candidate set — the usual recipe for controlled creativity.
• Truncation is applied to the post-temperature distribution. So a high temperature enlarges what a top-p nucleus admits, because mass has moved into the tail. The two parameters interact; they are not independent dials.
• Setting temperature to 0 makes top-k and top-p irrelevant. With all mass on the argmax, any nucleus containing the argmax selects it. This is the most common wasted configuration in production: temperature 0 alongside a carefully tuned top-p, where the top-p is doing nothing.
• Setting k = 1 also makes temperature irrelevant to the outcome, for the same reason in reverse.
• Beam search and sampling are alternative selection modes. Combining them exists (sampling within beams) but is unusual; if an API exposes beam width and temperature, expect one to dominate.

The single highest-value asset in this lesson. Learn the middle three columns cold.

And the one-line discriminations most likely to be asked directly:

Prompt: The best thing about winter is. Below is a constructed distribution over an illustrative shortlist — invented numbers so the mechanics are visible, not measured output from any model. Assume the full vocabulary tail holds the remaining 0.06.

token          probability     cumulative
" the"             0.34           0.34
" snow"            0.22           0.56
" that"            0.14           0.70
" how"             0.09           0.79
" its"             0.06           0.85
" waking"          0.04           0.89
" everything"      0.03           0.92
" hot"             0.02           0.94
(long tail)        0.06           1.00

Strategy 1 — Greedy. Take the argmax: " the". Every run, same token. Note what has happened: the most likely token is a function word carrying almost no content, which is a good preview of why greedy output reads as flat.

Strategy 2 — Top-k with k = 3. Keep " the", " snow", " that". Their probabilities sum to 0.70, so renormalise by dividing by 0.70:

" the"   0.34 / 0.70 = 0.486
" snow"  0.22 / 0.70 = 0.314
" that"  0.14 / 0.70 = 0.200

Sample from those three. " snow" now has a 31% chance rather than 22%, because truncation redistributed the removed mass proportionally. Truncation raises the probability of the survivors — worth knowing, because it is why an aggressive top-k can make the second-place token much more likely than the model thought it was.

Strategy 3 — Top-p with p = 0.9. Accumulate: 0.34, 0.56, 0.70, 0.79, 0.85, 0.89 — still under 0.9 — then 0.92, which crosses it. The nucleus is the first seven tokens, through " everything". Renormalise over 0.92 and sample. Note the count: seven, chosen by the distribution rather than by us. On a peakier step the same p = 0.9 might keep one token; on a flatter one, forty.

Strategy 4 — Temperature 0.5 then top-p 0.9. Halving the temperature doubles every logit, sharpening the distribution. Illustratively the mass might redistribute toward something like " the" 0.55 · " snow" 0.25 · " that" 0.10 · " how" 0.05 · …. Now accumulate to 0.9: 0.55, 0.80, 0.90 — the nucleus is three tokens. Same p, smaller set, purely because temperature moved mass toward the head. This is the interaction: temperature changes what a fixed p admits.

Strategy 5 — Beam search, width 2. Keep two beams: " the" and " snow". Extend both. Suppose — again constructed — that " the" leads to continuations whose best two-token cumulative log-probability is worse than " snow ...", because " the" opens onto a large set of mediocre options while " snow" opens onto a strong one. Beam search then prefers the " snow" branch even though " the" won at step one. This is the whole value of beam search: a locally second-best token surviving to prove it leads somewhere better. And it cost two beams' worth of compute and cache to find out.

What the five strategies produced. Greedy: one fixed, bland opening. Top-k 3: three possible openings, second place boosted. Top-p 0.9: seven possible openings, count set by confidence. Temperature 0.5 + top-p 0.9: three, focused. Beam 2: one opening, chosen for where it leads, at double the cost. Same model, same weights, same prompt, same forward pass. Only the selection rule changed.

Two decision rules that generalise.

If there is one correct output, do not sample. Sampling exists to produce variety, and variety is a defect when the requirement is a specific answer. This single rule resolves the majority of real-world decoding misconfigurations.

If output is too bland, raise temperature or p before you blame the model; if output is incoherent, lower them before you blame the model. Decoding is the cheapest experiment in the stack — no retraining, no redeployment, per-request. Try it first. But if the content is wrong rather than the style, decoding is not your problem.

Text-generation parameters are named explicitly in the highest-frequency reported topic tier for NCA-GENL, and the blueprint's must-know content for generation lists them individually: greedy, beam, top-k, top-p/nucleus, temperature, max tokens, repetition penalty, and the effect of each on determinism versus diversity. That last clause is the exam's actual target. It serves objective 1.9 on using prompt-engineering principles to achieve desired results — decoding settings are part of getting a desired result — and objective 1.3 on building LLM use cases, since a chatbot and a classifier want opposite settings.

"Top-k vs top-p" appears on the standing list of confusable pairs the exam is expected to exploit. Expect at least one question that hinges on exactly that distinction.

Question phrasings to expect:

• "Which decoding strategy always selects the highest-probability token?" → greedy.
• "What is the effect of increasing temperature?" → flattens the distribution, increasing diversity and randomness and reducing determinism.
• "What is the effect of decreasing temperature toward zero?" → sharpens the distribution toward the most likely token; at zero it is equivalent to greedy.
• "How does top-p differ from top-k?" → top-p selects the smallest set of tokens reaching a cumulative probability threshold, so the number of candidates varies with the model's confidence; top-k always keeps the same number.
• "Which setting would you use for reproducible output?" → greedy / temperature 0.
• "Which strategy considers multiple candidate sequences rather than one token at a time?" → beam search.
• "A summarisation system produces repetitive loops. Which parameters address this?" → sampling instead of greedy, and a repetition or frequency penalty.
• "Which decoding strategy is most associated with machine translation?" → beam search.
• "Which parameter caps the number of tokens generated?" → max tokens.
• "Why is beam search a poor choice for open-ended creative writing?" → optimising sequence likelihood yields generic, safe text; high-likelihood text is bland.

Distractor families, and why each is wrong:

Top-k keeps a fixed number of candidate tokens. Top-p keeps a fixed probability mass and therefore a variable number of candidates.

The reason this matters is that the model's confidence changes step by step, sometimes dramatically within one sentence. After The capital of France is the distribution is extremely peaked. Three tokens later, mid-clause in a descriptive sentence, it can be genuinely flat with dozens of good options. A single k cannot serve both: it is far too permissive at the peaked step, where it keeps k−1 tokens the model considered implausible and then raises their probabilities by renormalising, and too restrictive at the flat step, where it truncates options no worse than the ones it kept.

Top-p is defined so its candidate set tracks that confidence automatically. Peaked step, tiny nucleus, behaviour close to greedy. Flat step, large nucleus, real diversity. The adaptivity is not a nice-to-have; it is the reason nucleus sampling replaced top-k as the common default for open-ended generation.

Two things to add for completeness. They are frequently used together — top-k as a hard ceiling on the worst case, top-p adapting within it — and many APIs accept both simultaneously. And neither one is a quality judgement: both truncate a ranked list. If the model's ranking is wrong, truncating it more or less cleverly does not help.

In selection behaviour, effectively yes. In reproducibility guarantees, not quite — and the gap is worth understanding because it produces real support tickets.

As temperature approaches zero, dividing the logits by a vanishing number drives the softmax toward putting all mass on the largest logit. Sampling from a distribution with all its mass on one token returns that token. That is argmax, which is greedy. Most APIs implement temperature 0 as greedy directly rather than dividing by zero, and document them as equivalent.

Where the equivalence stops: greedy is a deterministic rule, not a guarantee of identical text across runs. Several things can perturb the logits or their comparison on real infrastructure:

• Floating-point arithmetic is not associative, so summation order affects low-order bits, and summation order can depend on batch composition and on which GPU kernel was selected.
• The same model served on different hardware, or with a different library version, or at a different numeric precision from 12-01, may produce slightly different logits.
• Quantization from 12-02 changes the numbers outright.
• If two candidate tokens are nearly tied, a tiny perturbation flips the argmax — and because generation is append-only, one flipped token changes everything after it. Small cause, total divergence.

So the correct statement is: temperature 0 removes deliberate randomness and gives you the most reproducible behaviour available, and it does not promise byte-identical output. Anyone building a regression suite needs to know this, which is why 09-11 exists as its own lesson and why 10-04 treats LLM CI as a tolerance problem rather than an equality check. Anything else in the system — a retrieval step whose index changed, a timestamp in the prompt, conversation history — introduces far more variation than the decoder ever will.

Because beam search searches harder for high-likelihood text, and high-likelihood text is bland.

The argument runs in three steps. First, a language model assigns probability based on what typically follows in its training data, so the highest-probability continuation is by construction the most typical one. Second, beam search's objective is to maximise the probability of the whole sequence, so it systematically prefers sequences made of typical continuations. Third, human writing that people find interesting is not maximum-likelihood writing — it contains choices that are locally somewhat surprising. Optimising for likelihood therefore optimises away the very property that makes open-ended text good. Wider beams make this worse, not better: more search, more thoroughly generic result. Published work on neural text degeneration made exactly this argument and is the reason nucleus sampling exists.

Contrast the case where beam search genuinely helps. Machine translation has a near-single correct output determined by the source; there is little legitimate diversity to preserve, and getting the sequence globally right matters more than sounding surprising. Same for constrained transduction tasks generally. That is why beam search is traditionally paired with encoder-decoder models from 04-03 and sampling is paired with open-ended decoder-only generation.

The generalisable rule: beam search suits tasks with one right answer; sampling suits tasks where many answers are acceptable and variety is desirable. And note the cost asymmetry — beam search is the only decoding choice here that multiplies the work of the generation loop, since each beam carries its own KV cache from 12-05.

• Decoding strategy — the rule that selects one token from the model's output distribution; a per-request runtime setting, not part of the weights.
• Greedy decoding — always take the highest-probability token; near-deterministic, prone to repetition, equivalent to beam width 1 and to temperature 0.
• Beam search — maintain B candidate sequences and score them as wholes; better for fixed-mapping transduction, worse for open-ended text, and the only strategy whose cost scales with its parameter.
• Beam width — how many candidate sequences are kept; width 1 is greedy.
• Temperature — a divisor applied to logits before the softmax; below 1 sharpens the distribution, above 1 flattens it, and it never reorders tokens or zeroes any of them.
• Top-k sampling — keep the k most likely tokens, renormalise, sample; a fixed candidate count.
• Top-p / nucleus sampling — keep the smallest set of tokens whose cumulative probability reaches p; a fixed mass and therefore an adaptive candidate count.
• Nucleus — the retained token set under top-p.
• Renormalisation — rescaling the surviving probabilities to sum to 1 after truncation; it raises every survivor's probability.
• Repetition penalty — a logit reduction applied to tokens already present in the context.
• Presence penalty / frequency penalty — a flat reduction for any repeated token, versus one that scales with how often it has appeared.
• Determinism versus diversity — the axis every parameter here trades along, and the phrasing the blueprint itself uses.

1. Decoding selects from a distribution the model already produced. It changes diversity and determinism. It never adds knowledge, and it never fixes a factual error.
2. The pipeline order is: logit penalties → temperature → softmax → top-k → top-p → select. Temperature acts on logits, truncation acts on probabilities, and that ordering explains how they interact.
3. Greedy takes the argmax. Most deterministic, least diverse, and structurally vulnerable to repetition loops because escaping requires not taking the maximum.
4. Temperature reshapes; truncation cuts. Temperature below 1 sharpens, above 1 flattens, never reorders, and never zeroes anything.
5. Top-k fixes the count; top-p fixes the mass. Top-p's candidate set therefore adapts to the model's confidence, which is precisely why it became the default for open-ended generation. This is the most predictable exam item in the module.
6. Truncation raises the survivors' probabilities through renormalisation — an aggressive k can promote a token well above what the model assigned it.
7. Beam search optimises sequences, not tokens. It helps translation-shaped tasks, hurts open-ended writing because high-likelihood text is bland, and it is the only decoding choice that multiplies the loop's cost.
8. Greedy = temperature 0 = beam width 1 in selection behaviour, and none of them promises byte-identical output on real infrastructure.
9. If there is one correct output, do not sample. That single rule prevents most real misconfigurations.
10. Repetition penalties are blunt instruments. They cure loops and break required repetition; prefer constrained decoding when structure matters.
11. Decoding is the cheapest experiment in the stack — per request, no redeployment. Try it early, but only for style and variety, never for correctness.

Every parameter in this lesson chooses tokens inside a budget nobody has yet made explicit. The prompt, the retrieved context, the conversation history, the system instructions, and every token the model generates all draw on the same finite pool — the context window. Max tokens caps only one part of it. Exceed the whole and you do not get degraded output; you get truncation, an error, or a silently dropped instruction, and the failure often lands on whichever component was appended last.

Next: 04-06 makes the context window concrete — what counts against it, how to budget prompt against completion, the difference between a representational limit and a memory limit, and why "just use a bigger window" is the most expensive answer in the module.