Causal Language Modeling: The Training Objective Behind Generation

Causal language modeling (CLM) is the training objective that teaches a decoder to predict the next token in a sequence using only the tokens that precede it — never the tokens that follow. [GROUND TRUTH] (Sources/ncp-genl/domain-2-prompt-engineering.md): "CLM — decoder-only, left-to-right next-token prediction. This is the objective that makes autoregressive generation possible." The word "causal" names the constraint directly: a prediction at position t may depend causally on positions 1 through t-1, and on nothing at position t+1 or later, because in a real generation scenario those later tokens do not exist yet — the model has not produced them.

This is deliberately the mirror image of the encoder's training objective. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "Encoder-only (BERT): Sees the whole context in both directions; trained by masking tokens and predicting them (MLM)." Masked language modeling lets a prediction look in both directions — left and right of a masked token — because an encoder's job is understanding a sequence that already exists in full, not producing one token at a time.

⚠️ UNVERIFIED — the exact proportion of tokens masked during MLM pretraining, commonly cited elsewhere as 15%, is not restated in this domain's own source material, so this lesson will not assert a specific figure as this cert's ground truth.

[GROUND TRUTH] (Sources/ncp-genl/domain-2-prompt-engineering.md) states the standing trap plainly: "CLM (decoder, causal) vs MLM (encoder, bidirectional). Prompt-engineered generation relies on CLM-trained decoders." Every technique this module has covered — few-shot exemplars, chain-of-thought reasoning, a structured template — is content injected into a CLM model's left-to-right context, hoping to shift what a causally-constrained next-token prediction produces. None of it would transfer to an MLM-trained encoder in the same way, because an encoder was never trained to produce a sequence one token at a time in the first place; it was trained to fill in a gap given the whole surrounding sentence.

L1 — Intuition: the training task is "guess what comes next," repeated at every position at once

Take an ordinary sentence from a training corpus: The valve failed under sustained pressure. Tokenized, this becomes a sequence of token ids. CLM training does not wait until the whole sentence is generated to check correctness — it takes the sequence as it already exists in the training data and asks, at every single position simultaneously, "given everything up to here, what token comes next?" At the position after The, the target is valve. At the position after The valve, the target is failed. At the position after The valve failed, the target is under. Every one of those predictions is scored against the token that actually follows in the real sentence, and all of them happen in one pass over the sequence.

L2 — Mechanism: masking future tokens, teacher forcing, and next-token cross-entropy

Three mechanical pieces make this work, and each answers a specific "how" a professional-level question can probe.

The causal attention mask. Inside the transformer's self-attention, a position t is prevented from attending to any position after it by a mask that sets those attention scores to negative infinity before the softmax, so their weight becomes exactly zero. This is the literal, computational meaning of "causal" — it is not a metaphor about training order, it is a mask applied inside every attention layer, every time, at both training and inference.

Teacher forcing. During training, the actual tokens from the training sequence are fed into the model at every position — not the model's own (possibly wrong) prediction from the previous step. This matters enormously for how training scales: because the correct previous tokens are already known in advance (they are sitting right there in the training corpus), the model can compute predictions for every position in the sequence in one single parallel forward pass, rather than needing to generate token 1, feed it back in, generate token 2, and so on. Teacher forcing is what lets CLM pretraining process an entire batch of long sequences efficiently on parallel hardware, and it is also precisely what does not carry over to inference, which section 3 covers.

Next-token cross-entropy loss. At each position, the model outputs a probability distribution over its entire vocabulary for "what token comes next." The loss at that position is the cross-entropy between that predicted distribution and the actual next token (encoded as a one-hot target) — in plain terms, the loss is low when the model assigned high probability to the token that actually came next, and high when it assigned low probability to it. The total training loss is the average of this per-position cross-entropy across every position in every sequence in the batch. [GROUND TRUTH] (Sources/ncp-genl/domain-2-prompt-engineering.md) describes the resulting capability in outcome terms — "left-to-right next-token prediction" — and cross-entropy loss over the vocabulary at each position is the standard mechanism by which that outcome is actually optimized during pretraining.

L3 — The exam-relevant edge case: training is parallel, inference is not, and that asymmetry is the whole reason KV caching exists

This is the single most consequential downstream fact about CLM, and it is worth holding precisely. During training, teacher forcing means every position's prediction is computed in the same parallel pass, because the correct tokens to condition on are already fully known from the training data. During inference — actually generating new text — there is no ground-truth next token sitting in a dataset to feed forward. The model must generate token t, then feed that exact generated token back in as input before it can generate token t+1, because nothing else is available yet. This is why generation is inherently sequential and autoregressive: one token at a time, each one depending on every token generated so far, with no way to skip ahead.

That single asymmetry — parallel at training time, sequential at inference time — is the direct cause of essentially every inference-time optimization concern this cert covers elsewhere, including why a KV cache exists at all (M4-04): without one, each new generated token would force the model to recompute keys and values for every prior token from scratch, because the "obviously already known" shortcut that teacher forcing provided during training is simply not available once you are generating text nobody has written yet.

Every technique this module has built up rests on one property of a CLM-trained decoder: it always predicts its very next token from everything currently in its context, whatever that context happens to contain. This is not a special mode the model enters when it sees a well-crafted prompt — it is the only thing the model ever does, at every position, because next-token prediction from left-to-right context is the entire training objective it was optimized against.

• In-context learning (M2-01) works because exemplars placed earlier in the context are, mechanically, just more tokens the causal attention mask allows the final prediction to see. The model was never trained on "tasks with examples" specifically; it was trained to predict the next token from whatever preceded it, and a well-chosen set of exemplars happens to make the locally most probable continuation the one you want.
• Chain-of-thought prompting (M2-02) works because a written intermediate step becomes part of the left-to-right context available to every later prediction — exploiting the exact same causal-context mechanism, just used to carry the model's own generated reasoning forward instead of a human-supplied exemplar.
• Prompt templates (M2-02) work because a structured, unambiguous instruction shifts the probability distribution CLM's next-token prediction draws from, at every position of the response, toward the shape the template specifies.

None of these techniques change the training objective, and none of them could substitute for it. A model that had never been trained with CLM in the first place — an encoder trained purely with MLM, for instance — could not be "prompted" into autoregressive generation no matter how carefully you structured the input, because it was never trained to produce a sequence one causally-constrained token at a time to begin with. This is the professional-level version of the trap named in section 1: CLM is not one architectural detail among many, it is the load-bearing precondition for every prompt-engineering technique this domain teaches.

The output-control techniques in M2-04 fit the same pattern from a different direction. Constrained decoding does not add a new capability to the model either — it intervenes at the exact point where CLM's next-token distribution is about to be sampled from, masking out any token that would violate a required structure before the sampling step occurs. That intervention is only possible because CLM's forward pass already produces a full probability distribution over the vocabulary at every position; constrained decoding simply narrows which entries of that already-existing distribution are eligible to be chosen. Take away the CLM objective underneath it, and there is no per-position distribution left to constrain in the first place — the entire mechanism only has something to act on because a CLM-trained decoder already produces it as a matter of course.

This is worth generalizing into a single governing rule for the whole module: every technique this domain covers is a way of shaping either the context a CLM-trained decoder predicts from, or the distribution it is allowed to sample from once that prediction is made — and nothing covered anywhere in this domain reaches inside the training objective itself. Objective 2.5's decision rule, covered in M2-05, is really asking a question that follows directly from this: once you have exhausted every way of shaping context and constraining output on top of a fixed CLM objective, and the gap that remains is still not closed, the only thing left to change is the objective's own learned parameters — which is fine-tuning, and a different domain of this exam entirely.

Constructed scenario, illustrative only — the tokens, ids, and losses below are invented for arithmetic clarity, not measured from any real training run.

Take the training sentence Valves fail under stress . tokenized (for this illustration) into five whole-word tokens with arbitrary ids: Valves=101, fail=205, under=88, stress=340, .=4.

Step 1 — build the shifted target sequence. CLM training does not need a separate "label" dataset; the labels are the same sequence, shifted one position to the left, because "predict the next token" means the target at position i is simply the token at position i+1.

Input sequence (positions 1-5):   Valves   fail    under   stress   .
Target at each position:          fail     under   stress  .        <end>

Step 2 — compute cross-entropy loss at each position, given the model's predicted probability for the correct next token.

Position 1 (context: "Valves"):        model assigns P(fail)   = 0.20 -> loss = -ln(0.20) = 1.609
Position 2 (context: "Valves fail"):    model assigns P(under)  = 0.65 -> loss = -ln(0.65) = 0.431
Position 3 (context: "...fail under"):  model assigns P(stress) = 0.10 -> loss = -ln(0.10) = 2.303
Position 4 (context: "...under stress"):model assigns P(".")    = 0.90 -> loss = -ln(0.90) = 0.105

Step 3 — average the per-position losses into the sequence's training loss.

Average loss = (1.609 + 0.431 + 2.303 + 0.105) / 4 = 4.448 / 4 = 1.112

Step 4 — read the result. Position 3 carries the highest loss (2.303), meaning the model assigned only 10% probability to the actual next word, stress, given Valves fail under as context — a training signal telling the model's parameters to shift probability mass toward stress (and away from whatever it favored instead) the next time it sees a similar left-to-right context. Position 4 carries the lowest loss, because a period is a highly predictable continuation after a short declarative clause. This position-by-position loss, averaged across every sequence in a training batch and accumulated over the entire pretraining corpus, is the actual signal that shapes a decoder's weights into a model capable of the left-to-right prediction that every later prompting technique in this module then exploits at inference time.

Step 5 — the property teacher forcing buys, made concrete. All four positions' losses above were computed in the same single forward pass, because the context fed into position 3 (Valves fail under) used the real training tokens, not whatever the model itself might have predicted at positions 1 and 2. If the model's own (possibly wrong) predictions had been fed forward instead — the way inference actually has to work — an early mistake could compound into every later position, and the four predictions could not have been computed independently in parallel. Teacher forcing is precisely what avoids that compounding during training, at the cost of the exact assumption inference cannot make: that the "true" previous tokens are already known.

Constructed scenario, illustrative only, continuing the numbers from section 4. Suppose the trained model above is now deployed, and a user's prompt ends with Valves fail under, asking the model to continue. This is where the training-versus-inference asymmetry from section 2's L3 becomes concrete rather than abstract.

Step 1 — generate one token, using the same probabilities the training example assumed. The model computes a distribution over its vocabulary given the context Valves fail under, and — using the same illustrative number from section 4's position 3 — assigns stress a probability of 0.10. Suppose the highest-probability token this time is actually load at 0.35, and the model samples (or greedily selects) load instead of stress.

Context: "Valves fail under"
Sampled next token: "load"   (P = 0.35, the model's top choice this time)

Step 2 — feed the sampled token back in, because there is no other option. Unlike training, where the true next token stress was already sitting in the corpus and fed forward regardless of what the model predicted, inference has no ground truth to fall back on. Whatever the model actually produced — load, not stress — is now part of the context for the next prediction, whether or not it matches what the original training sentence said.

New context: "Valves fail under load"

Step 3 — every subsequent prediction now depends on this one sampled token. The model computes its next distribution over Valves fail under load ___, and this prediction has no way to "undo" the earlier choice of load over stress — it simply continues from wherever generation actually is. If load turns out to be a poor continuation (say the intended meaning was about physical stress, not electrical load), nothing in the generation process catches or corrects that until the sequence is complete and evaluated after the fact.

Step 4 — contrast the two computations directly. During training, computing the loss at the position after Valves fail under never depended on what the model would have generated there — the actual next token, stress, was already fixed by the training data, letting every position's loss be computed independently and in parallel. During inference, the token generated at that exact position becomes a hard input to everything that follows, and it can only be computed after the tokens before it are already fixed by the model's own prior choices, one at a time. This is the single sentence worth carrying out of both worked examples together: the same causal, left-to-right structure that let training compute four losses in one parallel pass is exactly what forces inference to compute four tokens in four sequential passes — one objective, two completely different computational shapes, depending on whether the "next token" is already known (training) or has to be produced (inference).

Objective 2.3 places CLM inside Prompt Engineering's domain specifically because, as the domain's own scope note states, "Objective 2.3 also touches the decoder training objective (CLM) that makes generation possible." [GROUND TRUTH] (Sources/ncp-genl/domain-2-prompt-engineering.md). This is a deliberate placement choice worth noticing: CLM is architecturally a Domain 1 topic, but the exam tests it again here because prompt engineering is meaningless without it — you cannot reason correctly about what a prompt can and cannot change if you do not know what training process produced the thing you are prompting.

Expect the question in one of these shapes:

1. A bare identification. "Which training objective underlies autoregressive text generation?" — keyed to causal language modeling, with masked language modeling, contrastive loss, and next-sentence prediction as plausible-sounding distractors that name real objectives attached to the wrong architecture family.
2. An architecture-objective pairing trap. A scenario names BERT alongside next-token generation, or GPT alongside fill-in-the-blank prediction — the pairing is backwards in both cases, and the correct answer restores the actual mapping: encoder → MLM, decoder → CLM.
3. A mechanism-of-training question. "What allows a CLM model's training loss to be computed for an entire sequence in a single forward pass?" — keyed to teacher forcing, distinguishing training-time parallelism from inference-time sequential generation.

What the distractors typically look like

The standing trap this domain names directly is architecture-objective mismatch: pairing BERT with next-token generation, or GPT with masked language modeling, is wrong in either direction. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): "A question pairing 'BERT' with 'next-token generation,' or 'GPT' with 'masked language modeling,' is wrong." A second distractor family conflates training-time parallelism with inference-time behavior — an option claiming that generation itself happens "all at once" the way training loss computation does misunderstands that teacher forcing is available only during training, when the true tokens are already known.

Causal language modeling restricts every prediction to left-to-right context only — position t may use tokens 1 through t-1 and nothing after — which is what makes a CLM-trained decoder capable of generating text one token at a time, since at true generation time nothing after the current position exists yet. Masked language modeling instead lets a prediction see context in both directions around a masked token, which fits an encoder's job of understanding a complete sequence that already exists in full, but does not by itself produce a mechanism for generating new text token by token. [GROUND TRUTH] (Sources/ncp-genl/domain-1-llm-architecture.md): encoder-only models are "great for producing embeddings and for classification," while decoder-only, CLM-trained models are "the generation workhorse."

Because the parallelism depends entirely on already knowing the correct tokens to feed forward at every position, and that knowledge only exists during training, when the true continuation is sitting in the training corpus. At inference time, the model is producing text nobody has written yet — there is no "correct" token 6 to feed in while generating token 5, because token 6 does not exist until the model itself produces it. This forces inference into a strictly sequential, one-token-at-a-time process, which is the direct root cause of every latency concern this cert covers around autoregressive decoding, KV caching (M4-04), and streaming attention.

Yes, in a specific and important sense: because CLM restricts every prediction to context that has already appeared earlier in the sequence, no prompting technique can make a model's current-token prediction depend on something that has not yet been written into the context — a chain-of-thought step that has not yet been generated, or an exemplar placed after the point where it would need to matter, simply is not visible to the prediction it would need to influence. This is why ordering matters in every technique this module has covered: an exemplar, an instruction, or a reasoning step only shapes the predictions that come after it in the sequence, never the ones that came before, because that is the one hard boundary CLM's causal mask enforces regardless of how the prompt is written.

⭐ THE EARNED INSIGHT Every technique in this module works by adding tokens to a context a causally-masked decoder is about to predict from — and that is the entire trick, because causal language modeling never trained the model to do anything else. A prompt does not activate a special "reasoning mode" or a special "few-shot mode"; it changes what tokens are available to the one operation the model ever performs, next-token prediction from everything before the current position. Understanding that CLM is the whole mechanism, not one feature among several, is what lets you predict in advance which prompting tricks could plausibly work and which are asking the model to do something its training objective structurally cannot support — like attending to a token that has not been written yet.

• CLM predicts each token from only the tokens before it — the causal attention mask enforces this at every layer, at both training and inference.
• Teacher forcing lets training compute every position's loss in one parallel pass, because the true tokens are already known from the training corpus.
• Inference cannot use teacher forcing. Generation is sequential by necessity, one token at a time, because the "correct" future tokens do not exist until the model produces them.
• CLM (decoder, causal) and MLM (encoder, bidirectional) are the standing distractor pair — pairing BERT with generation or GPT with masked prediction is wrong in either direction.
• Every prompting technique in this module works only because the model was trained with CLM. In-context learning, chain-of-thought, and templates all operate by shaping the left-to-right context a CLM-trained decoder predicts from — none of them could substitute for the training objective itself.
• Ordering in a prompt matters because of the causal mask, not by convention: nothing later in a sequence can influence a prediction made earlier in it.

Causal language modeling explains how a decoder arrives at a probability distribution over its next token. It says nothing about what happens after that distribution exists — whether the token actually sampled from it is checked against a required output shape, or whether a malformed response gets caught before it reaches a downstream system. Next: M2-04 covers output control: constrained decoding and validation wrappers, the decoding-time and post-generation controls that sit on top of the CLM mechanism this lesson just opened up, without touching a single weight it produced.