Pretraining vs Continued Pretraining vs Instruction Tuning

Pretraining is the original, from-scratch training run that turns randomly initialised weights into a language model. The objective is self-supervised: given a span of text, predict the next token (for a decoder-only model) or predict masked tokens (for an encoder-only model). No human labels the data, because the data labels itself — the next word in the sentence is the answer key. This is the stage that produces a foundation model, and it is the stage almost nobody in an associate role will ever run, because it consumes a corpus measured in trillions of tokens and a compute budget measured in thousands of GPU-months. The mechanics of the objective itself are 01-01; the arithmetic of the loss it minimises is 01-05.

Continued pretraining — also called continued pretraining, continual pretraining, or domain-adaptive pretraining — restarts that same self-supervised objective on an already-pretrained model, using a new corpus. Same loss function, same "predict the next token" task, same lack of labels. What changes is the diet. Feed a general model ten million tokens of radiology reports and it becomes markedly more fluent in radiology-report language: it stops being surprised by the abbreviations, it learns the house style of the genre, and its internal representation of domain terms sharpens. What it does not learn is how to answer a question, because nothing in the objective ever showed it a question-answer pair.

Instruction tuning is supervised fine-tuning on a dataset of instruction-response pairs. Each training example is a demonstration: here is a request, here is what a good reply looks like. The loss is still next-token cross-entropy — that never changes across the whole stack — but it is computed over response tokens conditioned on an instruction, so what the model learns is the mapping from request to compliant answer. This is the stage that converts a raw text-continuation engine into something that behaves like an assistant. It is why a modern chat model, given "Summarise this in three bullets," produces three bullets instead of inventing more text that looks like a request.

Two framing points to hold on to. First, all three stages change weights. They sit above the line that separates weight-changing customisation from weight-preserving customisation — prompting and RAG sit below it, and 11-08 draws that line formally. Second, the stages are ordered and cumulative, not alternatives. You do not choose between pretraining and instruction tuning; instruction tuning presupposes a pretrained model. Every off-the-shelf instruct model you can download is the output of a pipeline that ran pretraining first, optionally continued pretraining, then instruction tuning, then often an alignment stage on top.

L1 — The intuition: same machine, different fuel and different wiring

Think of one machine that does exactly one thing: it reads text and gets scored on how well it predicts what comes next. That machine never changes. What changes across the stages is what you pour into it and which part of the output you grade.

• Pretraining: pour in everything, grade every token, start from noise.
• Continued pretraining: pour in a narrow genre, grade every token, start from a working model.
• Instruction tuning: pour in request-response pairs, grade only the response tokens, start from a working model.

That is the whole architecture of the ladder. The reason the outcomes feel so different is not that the learning algorithm differs but that the distribution differs, and a language model's behaviour is a mirror of the distribution it was last trained on.

L2 — The mechanics: objective, data shape, and what actually moves

Pretraining. Data is a stream of raw documents: web text, books, code, papers. It is chunked into fixed-length sequences at the tokeniser's granularity (02-02), and the model is trained with causal cross-entropy across every position. Because there are no labels, the corpus can be arbitrarily large, and it needs to be — this is the stage where scaling laws bite, where more parameters, more tokens, and more compute jointly determine capability. Emergent abilities like in-context learning (05-01) are properties this stage produces, not properties any later stage installs. Weight updates touch every parameter in the network, many times.

Continued pretraining. Data is again a stream of raw documents, but curated to a domain: your ten years of support tickets, a legal corpus, a genome-annotation corpus, an internal wiki dump. Same causal objective, same all-positions loss. Because the model starts from a good point rather than from noise, the learning rate used is normally far smaller than pretraining's — the goal is to move the model, not to rebuild it. This is also the stage where you should be most nervous, because pushing a model hard toward a narrow distribution is exactly the recipe for forgetting the broad one. That failure has a name and its own lesson: 11-03.

Continued pretraining is the correct answer to a specific and narrow question: is my model's problem that it does not speak this dialect? Symptoms that point at it include the model mis-tokenising your domain's identifiers, producing plausible-sounding but structurally wrong domain jargon, or showing high perplexity (09-02) on held-out samples of your own text relative to general text. Symptoms that do not point at it include wrong facts, stale facts, or bad output formatting.

Instruction tuning. Data is a set of records with at least two fields — an instruction (often plus an input) and a target response. In practice the pair is rendered into a single string using a chat template, the model-specific formatting convention that marks where the system prompt ends, where the user turn begins, and where the assistant turn begins. Getting this template wrong is one of the highest-frequency practical failures in fine-tuning, because a model trained with one delimiter convention and served with another is being asked to generalise across a boundary it has never seen. The loss is typically masked so that only the response tokens contribute: you want the model to learn to produce the answer, not to learn to produce the question.

Volume here is dramatically smaller than in either pretraining stage — thousands to low hundreds of thousands of examples rather than billions of tokens — because the model is not learning language, it is learning a convention. Quality dominates quantity, and the reason is structural: every example is a demonstration the model will imitate, so a sloppy example teaches sloppiness with full force. Curation discipline for exactly this is 08-01, and the label-noise failure mode is 08-02.

L3 — The full lifecycle, and where each stage's ceiling sits

Written as a pipeline, a modern instruct model's history looks like this:

random weights
   │  pretraining — trillions of unlabeled tokens, self-supervised
   ▼
base / foundation model            ← completes text; does not follow instructions
   │  (optional) continued pretraining — domain corpus, still unlabeled
   ▼
domain-adapted base model          ← fluent in the domain; still does not follow instructions
   │  instruction tuning / SFT — labeled (instruction, response) pairs
   ▼
instruct / chat model              ← follows requests, adopts a format and a persona
   │  (optional) alignment — preference data, RLHF or DPO
   ▼
aligned model                      ← prefers the responses humans rated better

Each arrow has a ceiling that the next arrow cannot raise and the previous arrow cannot substitute for:

• Pretraining sets the capability ceiling. If the base model cannot do multi-step arithmetic reasoning at all, no amount of instruction tuning will conjure the ability; instruction tuning can only surface and format capability that is already latent. This is the single most useful mental model for predicting whether a fine-tune will succeed.
• Continued pretraining sets the domain-fluency ceiling. It can teach the model that "SLA breach" and "ticket escalation" belong to a shared context. It cannot teach the model to answer in your escalation-template format, because it never saw a format instruction.
• Instruction tuning sets the behaviour and format ceiling. It is astonishingly effective at "do it in this shape, with this tone, at this length." It is unreliable at "know this fact," and that unreliability is the subject of 11-02.
• Alignment sets the preference ceiling — which of two acceptable answers the model reaches for. 11-06 is the mechanism.

One more L3 nuance worth carrying into the exam: base models and instruct models are different products, and using the wrong one is a common beginner error. A base model given a chat-style prompt often continues the prompt instead of answering it, which reads as a broken model but is exactly correct behaviour for something trained only on next-token prediction. If you plan to instruction-tune yourself, starting from a base model is usually the cleaner choice; if you plan to prompt, starting from an instruct model is almost always the right choice.

This is the table to memorise. Every column is a dimension the exam can ask about, and the data column is the single strongest discriminator in scenario questions.

Two confusables deserve their own contrast, because they trip people in opposite directions.

Continued pretraining vs instruction tuning. These are confused because both are called "fine-tuning" in casual speech and both operate on an existing model. The dividing line is whether your data has a target response. If you have a folder of PDFs, you have continued-pretraining data. If you have a spreadsheet with a "question" column and an "ideal answer" column, you have instruction-tuning data. You cannot convert one into the other by wishing; turning documents into instruction pairs is a data generation project in its own right.

Instruction tuning vs RAG. These are confused because both are proposed as "make the model know about our stuff." They are not substitutes: instruction tuning changes how the model behaves and RAG changes what the model can see. A model that has been instruction-tuned on your domain still has no access to a document written this morning. A RAG system with a perfect index still answers in whatever voice and format the underlying model happens to have. The full decision rule is 11-08; the first pass at it was 05-06.

Suppose a team supports an industrial-equipment product line and wants a model that answers technician questions in their house format. They inventory their data. The numbers below are a constructed scenario, not measurements — the point is the shape of the reasoning, not the magnitudes.

Their inventory:

• 40,000 internal maintenance documents, average 2,500 words each, no question-answer structure.
• 1,200 historical support conversations where a senior technician's reply was rated excellent.
• A general instruct model available in a 7-billion-parameter size.

Step 1 — convert documents to tokens. A common working rule for English prose is that one token is roughly three-quarters of a word, so a word count multiplied by about 1.3 gives a rough token count. That rule and its failure cases are 02-03.

40,000 docs × 2,500 words        = 100,000,000 words
100,000,000 words × 1.3 tok/word ≈ 130,000,000 tokens

130 million tokens. Is that enough for continued pretraining? It is enough to shift domain fluency measurably and far too little to build capability — which is exactly the right shape for continued pretraining and exactly the wrong shape for pretraining from scratch. Compare it to the trillions of tokens a foundation model consumed: their corpus is a rounding error against that total, which is the arithmetic reason "train our own model from scratch" is a wrong answer in essentially every scenario question you will see.

Step 2 — count the instruction-tuning data. 1,200 rated conversations. Suppose each yields, on average, 1.5 usable single-turn instruction-response pairs after cleaning.

1,200 conversations × 1.5 pairs ≈ 1,800 instruction-response pairs

1,800 pairs is a plausible starting SFT set for a narrow behavioural target — teach the house format, the escalation phrasing, the refusal style. It is not enough to install thousands of equipment-specific facts, and expecting it to is the misconception 11-02 exists to demolish.

Step 3 — compare the two stages' cost profile. Cost tracks tokens processed, and the ratio is stark:

Continued pretraining pass over the corpus:  130,000,000 tokens
SFT pass over 1,800 pairs at ~600 tokens ea:   1,080,000 tokens

Ratio ≈ 130,000,000 / 1,080,000 ≈ 120×

One epoch of continued pretraining costs roughly 120 times what one epoch of SFT costs on this data — a constructed figure that follows directly from the token counts above. And the SFT stage is the one that changes what the user actually sees. That asymmetry is why the practical ordering of effort for most teams is: prompt first, then RAG, then SFT, and only reach for continued pretraining when you have evidence of a genuine fluency gap and a corpus large enough to close it.

Step 4 — state the decision. For this team the defensible plan is: RAG over the 40,000 documents to supply facts with citations, plus a small SFT run on the 1,800 pairs to install the house format. Continued pretraining is deferred, with the note that 130 M tokens is enough to revisit it if evaluation shows a fluency rather than a retrieval failure. And the evaluation set that would show that difference has to exist before any weights move — 01-08 and 10-04 are why.

The last row is worth pausing on because it is a governance argument that people meet as a surprise. Once a fact has been trained into weights, there is no supported operation that removes it — you retrain from the previous checkpoint without that data. That is why any requirement containing the phrase "right to be forgotten," per-tenant deletion, or revocable consent pushes you toward retrieval and away from training. 13-05 is the full treatment.

The NCA-GENL blueprint puts fundamentals of machine learning and the ability to read research papers and identify emerging LLM trends inside the Core Machine Learning and AI Knowledge domain, which is the largest domain on the exam at 30% of the questions. It also asks candidates to assist in deployment and evaluation of model scalability and performance under supervision. Between them, those objectives make the training-stage vocabulary examinable in three distinct ways.

Phrasing family 1 — the definition question. "Which training stage uses unlabeled text with a self-supervised objective?" or "What distinguishes instruction tuning from pretraining?" These reward a crisp one-line identity for each stage, which is why section 1 is written as four identities rather than a narrative.

Phrasing family 2 — the scenario question. A short paragraph describes a team, a data holding, and a complaint, and asks what to do. This is the dominant format on this exam, and the deciding clue is almost always in the data description. "We have 50,000 unlabeled internal documents" and "we have 3,000 examples of correct answers written by our experts" point at different stages, and the question is testing whether you noticed. Candidate reports for this exam consistently describe questions as general-level rather than deep-technical, which means the win condition is recognising the situation, not reciting hyperparameters.

Phrasing family 3 — the stage-order question. "Place these in the order a model actually experiences them." Pretraining → continued pretraining → instruction tuning → alignment. The alignment sub-order is itself examinable and stricter: SFT → reward model → policy optimisation, which 11-06 drills.

Distractor families to expect:

A final calibration note. This exam favours the answer that names a real, proportionate engineering practice over the one that names the most powerful technique. When two options are defensible, prefer the cheaper rung of the customisation ladder, and prefer retrieval when the scenario mentions changing facts, citations, or deletion.

Not reliably, and this is the most consequential answer in the lesson. Instruction tuning optimises the model to produce responses shaped like the demonstrations it was shown. Facts inside those demonstrations do get absorbed into the weights to some degree — the model is, after all, predicting tokens that contain them — but what it primarily learns from a few thousand examples is the pattern, not the content. The practical consequence is a specific and dangerous failure mode: after instruction tuning on domain data, the model becomes better at sounding like your domain expert while remaining just as capable of inventing a part number. Fluency improves faster than accuracy, which makes errors harder for a reviewer to catch.

If your requirement is "the answer must be correct and traceable to a source document," the mechanism is retrieval, not training. If your requirement is "the answer must be in our format, at our length, in our voice, refusing the things we refuse," the mechanism is instruction tuning. Most real projects need both, in that order of confidence. 11-02 makes this the centrepiece of its argument, and 09-07 supplies the metrics that let you prove groundedness rather than assume it.

It depends who is speaking, which is exactly why the exam can test it. In the strict sense used here and in most NVIDIA-adjacent documentation, "fine-tuning" is an umbrella for any stage that updates a pretrained model's weights, and continued pretraining is one member of that family alongside supervised fine-tuning and alignment. In casual usage, "fine-tuning" almost always means supervised fine-tuning on instruction pairs, and continued pretraining is called out separately precisely because it is different.

Answer exam questions on the strict reading, and disambiguate in real conversations by naming the data. The three-question test:

1. Does your data have target responses? → Supervised fine-tuning.
2. Is your data raw documents with no targets? → Continued pretraining.
3. Is your data comparative judgements between two candidate responses? → Alignment.

If none of the three describes what you have, you do not yet have a training project — you have a data project, and 08-01 is where it starts.

You not only can, you almost certainly must, and this is the entire economic logic of the foundation-model era. Pretraining is a capital-intensive one-time act performed by organisations with the compute to do it; adaptation is a cheap, repeatable act performed by everyone else. The blueprint's framing of the associate role — contributing to development and QA of LLM systems under senior supervision, selecting models, training models, evaluating them — assumes you inherit a pretrained model and adapt it.

The corollary is that model selection is a more powerful lever than any adaptation stage. Picking a stronger base model raises the capability ceiling instantly; adapting a weak one raises nothing. When a scenario question offers "fine-tune the small model harder" against "evaluate whether a larger base model clears the bar," the second is usually the better engineering answer, and it is always the cheaper experiment to run first. 10-02 is the lesson that makes zero-shot capability testing the first move rather than an afterthought.

None of them. The cheapest thing to try is a better prompt, and the second cheapest is retrieval, and both leave the weights untouched — meaning both can be reverted in seconds, versioned in git, and evaluated the same afternoon. The customisation ladder exists in cost order for this reason: prompt engineering, then RAG, then prompt learning methods such as prompt tuning and p-tuning, then parameter-efficient fine-tuning with LoRA or adapters, then a full fine-tune, then alignment. Each rung up costs more in data, compute, and operational complexity, and each rung up is harder to undo.

Among the weight-changing stages specifically, supervised fine-tuning with a parameter-efficient method is the cheapest real option, which is why 11-05 gets the most minutes in this module. Continued pretraining is more expensive than SFT for most teams because it is token-bound rather than example-bound. Alignment is the most operationally demanding because it needs a preference-labelling pipeline with live humans in it, and 11-07 is where the cost of that data becomes visible.

• The data shape is the definition. Unlabeled corpus means a pretraining-family stage. Instruction-response pairs mean supervised fine-tuning. Comparative preference judgements mean alignment. This one test answers most exam questions in this family.
• All three change weights, which puts all three above the line from prompting and RAG — and makes all three subject to the same drawbacks: frozen facts, no citations, no deletion path.
• The objective never changes; the distribution does. Next-token cross-entropy runs through the entire stack. What differs is what you feed it and which tokens you grade.
• Continued pretraining buys domain fluency. Instruction tuning buys behaviour. Neither buys reliable facts, and confusing fluency with accuracy is how fine-tuned models become confidently wrong in a house voice.
• Pretraining is not on your menu. Any answer proposing training a foundation model from scratch on company data is wrong by orders of magnitude in both corpus size and compute.
• The pipeline is ordered. Pretraining → continued pretraining (optional) → instruction tuning → alignment. Alignment cannot precede SFT; SFT cannot precede pretraining.
• Capability is set early. Instruction tuning surfaces latent ability and formats it. It does not create ability the base model lacks — which makes model selection the strongest lever you have.
• Baseline before you train. Every stage in this module assumes an evaluation set that predates the run, because without one you cannot tell improvement from regression.

You now have the map, and the very next region on it is the one people get wrong most often. Supervised fine-tuning is the stage application teams actually run, it is the stage vendors market hardest, and it is the stage whose capabilities are most widely misunderstood: it changes style, format, tone, task framing, and refusal behaviour with striking reliability, and it installs facts only brittlely. Next: 11-02 takes SFT apart claim by claim — what it genuinely delivers, what it only appears to deliver, and how to tell before you spend the compute.