Prompt, RAG, or fine-tune? A first decision rule

It is the choice of where to intervene when a model's output is not what your application needs. Three interventions, three distinct things they change:

The full customization ladder, which is the framing the NCA-GENL blueprint uses and which you should be able to recite in order:

prompting → RAG → prompt learning (prompt tuning, p-tuning) → PEFT / LoRA / adapters → full fine-tuning → alignment (RLHF, DPO)

Cost, data requirement, expertise, and commitment all increase left to right. The dividing line the exam cares about most: the first two rungs change nothing about the model. Prompt learning trains a small artifact while the base model stays frozen. The last three produce weights. 11-05 covers LoRA, 11-06 covers RLHF, and 11-09 covers choosing under real constraints.

L1 — Intuition: three different kinds of missing

Read the failure, not the symptom. The output being "bad" is not a diagnosis.

• If a competent contractor who knew nothing about your company could have produced the right answer given only your prompt, and the model did not — your instruction was underspecified. Prompt.
• If nobody could have produced the right answer without access to your documents — the facts were missing. RAG.
• If the answer's content was right and its form, register, or task-specific behaviour was persistently wrong across many attempts and much prompt effort — the behaviour is missing. Fine-tune.

The single sharpest version, worth memorising: RAG changes what the model knows; fine-tuning changes how the model behaves; prompting changes what you asked for.

L2 — Mechanism: a diagnostic sequence you can actually run

Work it in this order. Each step is cheaper than the one after it and rules out a whole class of cause.

Step 0 — build or reuse the eval set. Twenty hand-written cases minimum (01-08). Without a baseline number, every subsequent step is opinion. This is not optional and it is the step teams skip.

Step 1 — tighten the prompt. Specific instruction, delimited context, explicit output contract, escape value (05-02). Re-measure. A large share of apparent capability failures die here, at zero recurring cost.

Step 2 — add examples, targeted at boundaries. Two to five demonstrations covering the label space and the edge cases you actually got wrong (05-01). Re-measure.

Step 3 — add reasoning structure, if and only if the task is multi-step. An enumerated checklist chain beats "think step by step" (05-03). Re-measure, and check cost and latency, not only accuracy.

Step 4 — constrain the output. If the remaining failures are format failures, this is a decoding-layer problem, not a prompt problem (05-05). Re-measure.

Step 5 — ask the knowledge question. Does the model produce confident wrong facts about your domain — your product names, your policies, events after its cutoff? If yes, no prompt fixes it. Go to RAG (07-09).

Step 6 — consider fine-tuning. Only when: the knowledge question is answered no or already handled by retrieval; the residual failure is behavioural or stylistic and survived steps 1–4; you have or can build hundreds to thousands of high-quality examples (08-01); the volume is high enough that a shorter prompt pays back the training cost; and you can measure whether it worked (Module 9).

The property that makes this sequence worth following is that each step is independently reversible and produces a measurement. Fine-tuning is the only rung that is expensive to undo, and it is the last one for that reason as much as for cost.

L3 — Where this first rule is wrong, stated explicitly

Three places, and being able to name them is worth more than the rule itself.

It underrates fine-tuning for format and style compliance. This is the acknowledged error direction. If your requirement is that every output follow a house style, a domain register, a legal template, or a rigidly structured form, and follow it on a hundred thousand calls a day, a fine-tune is often the correct answer rather than the escalation of last resort. The reasons are concrete: the prompt gets shorter, which cuts token cost and latency on every request; compliance becomes more consistent than any instruction achieves; and format and style are precisely what supervised fine-tuning is good at, because they are demonstrable in examples (11-02). The "prompt first" framing talks teams out of this, and they end up maintaining a 1,200-token style prompt on every call forever. Do the arithmetic before you accept the default.

It undersells RAG's difficulty. "Just add RAG" reads as one rung and is a pipeline: parse (06-01), chunk (06-02), deduplicate (06-04), embed, index, retrieve, rerank (07-07), assemble (07-08), and every stage has silent failure modes. It is cheaper than fine-tuning in commitment, not necessarily in engineering weeks. 07-12 is a whole lesson on when RAG is the wrong tool.

The rungs are not exclusive, and production systems usually combine them. A realistic mature system is a fine-tuned or instruction-tuned model, given a carefully structured prompt, filled with retrieved passages, emitting schema-constrained output. Any question or framing that presents these as mutually exclusive is oversimplifying — including this lesson's own table. The rule is a triage order, not a partition.

Three rows carry disproportionate exam weight.

Freshness. A fine-tuned model is frozen at training time. If your answers must reflect data that changes weekly, retraining weekly is not a strategy and RAG is the answer. This is the cleanest single discriminator between the two.

Provenance. RAG can return the passage it used, which is what makes an answer checkable — and which connects to NVIDIA's Transparency pillar and to the citation behaviour in NVIDIA's own described RAG pipeline [NVIDIA-DOC]. A fine-tuned model cannot cite anything; the knowledge is diffused through weights (01-02).

Access control. Documents in an index carry permissions you can filter on per user. Facts baked into weights cannot be revoked, per-user-filtered, or forgotten. If your corpus has mixed sensitivity, that alone can decide the question (07-05, 13-05).

The four scenarios below are constructed illustrative examples designed so each isolates one diagnosis. Real cases are messier and often need two rungs.

Scenario A — "The summaries are too long and inconsistent." A team summarises incident reports. Outputs run 200–600 words, sometimes bulleted, sometimes prose, sometimes with recommendations nobody asked for.

Diagnosis: missing instruction. Nothing here requires knowledge the model lacks, and the inconsistency is exactly what an unspecified length, audience, and structure produces. Fix: prompting. Name the audience, fix the structure to four bullets, bound each bullet at 20 words, forbid recommendations, and give one example of the shape (05-02). Cost: an afternoon. Anyone proposing a fine-tune here is spending weeks on a problem a sentence solves.

Scenario B — "It invents our refund policy." A support assistant confidently states refund terms that do not match the company's actual policy, which changed in April.

Diagnosis: missing knowledge, and it changes. The model never saw your policy, and even if it had, the April revision postdates its training. Fix: RAG. Index the policy documents, retrieve the relevant clauses per query, and require the answer to cite the clause it used. Two properties make this the right rung rather than a fine-tune: when the policy changes in October you reindex instead of retraining, and the citation makes every answer checkable. This is also the reported RAG heuristic in action [FIELD] — in scenario questions, when one option proposes RAG it is frequently correct, and this is the shape of scenario where that holds.

Scenario C — "Every output must follow our regulatory disclosure template, on 200,000 calls a day." Content is correct. Form must be exact: fixed section order, mandated phrasings, a specific register. The prompt that achieves it has grown to 1,400 tokens of instructions and examples, and compliance is still imperfect.

Diagnosis: missing behaviour, at scale. And this is the case where the "prompt first" rule is at its weakest — the honest answer is that fine-tuning is likely correct here, not as a last resort but on the merits. Format and style are demonstrable in examples, which is what supervised fine-tuning does well (11-02); a fine-tune moves the 1,400 tokens out of every request, cutting cost and prefill latency at 200,000 calls a day; and consistency on a rigid template tends to be higher from trained behaviour than from instruction-following. What you still owe before committing: the payback arithmetic (12-09), a dataset of several hundred to a few thousand compliant examples (08-01), a memory plan (11-04), and a way to measure compliance (Module 9). Note also that a decoding constraint (05-05) may cover part of the requirement far more cheaply, and a hybrid — fine-tuned for register, constrained for structure — is often the real answer.

Scenario D — "It answers questions about our API but gets the newest endpoints wrong and won't use our error-handling conventions." Two problems in one sentence.

Diagnosis: both. The newest endpoints are missing knowledge that changes — RAG over the API documentation. The error-handling conventions are behaviour, and if prompting cannot hold them, a fine-tune can. Fix: both rungs, and separate the measurements, because a combined change tells you nothing about which half worked (05-04, 10-03). This scenario is the most realistic of the four, and it is the one the neat table in §3 fits worst.

The last two rows are the most commonly needed answers in real life and the least satisfying ones. "You do not have enough information to choose" is a legitimate output of a decision rule, and it is the correct output more often than the enthusiasm in the room suggests.

This decision is examined from several directions at once. Objective 1.3 (build LLM use cases such as RAG, chatbots, and summarizers) and 4.2 (the same, in Software Development) put RAG in the objective list by name. Objective 1.4 (curate and embed content datasets for RAGs) assumes you have decided RAG is right. Objective 1.9 owns the prompting rung. Objectives 1.1 / 4.1 (assist in deployment and evaluation of model scalability, performance, and reliability) and 1.7 (read research papers to identify emerging trends) reach the customization ladder, which is where LoRA and PEFT live. RAG appears in the official objective list three times across domains, which is why this course allocates it heavily.

Two calibration facts shape how questions here are keyed [FIELD]. First, the RAG heuristic: in scenario questions, when one option proposes a RAG solution it is frequently the correct answer. That is a reported regularity, not a rule of nature, and it is taught here with its counter-cases precisely so you do not apply it blindly — RAG is wrong when there is no corpus, when the need is style or format compliance, and when a latency floor cannot absorb a retrieval hop (07-12). Second, NVIDIA-branded answers tend to be favoured when two options are technically defensible, so an option naming NeMo Retriever, NIM, or an AI Blueprint for a stack question is worth a second look (12-13).

Phrasings that recur:

• "A company wants an assistant that answers from its internal documentation, which is updated weekly. Which approach?" — RAG. Fine-tuning fails the freshness test.
• "Which approach lets the system cite its sources?" — RAG.
• "Which of these does not modify model weights?" — prompting and RAG. Distractors: LoRA, full fine-tuning, RLHF.
• "The model's answers are correct but always in the wrong tone. Cheapest effective fix?" — prompting first. Watch the word "cheapest."
• "When is fine-tuning preferred over RAG?" — for durable behaviour, style, format, or task specialisation, and to shorten a long prompt at high volume. Not for freshness or provenance.
• "Place these in order of increasing cost: RAG, prompting, full fine-tuning, LoRA." — prompting → RAG → LoRA → full fine-tuning.
• "What is the first step before choosing an adaptation strategy?" — establish a baseline on an evaluation set.

Distractor families:

When the gap is behaviour, not knowledge. Concretely, five conditions, and the honest answer is that you want most of them to hold:

1. The output's content is already acceptable and its form, style, register, or task-specific behaviour is the persistent problem.
2. Prompting and output constraints were tried and measured, and a residual gap survived them.
3. The behaviour is stable — it does not change weekly, because a fine-tune freezes at training time.
4. You have or can build hundreds to thousands of high-quality demonstration pairs (08-01).
5. Volume is high enough that removing a long prompt from every request pays back the training and maintenance cost (12-09).

The strongest positive case, and the one this lesson's rule underrates, is format and style compliance at scale. A rigid house template, a regulated disclosure form, a domain register, a consistent tone across millions of calls — these are demonstrable in examples, which is exactly what supervised fine-tuning learns from, and they are expensive to hold via a long prompt on every request. If your Scenario C looks like §4's, fine-tuning is a legitimate first-choice answer, not a failure of prompt engineering.

Where fine-tuning is the wrong answer, stated equally plainly: to install facts, to keep answers current, to obtain citations, to control per-user access, or to reduce hallucination without grounding. Those are all retrieval properties, and no amount of training buys them.

Prefer LoRA or another PEFT method over a full fine-tune for almost every application case: far less memory, a small portable artifact, faster iteration, and much less risk of catastrophic forgetting (11-05, 11-03). Full fine-tuning and alignment are provider-scale and research-scale activities that an associate contributes to rather than initiates, which matches how the official objectives are worded [OFFICIAL].

Yes, and the mature systems do. They address orthogonal gaps, so combining them is the normal end state rather than an exotic architecture.

The standard composition: fine-tune for how, retrieve for what. A model fine-tuned for your domain's register, output template, and task behaviour, given retrieved passages for the facts, emitting schema-constrained output. Each layer does the job it is actually good at — the fine-tune supplies consistent form without a 1,500-token prompt, retrieval supplies fresh, citable, permission-filtered content, and the constraint supplies a parseable payload.

Three practical cautions. Sequence them and measure separately, because deploying both at once leaves you unable to attribute the change, and one of the two may have contributed nothing (10-03). Fine-tuning does not remove the need to ground — a specialised model with no retrieval is a more confident hallucinator, not a less frequent one (09-12). And the operational surface doubles: you now version a training dataset, an adapter, a prompt template, a schema, and an index, and any of them can drift (05-04, 12-12, 12-14).

One combination to be wary of: fine-tuning on your corpus in the hope of avoiding retrieval. It is a recurring instinct and it inherits the worst of both — no citations, no freshness, no access control, plus training cost. If the corpus is the answer source, retrieve from it.

Build the evaluation set, and get a baseline number. Everything else is downstream of that.

Twenty hand-written cases with expected outputs is enough to start, and 01-08 deliberately keeps it crude — real inputs, obvious expected answers, exact match and eyeball. Then score the current system. The number will be uncomfortable and that is the point: it converts "the model is bad at this" into "the model is at 11/20, and 6 of the 9 failures are format problems." That breakdown is the diagnosis. Format failures point at prompting and constraints. Wrong-fact failures point at retrieval. Persistent form failures that survive prompting point at fine-tuning.

Then, before committing to a rung: estimate the recurring cost at projected volume (12-09), check the latency budget against what the rung adds (12-10), ask how often the underlying information changes, ask whether answers need provenance, and ask whether different users may see different content. Those five questions decide most cases faster than any amount of discussion about model capability.

And keep the rule's error bars in view. It is a triage order that is roughly right; it underrates fine-tuning for format and style compliance; the rungs compose rather than exclude; and you cannot yet evaluate the exception, because that needs Module 9's metrics and 11-04's memory arithmetic. That is the honest state of your knowledge at lesson 36 of 106, and knowing it is more useful than a confident rule you cannot audit.

1. Diagnose the gap first: missing instruction → prompt, missing knowledge → RAG, missing behaviour at scale → fine-tune.
2. Climb from the cheapest rung and stop at the first that measurably works.
3. Prompting and RAG change no weights. Prompt learning trains a small artifact. LoRA, full fine-tuning, and alignment produce weights.
4. RAG owns freshness, provenance, and access control. No amount of fine-tuning buys any of the three.
5. Fine-tuning owns durable behaviour, style, format, and task specialisation — and this rule's known weak spot is underrating it there.
6. Never fine-tune to install facts. No citations, stale on arrival, and it does not reliably work.
7. The rungs compose. Fine-tune for how, retrieve for what, constrain the output — but sequence and measure them separately.
8. Prefer LoRA/PEFT over a full fine-tune for application work: less memory, portable artifact, less forgetting.
9. The eval set comes before the decision. A failure breakdown is the diagnosis; without a baseline every choice is a guess.
10. Apply the RAG heuristic with its counter-cases attached — no corpus, style-and-format needs, and hard latency floors are where it fails.
11. This is a first rule with stated error bars. 11-08 gives the full version once you can measure the exception.

If your diagnosis came out as "missing knowledge," you have chosen retrieval — and retrieval starts several stages earlier than most tutorials admit. Before any embedding, any vector search, any ranking, there is a question nobody asks: what did your documents become when a parser read them? A PDF's two-column layout interleaved into nonsense, a table flattened into a run of unlabelled numbers, a scanned page that produced no text at all and no error either. Every one of those failures is silent, and every downstream stage will faithfully process the wreckage. Next: 06-01 covers document parsing for RAG — PDFs, tables, and the silent failures that determine your retrieval ceiling before you have written a line of retrieval code.