Zero-Shot and Few-Shot Capability Testing: The Cheapest LLM Experiment

Zero-shot capability testing gives the model a task description and an input, with no worked examples, and scores the output. Few-shot capability testing gives the model the same task description plus k worked input–output pairs (the "shots") before the real input, and scores the output again. The pair of measurements — zero-shot score and few-shot score on the same items — is the capability test. Neither number alone tells you much; the two together, plus the requirement you are trying to hit, tell you what to build.

Three properties make this the cheapest experiment in the field:

• No training. No weights change, so there is no GPU budget, no data pipeline, no fine-tuning job, no evaluation of a training run. 11-01 and 11-02 cover what changing weights actually involves; the point of a capability test is that you do none of it.
• No infrastructure. No vector database, no chunking strategy, no retrieval stack. Twenty items, a prompt, an API call, a spreadsheet.
• No commitment. The output is a number and a failure list, both disposable. If the answer is "the base model already does this at 95%", you have saved the entire project cost for the price of an afternoon.

The word "capability" is doing real work in the name. You are not measuring the finished product's quality, you are measuring the raw material's — what the model brings to the table before your engineering is added. This distinguishes it from the other tests in the module: a regression suite (10-04) measures whether your system got worse, an online A/B test (10-03) measures whether users noticed, and a capability test measures whether the thing you are about to build is necessary.

There is a specific, load-bearing distinction to keep straight throughout. In-context learning — the phenomenon by which a model appears to acquire a task from examples in its prompt, taught in 05-01 — is what makes few-shot prompting work. Few-shot capability testing is the experiment that measures how much it is worth on your task. One is a mechanism; the other is a measurement of that mechanism's payoff in your specific case. Candidates conflate them constantly, and the exam rewards knowing that in-context learning changes no weights: the "learning" is transient, confined to a single forward pass, and gone the moment the context window is cleared.

L1 — Intuition: ask before you build

You have been asked to build a system that classifies incoming support tickets into twelve categories. The proposal on the table is a fine-tuned classifier trained on a year of labelled tickets, and it is scoped at six weeks.

Before any of that, take twenty real tickets, write down the correct category for each by hand, paste the twelve category definitions into a prompt, and ask a base model to classify each ticket. That is a zero-shot capability test, it takes about ninety minutes including the labelling, and it has exactly three possible outcomes:

1. The model is already good enough. Ship a prompt. The six-week project was unnecessary and you found out on day one.
2. The model is close but not there. Now the question is which intervention closes a known-size gap: better instructions, few-shot examples, retrieval of similar past tickets, or a fine-tune. You have converted "build a classifier" into a targeted engineering problem with a measurable target.
3. The model is far off. This is the most valuable outcome, because it usually means the failure list tells you the task was misdescribed — two of the twelve categories overlap, or the ground-truth labels disagree with each other, or the tickets do not contain the information needed to decide. No amount of modelling fixes any of those, and you would have discovered them in week five.

Outcome 3 is why this lesson exists in the experimentation module rather than in the prompting module. The capability test is not primarily a prompting exercise. It is a cheap probe of whether your problem is well-formed, and ill-formed problems are the dominant cause of failed LLM projects.

L2 — Mechanism: the seven-step protocol

A capability test that produces a defensible number follows a fixed protocol. Each step exists because skipping it is a known way to get a misleading result.

Step 1 — Freeze the item set before you look at any output. Draw 20 to 50 items from your real input distribution, including the awkward ones, and write the expected output for each. Freeze it. 01-08 covers construction and 09-01 covers scaling to a hundred items; for a capability test, 20 is a legitimate size provided you interpret it as 20 — which the arithmetic in §4 makes concrete.

Step 2 — Define the scoring rule before you see a single response. Exact match, a rubric with named levels, a keyword check, a metric from 09-05. Write it down. The failure mode here is deciding what counts as correct after seeing the outputs, which converts a measurement into a rationalisation.

Step 3 — Fix the decoding settings and record them. Temperature, top-p, max tokens, seed if available. Then hold them constant across every arm of the experiment. 04-05 explains what the settings do; the experimental requirement is only that they do not vary between conditions, because a decoding change is a confound that will be attributed to whatever else you changed.

Step 4 — Run the zero-shot arm. Instructions plus input, no examples. Score every item. This is your control, and it is the number every later claim is measured against.

Step 5 — Run the few-shot arms, one k at a time. Typically k = 1, 3, and 5, with the examples drawn from a pool disjoint from the eval items — otherwise you are showing the model the answers to its own exam, which is contamination committed by your own hand and precisely the mechanism 10-01 warned about. Score each arm on the same frozen items.

Step 6 — Ablate. Change one thing at a time and re-score: remove the instruction and keep the examples; keep the instruction and remove the examples; shuffle the example order; swap in different examples of the same count; add an output-format specification. Each single-variable comparison attributes a share of the score to a component. Multi-variable changes tell you nothing about which change mattered.

Step 7 — Write up the failure list, not just the score. Every wrong item, categorised by why it was wrong. The list is the deliverable; the score is a summary of it. 09-13 is the general treatment of turning failures into a fix list, and a capability test is its first and cheapest application.

L3 — Depth: what the shots are actually doing, and why more is not better

Few-shot examples are doing at least four separable jobs, and knowing which one your examples are performing tells you whether adding more will help.

Two mechanisms cut against adding shots, and they are the reason "more examples" is not a strategy:

Context cost. Every shot occupies context-window budget and every token costs money and latency. If each example is 300 tokens, k=5 spends 1,500 tokens on every single request, forever. That is a per-request tax paid for a one-time gain measured on 20 items. 04-06 covers context budgeting and 12-09 covers per-token cost; the experimental point is that the shots' benefit must be weighed against a recurring cost, which makes k a genuine optimisation variable rather than a knob to max out.

Example-selection sensitivity. The reported few-shot number depends on which examples you chose and, measurably in the literature, on the order they appear in. This is not a nuisance to be smoothed over — it is a result. If your k=3 score swings materially when you swap the three examples for three others from the same pool, then your headline few-shot number is a property of your example choice, not of the model's capability, and you must report it as a range across several example sets rather than as a point. Running the same k with three different example sets and reporting the spread is the single most under-practised discipline in capability testing.

A third depth point, easy to miss: a zero-shot failure and a few-shot failure have different diagnostic value. Zero-shot failure with few-shot success says the model can do the task but did not know what you wanted — a communication problem, cheaply fixed in the prompt. Failure in both says the capability or the knowledge is absent — which is where 05-06 and 11-08's decision rule engages: missing knowledge points toward retrieval, missing behaviour or format discipline points toward fine-tuning, and a missing ability to reason through the task points toward a different model.

The comparison table below is the exam's favourite shape in this area, because every row is a distractor for every other row.

Two rows deserve emphasis because they are where questions are keyed. First: few-shot prompting modifies no weights. An option claiming that few-shot learning "trains the model on the examples" or "updates the model with your data" is wrong, however plausibly worded. Second: the RAG-versus-fine-tune split is knowledge versus behaviour. If the failure list says the model does not know something, retrieval is indicated; if it says the model does not behave the way you need, fine-tuning is indicated. 11-08 is the full decision rule and 05-06 is the first-pass version.

A second, tighter comparison — the three test types this module contains:

Constructed scenario. Every number below is invented so the arithmetic is checkable. None is a measured result for any real model, dataset or product. The arithmetic itself is exact from the stated inputs.

The task: classify an inbound support ticket into one of six categories. The requirement handed to you is 85% accuracy, because below that a human has to re-check everything and the automation saves nothing.

Setup. You build a frozen set of 25 tickets sampled across the six categories, including four that you personally found ambiguous. You label them by hand, and a colleague independently labels the same 25; you disagree on 3, discuss, and resolve all 3 — that disagreement rate is itself a finding, recorded, because it caps how high any measured accuracy can meaningfully go (09-03 treats inter-annotator agreement properly). Scoring is exact match on the category label. Temperature 0, max tokens 20, same template shape in every arm. Few-shot examples come from a separate pool of 30 tickets that are not in the 25.

Arm A — zero-shot. Instruction plus the six category definitions plus the ticket.

Correct: 16 / 25 = 0.640
SE = sqrt(0.640 * 0.360 / 25) = sqrt(0.2304 / 25) = sqrt(0.009216) = 0.0960
95% interval: 0.640 ± 1.96 * 0.0960 = 0.640 ± 0.188 → [0.452, 0.828]

Stop and read that interval, because it is the most important number in this lesson. With 25 items your estimate of accuracy spans roughly 45% to 83%. The requirement is 85%. You cannot yet distinguish "this model is nearly useless" from "this model nearly meets the bar." A 25-item capability test is a screening instrument, not a decision instrument — and reporting 64% without the interval is how a team talks itself into a six-week project on evidence that supports neither the project nor its cancellation.

Arm B — few-shot, k=3, example set 1. Three worked examples prepended.

Correct: 20 / 25 = 0.800
SE = sqrt(0.800 * 0.200 / 25) = sqrt(0.16 / 25) = sqrt(0.0064) = 0.0800
95% interval: 0.800 ± 1.96 * 0.0800 → [0.643, 0.957]

The paired comparison, which is the one that matters. The two arms saw the same 25 items, so the right analysis is on the items whose outcome changed. Suppose 5 items went wrong→right and 1 went right→wrong; the other 19 were unchanged. The discordant count is 6.

Net change: +5 - 1 = +4 items = +0.160 accuracy
Discordant pairs: n_d = 6
SE of the paired difference ≈ sqrt(n_d) / n = sqrt(6) / 25 = 2.449 / 25 = 0.098
95% interval on the change: 0.160 ± 1.96 * 0.098 = 0.160 ± 0.192 → [-0.032, 0.352]

The paired interval just includes zero. The honest write-up is: "few-shot with three examples appears to help by about 16 points, but on 25 items the estimate is not distinguishable from no effect." That is a real finding and it is not a failure of the experiment — it is the experiment correctly telling you that 25 items cannot resolve a 16-point difference, which is a fact about your sample size and not about the model.

Arm C — the sensitivity check that most people skip. Re-run k=3 with two different example sets from the same pool of 30.

Example set 1: 20 / 25 = 0.800
Example set 2: 17 / 25 = 0.680
Example set 3: 21 / 25 = 0.840
Spread across three example sets: 0.680 to 0.840 (16 points)

The spread from merely changing which three examples you used is the same size as the effect you were trying to measure. This single check reframes the whole result: the headline "few-shot gets us to 80%" is not a property of the model, it is a property of example set 1. Report the range. And note the actionable consequence — if example choice matters this much, dynamic example selection (retrieving the k nearest past tickets per query rather than using three fixed ones) is now a well-motivated design candidate, discovered from a 75-call experiment rather than from a blog post.

Arm D — the format ablation. Add "Respond with only the category name, exactly as written above" to the instruction, zero-shot, no examples.

Zero-shot + format instruction: 19 / 25 = 0.760
vs plain zero-shot:             16 / 25 = 0.640
Change: +3 items = +0.120

Inspection of the 3 newly-correct items shows all three had previously produced the right category buried in a sentence of explanation that the exact-match scorer marked wrong. So a meaningful share of what looked like a capability gap was a formatting gap, fixable with one sentence and zero recurring token cost — whereas k=3 few-shot buys a similar-looking gain and charges for three examples on every request forever. That comparison is only visible because you ablated one variable at a time.

How to size the follow-up. You now want to resolve whether the system reaches 85%. To detect a difference of about 10 percentage points around p ≈ 0.8 with conventional 80% power at the 5% level, the standard two-proportion sizing arithmetic gives roughly:

n per arm ≈ 16 * p * (1 - p) / d^2
          = 16 * 0.8 * 0.2 / 0.10^2
          = 16 * 0.16 / 0.01
          = 2.56 / 0.01
          = 256 items per arm

Two hundred and fifty-six items per arm to resolve a 10-point difference. That is the number nobody wants to see, and it is the honest cost of a decision-grade offline comparison. Halve the difference you want to detect and the requirement quadruples: at d = 0.05 it is 1,024 per arm. The 16 in that formula is a standard approximation bundling the z-values for 80% power and a two-sided 5% test; 09-09 develops it properly. The practical consequence for your project plan is that a 25-item test screens and a 250-item set decides, and the transition between them is exactly the work 09-01 describes.

The write-up that comes out of this. Not a score — a decision memo:

On 25 hand-labelled tickets, the base model classified 64% correctly zero-shot (95% CI 45–83%). Adding a format instruction alone lifted this to 76%; three inspected gains were formatting, not classification. Three-shot prompting reached 68–84% depending on which three examples were used, a spread as large as the effect. Human labellers disagreed on 3 of 25 items before adjudication, which bounds achievable accuracy. Conclusion: the 85% requirement is plausibly reachable without fine-tuning, the binding constraints appear to be output formatting and two overlapping category definitions, and the next step is a 250-item eval set plus a category-definition review — not a training pipeline. Estimated cost of next step: two days, versus six weeks for the original proposal.

That memo is the product of the lesson. It is also, in miniature, a proof-of-concept report: hypothesis, method, numbers with uncertainty, negative findings included, and a recommendation whose cost is stated.

The one rule to carry: a capability test's job is to make the next decision cheaper, not to be the last word. Run it small, read the interval honestly, and let the failure list — not the score — drive what you build.

The Experimentation section of the blueprint carries 22% of the exam, and zero-shot testing is named explicitly in that section's suggested-reading list — one of the few places where a specific evaluation practice is called out by name rather than implied. The section's scope statement, "the study of how to perform, evaluate, and interpret experiments, including AI model evaluation and the use of human subjects in labeling or reinforcement learning from human feedback," covers this lesson twice over: the capability test is the experiment, and the hand-labelling of your 25 items is the use of human subjects in labelling.

The objective-numbering defect, and how to cite around it. There is a verified defect in the official study guide that you should understand before it costs you a question. The objectives printed under Experimentation as 3.1–3.5 are a verbatim duplicate of the objectives printed under Data Analysis and Visualization as 2.1–2.5 — they describe data mining, comparing models with statistical metrics, conducting data analysis under supervision, creating graphs and charts, and identifying relationships and trends. Read literally, the section that is 22% of the exam would contain no stated coverage of model evaluation or RLHF, contradicting its own scope sentence. The scope statement, the section's course objectives and its reading list all agree with each other and disagree with the printed objective text, so the derived scope governs and the printed ids serve only as traceability.

For this lesson the ids that genuinely apply are 1.9 ("experimentation" in the job-role description, which names A/B testing, evaluating prompts, evaluating models and producing POCs as associate responsibilities) and 2.2 / 3.5 in their printed forms — comparing models using statistical performance metrics, and identifying factors that could affect the results of research. Example-selection sensitivity, the interval on a 25-item proportion, and human label disagreement are all, precisely, factors that could affect the results of research. The lesson also serves the official responsibility language directly: an associate is expected to select models and produce proofs of concept under senior supervision, and a capability test is how both are done defensibly.

Question phrasings to expect:

Distractor families:

1. "Few-shot means training on a few examples." The most common trap in this area. Few-shot prompting performs no gradient update. If an option describes examples "training", "updating" or "teaching the weights", it is wrong.
2. "More shots is always better." Wrong on two counts: gains flatten quickly, and the token cost is recurring. An option recommending "add as many examples as fit in the context window" is a distractor.
3. "Fine-tune first." An option that jumps to fine-tuning without any baseline measurement is nearly always the wrong answer on this exam, both because it is more expensive and because it is unjustified without a measured gap.
4. "Use a public benchmark to test capability on our task." Ruled out by 10-01: the benchmark's population is not your task's.
5. The RAG-heuristic overreach. Candidate reports note that when an option proposes RAG it is frequently correct — but here the counter-case matters: if the failure is format or style, RAG does nothing. Apply the heuristic to knowledge gaps, not to every gap. 07-12 covers when RAG is the wrong tool.

No, and this is the single most examinable confusion in the area. Few-shot prompting places examples in the model's context window at inference time. No gradient is computed, no parameter changes, and the effect vanishes the moment that request ends. Fine-tuning — including the parameter-efficient variety in 11-05 — computes gradients and modifies weights (or adds trained adapter weights), and the change persists across all future requests with no examples in the prompt.

The practical consequences follow directly from that mechanical difference. Few-shot costs tokens on every request but nothing up front, and you can change the examples instantly. Fine-tuning costs GPU time and a data pipeline up front but nothing per request, and changing its behaviour means training again. Few-shot cannot reliably install new factual knowledge, and neither can fine-tuning — that is retrieval's job. Where fine-tuning genuinely wins is permanent, consistent behaviour at zero per-request context cost, which matters exactly when you are paying for the same 1,500 example tokens on millions of requests.

The order of operations that follows: measure zero-shot, add a format instruction, add few-shot, add retrieval if the gap is knowledge, and consider fine-tuning last — when you can name the behaviour that prompting could not stabilise and you have a measured baseline to beat.

Start at k=1 and increase only while the measured gain exceeds the token cost. There is no universal optimum, and treating this as a number to be looked up rather than measured is the mistake.

What the mechanism suggests you will see: if the shots are doing format specification, k=1 captures nearly the whole benefit and k=5 wastes four examples' worth of tokens on every request. If they are doing label-space definition, expect the gain to run to roughly one example per class and then flatten. If they are doing boundary calibration, only examples near the actual confusion help, so k matters less than which examples you pick — and that is the finding that motivates dynamic, retrieval-based example selection.

The disciplined way to answer for your task: run k = 0, 1, 3, 5 on the same frozen items with the same decoding, run each k with at least three different example sets, and plot score against total prompt tokens rather than against k. That last substitution is the one people miss. The decision variable is not "how many examples" but "how much context am I willing to spend per request, forever, for this much accuracy" — and once it is framed that way, 04-06's context budget and 12-09's cost model are part of the same analysis.

Yes, in three distinct ways, and each is a legitimate saving rather than a disappointment.

It can cancel the project because the work is already done. If the base model hits your requirement zero-shot, the remaining work is a prompt, a template version (05-04) and a regression gate. Everything else that was proposed was solving a problem that did not exist.

It can cancel the project because the problem is not the one described. The failure list, not the score, does this. When misses cluster on two overlapping category definitions, or on items whose ground-truth labels your own annotators disputed, or on inputs that simply do not contain the information the task requires, no model change is the fix. The correct output of the experiment is a redefinition, and it arrived in week one instead of week five.

It can cancel the project because the ceiling is below the requirement for a structural reason. If human annotators agree on only 88% of items, an 85% requirement is near the noise floor of the ground truth itself, and a "95% accuracy" target is not a model problem — it is an impossible specification that someone needs to be told about before money is spent.

The candour to keep: a capability test cannot prove a project unnecessary on 25 items, because the interval is too wide for a proof. What it does is reallocate the burden of proof. After a capability test, the person proposing six weeks of work has to explain what specifically the measured gap requires, and that conversation is dramatically better informed than the one you would otherwise have had.

Use the settings you intend to ship with, and hold them identical across every arm. If that is not yet decided, run at temperature 0 (or the lowest your provider allows) as the default, because you are measuring capability rather than sampling behaviour and you want run-to-run variation minimised.

Two honest caveats. First, temperature 0 is not fully deterministic in practice — batching, hardware non-determinism and provider-side changes all introduce variation, which is 09-11's subject. So re-run at least one arm twice and record whether the score moved; if it did, that variation is a floor under every difference you can claim to detect. Second, if your product will ship at a higher temperature because it needs varied phrasing, then a temperature-0 capability test overstates what users will get, and you should run at least one arm at the shipping temperature with several samples per item to see the spread. Recording the settings is not bureaucracy — it is what makes the number reproducible, and an unlabelled score is the same unusable artefact 10-01 described for public benchmarks.

1. Run it first, always. A 20–50 item capability test on your own data costs hours and is the cheapest way to learn whether a proposed LLM project is correctly scoped.
2. The zero-shot arm is the control. Without it, a few-shot number is not a comparison and supports no inference.
3. Few-shot prompting changes no weights. In-context learning is transient, per-request, and gone when the context clears. Anything describing it as training is wrong.
4. Print the interval. On 25 items, a 64% score carries a 95% interval of roughly 45–83%. Small-sample scores screen; they do not decide.
5. Use a paired analysis for before/after on the same items. Counting discordant pairs is tighter and more honest than treating the two arms as independent samples.
6. Measure example-selection sensitivity. Run the same k with three different example sets. If the spread rivals the effect, your headline number is a property of your example choice — and dynamic example retrieval becomes a motivated design.
7. Ablate one variable at a time. A format instruction and three few-shot examples can produce similar-looking gains at wildly different recurring costs; only single-variable ablation tells them apart.
8. Keep the example pool disjoint from the eval set, or you have contaminated your own experiment exactly as a leaked benchmark is contaminated.
9. Read the failure list, not the score. Misses that cluster on ambiguous label definitions or on disputed ground truth are a specification problem no model fixes.
10. Diagnose the gap before choosing the remedy: missing knowledge → retrieval; wrong format or style → instruction, then few-shot, then fine-tuning; absent reasoning capability → a different model. Never fine-tune without a measured baseline.
11. Sizing arithmetic, memorised: to detect a difference d at p, roughly 16·p(1−p)/d² items per arm. At p=0.8 and d=0.10 that is 256 per arm; halving d quadruples the requirement.
12. On objective ids, remember the module's documented source defect — the printed Experimentation objectives 3.1–3.5 duplicate Data Analysis 2.1–2.5 — so cite ids for traceability and rely on the derived scope, which names zero-shot testing explicitly.

Everything in this lesson happened offline, on items you chose, scored by a rule you wrote. That is exactly why it is cheap — and exactly why it cannot tell you whether real users are better off. Your 25 tickets are not your traffic, your exact-match scorer is not user satisfaction, and a 16-point offline gain has an entirely unknown relationship to whether anyone resolves their problem faster. The only instrument that closes that gap is a randomised experiment on live traffic: assign users to control and treatment, define a primary metric and the guardrails that must not degrade, decide the sample size before you start, and resist looking at the result early.

Next: 10-03 builds that experiment — randomised assignment, guardrail metrics, the peeking problem, multiple-comparison inflation, and how to read a production A/B result correctly instead of the way most dashboards invite you to.