A/B Testing an LLM Feature in Production: Design, Guardrails, and Reading the Result

An A/B test — a randomised controlled online experiment — has six components, and a test missing any of them cannot support a causal claim.

For an LLM feature, "the change" is typically one of: a new prompt template, a different base model, a retrieval configuration, a reranker added or removed, a decoding-parameter change, a fine-tuned model swapped in, or a guardrail policy adjusted. The mechanics of the experiment are the same in each case; what differs is the size of the effect you should expect and how many things the change quietly touched.

The distinction the exam cares about is online versus offline evaluation. Offline evaluation scores a fixed dataset with a proxy metric: it is cheap, repeatable, safe, has no user risk, and measures a proxy. Online evaluation measures real user behaviour on real traffic: it is expensive, non-repeatable, carries user risk, and measures the outcome you actually care about. They are complements, not substitutes. The correct sequence is offline first to rank many variants cheaply, then online to confirm that the best offline variant moves a real outcome — because the correlation between offline proxy gains and online outcome gains is real but far from perfect, and that imperfection is the reason online tests exist.

L1 — Intuition: why randomisation is the whole trick

Suppose you ship a new prompt on Tuesday and satisfaction rises on Wednesday. What caused it? The prompt — or Wednesday's traffic mix, or the marketing email that went out Tuesday night, or the fact that a slow upstream service recovered, or the seasonal pattern that makes Wednesdays different from Tuesdays. You cannot distinguish these, and no amount of data collected the same way will help, because every one of those explanations produced exactly the same observation.

Now suppose that on Tuesday you sent a random half of users to the new prompt and a random half to the old one. Marketing hit both arms. The upstream service was slow for both arms. Wednesday happened to both arms. Every confounder you can name — and, crucially, every confounder you cannot name — is distributed across both arms by the coin flip rather than by anything correlated with the treatment. The only systematic difference between the arms is the prompt. That is why the comparison licenses a causal claim, and it is the entire reason to endure the cost of an online experiment.

The corollary is the thing to hold onto: randomisation handles unknown confounders, which is a property no amount of statistical adjustment after the fact can replicate. If you assign by anything correlated with user attributes — geography, account tier, whoever happened to be in the beta group, the users your PM picked — you have re-introduced confounding and destroyed the one advantage the design had.

L2 — Mechanism: designing the experiment, step by step

Step 1 — State the hypothesis with a direction and a minimum size. Not "the new prompt is better", but "the new prompt increases the rate of sessions ending in a resolved ticket, and an increase below 2 percentage points is not worth the added token cost." That second clause is the minimum detectable effect (MDE) — the smallest effect worth acting on — and it is what makes the experiment sizeable. Choosing it is a product decision, not a statistical one, and choosing it before the data arrives is what stops the threshold sliding to wherever the result landed.

Step 2 — Choose the randomisation unit. Almost always the user or the session, not the individual request. Randomising per request means one user sees both variants, which contaminates their experience, makes the arms non-independent, and destroys any metric measured over a session. The rule: randomise at the level at which your metric is measured, or coarser. If the metric is per-session resolution, randomise per session or per user. If users can influence each other — a shared workspace, a team inbox — randomise at the group level, or the arms leak into each other.

Step 3 — Implement assignment as a stable hash. hash(user_id + experiment_name) mod 100 gives you a deterministic bucket that survives restarts, does not require a database lookup, and — because the experiment name is in the hash — does not correlate with any other experiment's assignment. Two properties matter: stability (a user must not flip arms mid-experiment, or their experience is incoherent and their data uninterpretable) and independence (assignment must not be derivable from any user attribute).

Step 4 — Pick one primary metric and declare it. One. The primary metric is the single number that decides the experiment. If you have three "primary" metrics you have no primary metric, you have three chances to find a win, and §L3 explains exactly how much that costs you.

Step 5 — Declare the guardrail metrics. Guardrails are metrics that must not degrade, even if the primary improves. They exist because an LLM change can improve the thing you were watching while damaging something you were not. The canonical guardrail set for an LLM feature:

Step 6 — Compute the sample size before you launch, and derive a duration from it. §4 does the arithmetic. Then convert items into calendar time using your traffic rate, and round up to whole weeks. Whole weeks matter because traffic composition has a weekly cycle — weekday versus weekend users behave differently — and stopping mid-cycle gives each arm the same days but your inference an unrepresentative traffic mix.

Step 7 — Run a sanity period, and check the arms are actually balanced. Before trusting any result, verify: are the arms roughly the size the split ratio implies? Do pre-treatment characteristics match (traffic mix, device split, geography, historical activity)? A sample ratio mismatch — a 50/50 split that yielded 52/48 in a way sampling noise cannot explain — is evidence the assignment or logging is broken, and it invalidates the experiment regardless of how good the result looks. It is the single most useful diagnostic in online experimentation and the one most often skipped.

Step 8 — Read the result once, at the pre-declared end. Then apply the decision rule you wrote in step 1.

L3 — Depth: peeking, p-hacking, and multiple-comparison inflation

Three distinct statistical failures produce false wins in online experiments. They have different mechanisms and different fixes, and the exam-relevant skill is telling them apart.

Peeking. A fixed-horizon test's guarantee — a 5% false-positive rate — applies to looking once, at the pre-declared sample size. A running experiment's observed difference wanders; early on it wanders a great deal, because the standard error is largest when n is smallest. If you check the dashboard every day and stop the moment p < 0.05, you have not run a 5% test. You have run a test that stops whenever the random walk crosses a threshold, and a random walk will cross a fixed threshold eventually with probability far above 5%. Under repeated daily looks the true false-positive rate can rise to a substantial multiple of the nominal rate. Fix: commit to the horizon and read once; or, if you genuinely need to monitor continuously, use a method designed for it — sequential testing with alpha-spending, or always-valid confidence sequences — and accept the cost, which is that these methods require more data to reach the same confidence. There is no free continuous monitoring.

P-hacking / metric shopping. Peeking is looking many times at one metric. P-hacking is looking once at many things and reporting the winner: twenty metrics, five user segments, three time windows. It also covers post-hoc slicing ("it didn't work overall but it worked for mobile users in Germany"), dropping "outlier" days after seeing them, and switching the primary metric once the original disappointed. Fix: pre-register — write the primary metric, the guardrails, the segments you will examine, the exclusion rules and the decision threshold before launch, and treat everything discovered afterwards as a hypothesis for a new experiment rather than as a finding from this one. A post-hoc segment result is not a result; it is an idea.

Multiple-comparison inflation. This one has exact arithmetic, which makes it examinable. If you run m independent tests each at α = 0.05, the probability of at least one false positive is:

P(at least one false positive) = 1 - (1 - α)^m

m = 1:   1 - 0.95^1  = 0.050   ( 5.0%)
m = 3:   1 - 0.95^3  = 0.143   (14.3%)
m = 5:   1 - 0.95^5  = 0.226   (22.6%)
m = 10:  1 - 0.95^10 = 0.401   (40.1%)
m = 20:  1 - 0.95^20 = 0.642   (64.2%)

Twenty metrics on a change that does nothing at all, and you are more likely than not to find a "significant" one. This is why a single pre-declared primary metric is a design requirement rather than a stylistic preference. Fix: one primary metric decides the experiment; if you must test several primaries, correct the threshold. The Bonferroni correction divides α by the number of tests — α = 0.05 across 5 primary metrics means each is judged at 0.01 — which is conservative but trivially defensible and easy to explain to a sceptic. Guardrails are treated asymmetrically and deliberately: they are not primaries, and you want to be sensitive to guardrail damage, so applying a multiplicity correction to a guardrail (making it harder to detect harm) is the wrong direction.

The fourth failure, which is not statistical: confusing statistical significance with practical significance. A p-value answers "is this difference distinguishable from zero?" It says nothing about whether the difference is worth anything. With very large samples, a 0.2-percentage-point difference becomes statistically significant and remains commercially irrelevant — and if that change also raised cost per request by 30%, the statistically significant result is a business loss. The reverse also happens and is more often mishandled: a genuinely valuable 5-point improvement can fail to reach significance because the experiment was underpowered, and "not significant" then gets misreported as "no effect". A non-significant result with a wide interval means you did not learn anything, not that the effect is zero. Report the effect size with its confidence interval and compare that interval against the MDE you declared in step 1. That comparison, not the p-value, is the decision.

LLM-specific complications that generic A/B advice misses:

• Non-determinism. The same input can produce different outputs across calls even at temperature 0, for the reasons 09-11 details. This adds within-arm variance, which widens intervals and reduces power. The consequence is concrete: an LLM A/B test needs more traffic than an equivalent deterministic UI test to detect the same effect size.
• Novelty and primacy effects. Users react to change as change. A new interaction style can produce a temporary lift (novelty) or a temporary drop (primacy, as users unlearn habits) that decays over weeks. A one-week test can measure the transient and miss the steady state. Mitigation: run long enough to see the trend flatten, and inspect the effect by user tenure — if the lift is concentrated entirely in first-time exposures, suspect novelty.
• Cost and latency are treatment variables, not just guardrails. In most A/B testing, the change is free to serve. In LLM work, the treatment often is a cost and latency change — a bigger model, more retrieved context, an extra reranking pass. This makes the decision explicitly multi-objective, and it is why cost and latency belong in the pre-declared design and not in a post-hoc footnote.
• Interference through shared state. If both arms write to the same conversation history, cache, feedback store, or retrieval index, treatment can leak into control. Check for shared mutable state before launch; a contaminated control arm is worse than no control arm, because it biases the estimate toward zero while looking perfectly healthy.
• Ratio and per-session metrics need care. "Average tokens per response" and similar ratio metrics have variance structures that naive per-request arithmetic mishandles when requests are clustered within users. If your randomisation unit is the user and your metric is per-request, the effective sample size is closer to the number of users than the number of requests, and treating requests as independent overstates your confidence — sometimes dramatically.

Two rows carry most of the exam value. Shadow deployment is not an A/B test — it validates that the new system does not fall over, which is necessary and not sufficient; it produces no user-outcome evidence because no user saw the output. And canary rollout is risk management, not measurement: staged percentages limit damage, but unless the stages are randomised and analysed as arms, a canary tells you the system did not crash, not that it is better.

The sequence a mature team runs, and the sequence questions in this area tend to describe: offline eval on a frozen set → shadow deploy to check latency, cost and error behaviour → randomised A/B at a modest share → staged ramp with guardrails watched → full rollout with monitoring in place. Each stage answers a question the previous one could not, and each is cheaper than the one after it.

Constructed scenario. Every number below is invented so the arithmetic can be checked line by line. None is a measured result from any real product. The statistical arithmetic is exact from the stated inputs.

The change. A support assistant currently answers from a retrieval pipeline with a plain prompt. The treatment adds explicit citation instructions and one worked example, which raises the prompt by about 400 tokens per request.

The hypothesis, written before launch. "Adding citation instructions and one worked example increases the session resolution rate — the share of sessions ending without escalation to a human — from its current 24% baseline. An increase below 2 percentage points does not justify the added token cost, so the MDE is 2 points absolute."

Primary metric: session resolution rate. Randomisation unit: user, stable hash. Split: 50/50. Guardrails: p95 latency (must not rise more than 200 ms), cost per session (must not rise more than 15%), error rate, refusal rate, safety-violation rate.

Sizing arithmetic. For two proportions with 80% power and a two-sided 5% test, the working approximation is:

n per arm ≈ 16 * p * (1 - p) / d^2

p = 0.24 (baseline resolution rate)
d = 0.02 (minimum detectable effect, absolute)

n ≈ 16 * 0.24 * 0.76 / (0.02)^2
  = 16 * 0.1824 / 0.0004
  = 2.9184 / 0.0004
  = 7,296 users per arm

So roughly 7,300 users per arm, 14,600 total. The 16 bundles the z-values for 80% power and a two-sided α = 0.05; 09-09 derives it. Two sanity checks on that figure, both worth doing every time:

Halve the MDE to 1 point:   16 * 0.1824 / 0.0001 = 29,184 per arm   (4x)
Double the MDE to 4 points: 16 * 0.1824 / 0.0016 =  1,824 per arm   (1/4)

Sample size scales with 1/d². This is the most important practical fact in experiment design: detecting a small effect is quadratically expensive. If you cannot afford the sample, your only honest options are to accept a larger MDE (and say so), extend the duration, or accept lower power (and state that a null result will be uninformative).

Duration. At 1,200 experiment-eligible users per day, 14,600 users total:

14,600 / 1,200 = 12.2 days → round up to 14 days (two whole weeks)

Two whole weeks, not twelve days, so both arms cover the same weekday/weekend composition and the inference is not skewed by a partial cycle. Two weeks also gives a first look at whether any early lift is decaying, which is your novelty-effect check.

The result, read at day 14.

Control:   1,776 resolved / 7,341 users = 0.2419
Treatment: 1,985 resolved / 7,352 users = 0.2700
Observed lift: 0.2700 - 0.2419 = 0.0281  (2.81 percentage points)
Relative lift: 0.0281 / 0.2419 = 11.6%

Sample ratio check first, before anything else. Expected 50/50; observed 7,341 vs 7,352 out of 14,693. Expected per arm 7,346.5; the deviation is 5.5, while the standard deviation of the split is √(14,693 × 0.25) = √3,673 ≈ 60.6. A deviation of 5.5 against an SD of 60.6 is entirely ordinary. The split is healthy and the assignment mechanism is not suspect.

Standard error of the difference.

SE_c = sqrt(0.2419 * 0.7581 / 7341) = sqrt(0.18339 / 7341) = sqrt(0.00002499) = 0.004999
SE_t = sqrt(0.2700 * 0.7300 / 7352) = sqrt(0.19710 / 7352) = sqrt(0.00002681) = 0.005178
SE_diff = sqrt(0.004999^2 + 0.005178^2)
        = sqrt(0.00002499 + 0.00002681)
        = sqrt(0.00005180)
        = 0.007197

95% CI on the difference: 0.0281 ± 1.96 * 0.007197
                        = 0.0281 ± 0.0141
                        = [0.0140, 0.0422]
z = 0.0281 / 0.007197 = 3.90

How to read this correctly, in the right order.

1. Statistical significance. z = 3.90, and the interval [1.40, 4.22] percentage points excludes zero. The effect is distinguishable from noise.
2. Practical significance against the pre-declared MDE. You declared 2 points as the threshold worth acting on. The point estimate is 2.81 points, which clears it — but the lower bound is 1.40 points, which does not. The honest statement is: "the effect is very likely positive; it is probably but not certainly above our 2-point bar." That is a materially different sentence from "we got a 2.8-point lift," and it is the sentence a senior reviewer will ask you for.
3. Guardrails, before any celebration. Suppose p95 latency rose 120 ms (inside the 200 ms limit), error rate was flat, refusal rate was flat, safety violations were flat — and cost per session rose 18% against a 15% limit. A guardrail was breached. The correct outcome is not "ship it": it is a decision that now requires an explicit trade of 2.8 points of resolution against 18% cost, escalated to whoever owns that budget. Guardrails exist precisely to force this conversation rather than let it be discovered in next month's invoice.
4. Multiplicity check. One primary metric was declared, so no correction applies to it. Five guardrails were also examined — and because guardrails are checked for harm, no correction is applied there either; you deliberately keep them sensitive.
5. Novelty check. Split the fortnight: if week 1 showed 3.4 points and week 2 showed 2.3, the trend is downward and the steady-state effect may be below the MDE. That is not a reason to discard the result; it is a reason to state the trend and to consider a longer confirmation run.

What you may and may not conclude. You may say: among users randomised in this experiment over these two weeks, the citation prompt caused a resolution-rate increase whose 95% interval is 1.4 to 4.2 points, at an 18% cost increase per session. You may not say: it will produce 2.8 points next quarter (traffic mix changes), it will work in the other product surface (not randomised there), it works because of the citations specifically (two things changed — the citation instruction and the worked example — so this is a bundled treatment and attributing the effect to either component requires another experiment), or that the treatment is better for enterprise users (you did not pre-register that segment).

That last point is worth stating flatly: this experiment bundled two changes. The result is valid for the bundle. Splitting them is a second experiment, and it needs its own sizing arithmetic — and since each component's effect is presumably smaller than the bundle's, the 1/d² law says it will need more traffic, not less.

The underlying rule: an A/B test buys you a causal claim about a user outcome, and it costs traffic, time and risk. Spend it when the decision is genuinely uncertain, the effect is plausibly detectable given your traffic, and no user is harmed by being in the worse arm. When any of those three fails, a different instrument is the professional choice — and saying so is a stronger answer than running an underpowered test and reporting its noise.

A/B testing is named explicitly, twice, in the official material. It appears in the Experimentation section's suggested-reading list, and it appears in the certification's own job-role description, which states that the associate's responsibilities include "experimentation (A/B testing, evaluating prompts, evaluating models, producing POCs)." Very few practices in this blueprint are named that directly. The Experimentation section carries 22% of the exam, and its scope statement — "the study of how to perform, evaluate, and interpret experiments" — describes this lesson almost word for word. "Perform, evaluate and interpret" maps onto design, guardrails, and reading a result.

The objective-numbering defect and how to handle it. The objectives printed under Experimentation as 3.1–3.5 are a verified verbatim duplicate of the objectives printed under Data Analysis and Visualization as 2.1–2.5. They describe data mining, comparing models with statistical metrics, conducting data analysis under supervision, producing charts, and identifying relationships and trends. Read literally, the printed text for a section explicitly scoped to experiments and RLHF would describe neither — 22% of the exam would have no stated coverage of model evaluation. The scope statement, the section's own course objectives, and its reading list (which names A/B testing) all agree with each other and disagree with the printed objective text, so the derived scope governs and the printed ids are traceability only.

For this lesson, the printed id that genuinely applies is 3.5 / 2.5 — "identify relationships and trends or any factors that could affect the results of research." That is, almost precisely, this lesson's subject: confounders, peeking, multiplicity, novelty effects and sample ratio mismatch are all factors that affect the results of research. The job-role responsibility language (A/B testing, producing POCs) does the rest of the authorising work. So: cite 3.5 with the defect noted, and do not pretend the printed 3.1 text about data mining describes randomised online experimentation.

Question phrasings to expect:

Distractor families:

1. The pre-post distractor. An option proposes comparing this week to last week, or before-launch to after-launch, and calls it an A/B test. No concurrent control means no causal claim, ever.
2. The bigger-sample-fixes-bias distractor. An option proposes running longer or with more users to fix a design flaw — a broken split, a leaky control arm, a post-hoc metric. More data shrinks variance; it does not touch bias.
3. The significance-equals-importance distractor. An option treats p < 0.05 as sufficient grounds to ship, ignoring effect size, cost and guardrails.
4. The shadow-deployment-as-A/B distractor. An option claims that running the new model in shadow mode establishes that it is better. Shadow mode measures operational health; no user saw the output, so there is no outcome evidence.
5. The offline-is-enough distractor. An option asserts that a strong offline eval improvement means users will benefit. Proxy gains and outcome gains are correlated but not identical — that gap is why online tests exist.
6. The many-primaries distractor. An option lists five metrics as co-primary with no correction. That design has an inflated false-positive rate by construction.

Offline evaluation scores a fixed dataset with a proxy metric and no users involved; online evaluation measures real user behaviour on live traffic. The distinction is not just cost — it is what the number means.

The right relationship between them: offline evaluation is a filter, online evaluation is a decision. Offline lets you try twenty prompt variants in an afternoon and discard eighteen. Online tells you whether the surviving one matters. Teams that skip offline waste traffic on variants that a fixed eval set would have eliminated for free; teams that skip online ship proxy improvements and are surprised when nothing moves. A useful habit is to record, for each experiment, both the offline delta and the online delta — over a year that pairing tells you how well your proxy actually predicts your outcome, which is the most valuable meta-measurement an LLM team can own.

Because the 5% false-positive guarantee of a fixed-horizon test is a promise about one look at a pre-declared sample size, and peeking converts one look into many.

The mechanism, concretely: at any moment the observed difference between arms is the true difference plus noise, and the noise term is largest early because the standard error shrinks with √n. Plot the running difference and you get a random walk that starts wide and narrows. A fixed threshold — "significant at p < 0.05" — is a fixed pair of boundaries on that plot. A random walk given many opportunities to cross a boundary will cross it, even when the true difference is exactly zero. Stopping at the first crossing therefore selects for noise excursions, and the realised false-positive rate rises well above the nominal 5% as the number of looks increases.

The asymmetric damage is worth noting: peeking makes you stop early on positive noise excursions much more often than it makes you stop on negative ones, because a positive result gets shipped and a negative one gets "let it run a bit longer." So peeking does not merely add noise — it systematically biases your portfolio of shipped changes toward things that never worked.

If you genuinely need to monitor a running experiment — and for guardrails you should, because you must be able to abort a harmful treatment — the resolution is to separate the two purposes. Monitor guardrails continuously with pre-declared abort thresholds; that is safety, not inference, and stopping a harmful test early is correct. Judge the primary metric once, at the horizon. If you need valid continuous inference on the primary metric, use a sequential design with alpha-spending or always-valid confidence sequences, and budget for the extra data they require. The one thing you may not do is watch a fixed-horizon primary metric daily and stop when it looks good.

Compute it from three numbers you must decide before launch: the baseline rate p, the minimum detectable effect d you care about, and your tolerance for error (conventionally 80% power and a two-sided 5% test). The working approximation is:

n per arm ≈ 16 * p * (1 - p) / d^2

Three facts to take from that table. Sample size scales as 1/d² — halving the effect you want to detect quadruples the cost. Variance peaks at p = 0.5, so mid-range rates are the most expensive to measure. And for a continuous metric the same logic applies with variance in place of p(1−p): n ≈ 16σ²/d², which is why reducing the metric's variance (winsorising extreme values, using a per-user average, or applying variance-reduction with pre-experiment covariates) buys real power without buying traffic.

Then adjust for LLM reality in three ways. Add headroom for output non-determinism, which inflates within-arm variance. If your randomisation unit is the user but your metric is per-request, do not treat requests as independent — clustering means your effective sample size is nearer the user count. And convert to whole weeks of calendar time, rounding up.

Finally, the honest move when the arithmetic exceeds your traffic: do not shrink the test, state the MDE your traffic can actually support. "With two weeks of traffic we can detect a 5-point change; we cannot detect a 2-point change" is a professional answer that lets the decision-maker choose. Running an underpowered test and reporting its null as evidence of no effect is not.

No — and the reasoning is worth understanding because it looks inconsistent until you see the asymmetry.

Multiple-comparison correction exists to stop you claiming a win you did not earn. It raises the bar for declaring an effect, which reduces false positives at the cost of missing real effects. That trade is right for primary metrics: you want to be sceptical about your own successes.

Guardrails serve the opposite purpose. They ask "did this change break something?", and the expensive error there is a false negative — shipping a change that quietly tripled cost or doubled refusals. Correcting a guardrail's threshold makes harm harder to detect, which is precisely the wrong direction. So guardrails are deliberately kept sensitive: uncorrected thresholds, and often pre-declared absolute limits ("p95 must not rise more than 200 ms") rather than significance tests at all, because an absolute operational limit is a clearer contract than a p-value.

The practical structure that falls out: one primary metric, judged strictly and once, at the horizon; several guardrails, judged leniently and continuously, with pre-declared abort thresholds. Anything else you look at is exploratory, and exploratory findings are hypotheses for the next experiment rather than results from this one. Writing those three categories down before launch is most of what "pre-registration" means in an industrial setting, and it takes about ten minutes.

1. Randomisation is the whole point. It distributes confounders you have not thought of, which is why an A/B test supports a causal claim and a before/after comparison never does.
2. Randomise at or coarser than the metric's unit. Per-request assignment breaks session metrics and independence; use a stable hash(user_id + experiment_name).
3. One pre-declared primary metric. With m independent tests at α = 0.05, the chance of a false positive is 1 − 0.95^m: 14% at three, 40% at ten, 64% at twenty.
4. Guardrails are mandatory for LLM changes — p95 latency, cost per request, error rate, refusal rate, safety violations, escalation rate. LLM treatments routinely move cost and latency as a side effect, so those are design variables, not footnotes.
5. Guardrails are judged asymmetrically: monitored continuously, uncorrected, with pre-declared abort thresholds, because the expensive error for a guardrail is a false negative.
6. Size before you launch: n per arm ≈ 16·p(1−p)/d². Sample size scales as 1/d², so halving the detectable effect quadruples the traffic needed, and variance peaks at p = 0.5.
7. Run whole weeks and round up, because traffic composition has a weekly cycle.
8. Do not peek. A fixed-horizon α applies to one look; repeated looks let noise cross the threshold, and the bias favours shipping things that never worked. Use sequential methods if you truly need continuous inference.
9. Check sample ratio mismatch first. A split that noise cannot explain invalidates the experiment however attractive the result.
10. Significance is not importance. Report the effect size with its confidence interval and compare it against the MDE you declared; a null with a wide interval is inconclusive, not negative.
11. Shadow deployment and canary rollout are not A/B tests. They establish operational safety and limit blast radius; neither produces user-outcome evidence.
12. Offline filters, online decides. Offline evaluation ranks many variants cheaply; only randomised live traffic tells you a proxy gain became a user gain.
13. On objective ids, remember the documented source defect — printed Experimentation objectives 3.1–3.5 duplicate Data Analysis 2.1–2.5 — while A/B testing is named explicitly in both the section's reading list and the certification's job-role description.

An A/B test is an event. You design it, run it for two weeks, read it once, and ship. Then the next change arrives — a new prompt version, a re-embedded index, a provider model update you did not ask for — and you cannot afford a two-week randomised experiment for each of them. What you need is the continuous, automated, cheap counterpart: an evaluation suite that runs on every commit, compares against a stored baseline, and fails the build when quality drops. That is where the measurement thread of this course terminates in an engineering artefact, and it is the single most transferable thing you will build.

Next: 10-04 turns everything in the module into a build-server gate — how to structure a regression suite, how to choose thresholds that catch real regressions without failing on non-determinism, how to version an eval set alongside code, and what to do when a provider silently changes the model underneath you.