Regression Testing an LLM System in CI/CD: The Eval Gate That Fails the Build

A regression test asks one question: is this version worse than the last one? Not "is it good" — that is what evaluation and A/B testing are for. Regression testing is differential. It compares the system as it is now against a recorded baseline, on a fixed set of items, and it exists to catch damage rather than to certify quality.

In CI/CD terms, it is a build stage with five parts:

The last row is the whole point, and it is where most teams stop short. A suite that produces a report nobody reads is not a gate. A gate is a stage whose failure blocks the merge. If quality regressions cannot block a merge, the eval suite is documentation, and documentation does not prevent regressions.

Three properties distinguish an LLM regression suite from an ordinary unit-test suite, and they are the source of every genuinely hard problem in this lesson:

• The system under test is non-deterministic. The same input can give different output across runs, so a strict output-equality assertion fails randomly. This is not a bug you can fix, for the reasons in 09-11, so the suite must be designed around it.
• Correctness is graded, not binary. "Right answer, slightly different wording" is a pass. A unit test has no vocabulary for that; a regression suite must, which is why it aggregates scores rather than counting assertion failures.
• Dependencies change without a commit. A provider updates a model behind a stable name; an index gets re-embedded; a document in the corpus is edited. Your code did not change and your system's behaviour did. Ordinary CI assumes the repo is the only variable, and here it is not — which is why a scheduled run against unchanged code is a required part of the design rather than a nicety.

L1 — Intuition: the ratchet

Think of the baseline as a ratchet. Once your system reaches 82% on the eval set, that number is recorded, and no change is allowed to take it meaningfully below 82%. Improvements move the ratchet up; regressions cannot move it down without a human explicitly deciding to.

That mental model gets three things right that a "just run the evals" framing misses. First, the point of comparison is your own last good state, not an absolute standard — which is what makes the gate useful at any quality level, including a system that is currently mediocre. Second, the ratchet must be explicitly re-set when quality legitimately changes, and that reset is a reviewed action with a name attached: a baseline update is a code change, not a build artifact silently overwritten. Third, the ratchet needs slack, because a non-deterministic system will produce 81.4% on a good day and 82.6% on another with no change at all. Slack sized wrongly is the most common way these suites fail: too tight and the build fails randomly until someone disables it, too loose and real regressions slip through.

L2 — Mechanism: building the gate, stage by stage

Stage 1 — Pin everything that can move. Before you can attribute a score change to a code change, freeze the alternatives: model name and version string (never a floating alias like "latest" in CI), temperature and all decoding parameters, the embedding model version, the index snapshot or its content hash, the prompt template version, the retrieval top-k, and the library versions. Record every one of these in the run's metadata. When the score moves, this record is what tells you whether the cause was in the diff.

Stage 2 — Choose the tiers of the suite. One suite that runs everywhere is a design error: it will be either too slow for a pull request or too shallow for a release. Three tiers is the standard shape.

Splitting by tier is what makes the gate survive contact with a real team. The pull-request tier has to be fast and cheap or developers will route around it; the nightly tier is where the expensive, slow and flakier signals live, and where a scheduled run with no code change catches provider-side drift.

Stage 3 — Pick assertions that survive non-determinism. This is the core engineering skill of the lesson. Order your checks from most to least brittle and prefer the strongest one that is genuinely stable for your task:

The rule of thumb: push as much of the gate as possible onto deterministic checks, and reserve fuzzy metrics for the tier where flakiness is affordable. A surprisingly large share of what looks like it needs a judge is actually a schema check plus a required-key check, and those never flake.

Stage 4 — Measure your own noise floor before you set a threshold. This is the step nobody does and everybody needs. Run the unchanged suite N times — five is a workable minimum, ten is better — and record the aggregate score each time. The spread across those runs is your noise floor. A threshold tighter than the noise floor produces random failures; a threshold much wider than it lets real regressions through. §4 does this arithmetic explicitly, and it is the most valuable arithmetic in the lesson.

Stage 5 — Define the gate as a compound rule, not a single number. A one-line "aggregate must not drop more than 3 points" gate misses the failure that matters most: an aggregate can hold steady while a specific critical capability breaks. A robust gate has several clauses:

FAIL the build if ANY of:
  1. aggregate_score < baseline_score - tolerance          # overall regression
  2. any item tagged "critical" regressed pass -> fail     # named must-never-break behaviours
  3. any deterministic assertion failed                    # schema, forbidden strings
  4. safety_violation_count > baseline                     # never allow a safety regression
  5. p95_latency > latency_ceiling                         # performance guardrail
  6. cost_per_item > cost_ceiling                          # cost guardrail
WARN (do not fail) if:
  7. aggregate_score improved by more than +tolerance      # suspicious: check for a leak or a metric bug

Clause 2 is the one that earns its keep. A handful of items marked critical — the regulatory disclaimer that must always appear, the refusal that must always fire, the calculation that must always be right — get item-level gates rather than being averaged away. Clause 7 is the underrated one: a large unexplained improvement is usually a bug in the metric, a leak of expected answers into the prompt, or an eval-set edit, and treating it as suspicious rather than as good news catches a whole class of self-deception.

Stage 6 — Report a diff, not a score. The CI output a developer sees should be the item-level delta: which items flipped from pass to fail, which flipped the other way, with inputs and both outputs shown. A build that fails saying "score 0.78 vs baseline 0.82" tells a developer nothing actionable. A build that fails saying "these four items regressed, here is the previous output and the new one" is a debugging session already half done.

Stage 7 — Make baseline updates an explicit, reviewed change. When a change legitimately alters behaviour, the baseline file is updated in the same pull request, with the new numbers visible in the diff and a note explaining why. Never let CI auto-commit a new baseline: an auto-updating baseline is a ratchet with the teeth filed off, and it will silently follow your quality downhill.

L3 — Depth: the four problems that break these suites in practice

Problem 1: flakiness, and the choice between suppressing and measuring variance. You have four levers and they trade off differently.

• Reduce the variance at the source. Temperature 0, fixed seed where the provider supports it, pinned model version. Cheapest and always worth doing — but it does not reach zero, because batching and hardware non-determinism remain (09-11).
• Average it away. Run each item k times and score the mean. This cuts the standard error of your aggregate by √k — genuinely effective, and it multiplies your CI cost and runtime by k. At k=3 you pay triple for a 42% reduction in noise.
• Loosen the assertion. Semantic similarity with a threshold instead of exact match; "contains the required key" instead of "matches the reference". Free, and it lowers sensitivity to small real regressions too.
• Widen the tolerance. Set the gate outside the measured noise band. Free, and it is the honest option provided the band was actually measured rather than guessed.

The professional posture is to do all of the first, measure the noise that remains, and then choose between the last three with the number in hand. The failure mode is choosing the tolerance by feel, watching the build fail twice on nothing, and marking the stage continue-on-error — at which point you have a suite and no gate.

Problem 2: the eval set decays. A frozen eval set is essential for comparability and it silently loses value over time. Three decay mechanisms:

• You fix everything it catches. After a few months the suite is all passes, and a suite that never fails has stopped providing information. The fix is to keep feeding it: every production incident and every failure found in error analysis (09-13) becomes a new item. A regression suite should grow monotonically, and the items added after an incident are its highest-value contents because they encode a failure you have actually experienced.
• Your traffic changes. The item distribution drifts away from reality, so the suite measures a task you no longer have. Periodically re-sample from recent production logs and add — do not replace — items. Replacement destroys comparability with the baseline; addition needs a baseline re-set, which is fine because it is explicit.
• Overfitting to the suite. Every prompt tweak selected because it improved the suite is a tiny act of fitting to the test set, and after a hundred such tweaks the score overstates real quality. This is the same statistical damage as benchmark contamination in 10-01, arriving through your own hands rather than a crawler. The mitigation is a holdout eval set that CI never sees, run manually at release time. If the CI score and the holdout score diverge over months, you have measured your own overfitting — and that measurement is worth the cost of maintaining a second set.

Problem 3: cost and runtime. A 300-item suite that calls a model three times per item is 900 calls per pull request. On a busy repository that is a real budget line and a real wait. The levers, in the order they usually pay off:

• Cache aggressively. Key a response cache on (model version, prompt hash, decoding params). Most pull requests change one prompt; every item whose full input is unchanged can be served from cache. This is often an order-of-magnitude saving and it is the single highest-leverage optimisation available.
• Tier the suite, as in stage 2, so the expensive tier runs nightly rather than per-commit.
• Select tests by what changed. A retrieval-config change need not re-run generation-only items; a prompt change need not re-run the retrieval-recall suite. This requires tagging items by which stage they exercise, which is worth doing anyway for diagnosis.
• Use a cheaper model for the fast tier — with a serious caveat. If the fast tier runs against a different model than production, its baseline is a different system's baseline, and it will miss regressions specific to the production model. Acceptable for smoke checks (schema, forbidden strings); not acceptable for the quality gate.
• Run in parallel. These calls are embarrassingly parallel and usually network-bound; concurrency turns hours into minutes for the cost of respecting rate limits.

Problem 4: dependency drift with no commit. This is the LLM-specific problem that ordinary CI has no concept of. Your provider updates the model behind a stable name. Your embedding provider changes a model, so old and new vectors are no longer comparable and the index needs re-embedding (12-12). A document in your corpus is edited. A library's default changes. In every case, the repository is unchanged and the system's behaviour is different.

The defences are structural rather than clever. Pin explicit versions everywhere and never use floating aliases in CI, so a change is at least an event you chose. Run the suite on a schedule against unchanged code — nightly, at minimum — so drift shows up as a failing scheduled build with an empty diff, which is an unambiguous signal that the change came from outside. Record the full environment fingerprint with every run: model version, embedding model version, index hash, library versions. And keep the provider-drift runbook short: when a scheduled run fails with no code change, compare fingerprints first, re-run to rule out noise second, and only then start reading outputs.

The most examinable table in this lesson. Each column is a distractor for every other column.

Three contrasts carry most of the exam value:

Regression testing versus A/B testing. A regression suite is retrospective and automated — it protects what you already have, on every change, forever, with no users involved. An A/B test is prospective and manual — it establishes that a specific new thing helps, once, using real users. They answer different questions and neither substitutes for the other. A team with only A/B testing regresses quietly between experiments; a team with only regression testing never learns whether any change helped.

Regression testing versus monitoring. Both watch for degradation, and the difference is when and against what. The regression suite runs before the deploy, on fixed items, against a baseline, and can block. Monitoring runs after the deploy, on live traffic, against a trend, and can only alert. A regression suite catches what you thought to include; monitoring catches what you did not. 12-14 is the monitoring side of the pair, and the two are complements — which is exactly why this module precedes it in the course.

Regression testing versus unit testing. Unit tests assert exact behaviour of deterministic code and must never be tolerant. Regression tests assert no meaningful degradation of a stochastic system and must be tolerant, or they flake. The trap is applying unit-test instincts to LLM output: an assertEqual(response, expected_text) on generated prose is a test that fails randomly and teaches your team to ignore red builds. Your LLM system still needs ordinary unit tests — for chunking, parsing, token counting, retrieval plumbing, prompt assembly — and those should be strict. Keep the two categories separate in the codebase, because they have opposite tolerance philosophies.

Constructed scenario. Every number below is invented so the arithmetic can be checked. None is a measured result from any real system. The arithmetic is exact from the stated inputs.

You have a 200-item RAG eval set, scored pass/fail per item by a rule combining required-key presence and a faithfulness heuristic. You want a CI gate. The naive instinct is "fail if the score drops at all." Here is why that is wrong and what to do instead.

Step 1 — Measure the noise floor. Five runs, no code change, nothing touched.

Run 1: 166 / 200 = 0.830
Run 2: 163 / 200 = 0.815
Run 3: 169 / 200 = 0.845
Run 4: 164 / 200 = 0.820
Run 5: 167 / 200 = 0.835

mean = (0.830 + 0.815 + 0.845 + 0.820 + 0.835) / 5
     = 4.145 / 5
     = 0.829

deviations from the mean:
  +0.001, -0.014, +0.016, -0.009, +0.006
squared:
  0.000001, 0.000196, 0.000256, 0.000081, 0.000036   sum = 0.000570

sample variance = 0.000570 / (5 - 1) = 0.0001425
sample SD       = sqrt(0.0001425) = 0.01194   (about 1.2 percentage points)
observed range  = 0.845 - 0.815 = 0.030       (3.0 percentage points)

Read that before setting any threshold. With no change at all, this suite's score moves across a 3-point range and has a standard deviation of about 1.2 points. A gate at "no drop at all" would fail roughly half of all builds on pure noise. A gate at "no drop more than 1 point" would fail frequently. Neither survives a week of real use.

Step 2 — Set the tolerance from the measured SD. A defensible rule is three standard deviations below the baseline, which under an approximately normal spread makes a noise-only failure rare:

tolerance = 3 * 0.01194 = 0.0358  → round to 0.035 (3.5 percentage points)
baseline  = 0.829 (the mean of the five runs — not the best run)
gate      = fail if aggregate < 0.829 - 0.035 = 0.794

Two design decisions inside that arithmetic, both of which matter. The baseline is the mean of several runs, not the best run. Baselining on your luckiest run guarantees that half your subsequent builds start below baseline. And three SDs, not two: at two SDs you would expect a noise-only failure in roughly one build in twenty, which on a repository with a hundred builds a week is five false alarms a week and a team that stops trusting the gate.

Step 3 — Know what the tolerance costs you in sensitivity. A 3.5-point tolerance means a real regression smaller than 3.5 points passes silently. Is that acceptable? On 200 items, 3.5 points is 7 items. So the gate's honest specification is: "this gate catches regressions of about 7 items or more, and is blind to smaller ones." Write that sentence in the repository. A gate whose sensitivity is undocumented gets mistaken for a guarantee it never offered.

Step 4 — Buy sensitivity, if you need it, by reducing noise. Two options, with their costs.

Option A — average k runs per item. The standard error of a mean of k runs scales as 1/√k:

k = 1: SD = 0.0119  → 3-SD tolerance = 0.036  (7.2 items)
k = 3: SD = 0.0119 / sqrt(3) = 0.0069 → tolerance = 0.021  (4.2 items)
k = 5: SD = 0.0119 / sqrt(5) = 0.0053 → tolerance = 0.016  (3.2 items)

Triple the CI cost and runtime to roughly halve the tolerance. That is the actual trade, stated in the units the decision is made in.

Option B — grow the eval set. Sampling variance for a proportion scales as 1/√n:

n = 200: binomial SD = sqrt(0.829 * 0.171 / 200) = sqrt(0.14176/200) = sqrt(0.000709) = 0.0266
n = 500: sqrt(0.14176/500) = sqrt(0.000284) = 0.0168
n = 800: sqrt(0.14176/800) = sqrt(0.000177) = 0.0133

Note that the binomial SD at n=200 (0.0266) is larger than the observed run-to-run SD (0.0119). That is not a contradiction and the reason is instructive: the binomial figure describes variation if you drew 200 different items each run, while your suite scores the same 200 items every time, so most items are stably right or stably wrong and only the borderline ones flip. Fixing the items removes item-sampling variance and leaves only output non-determinism — which is precisely why a frozen eval set is more sensitive to real changes than a freshly sampled one, and one of the strongest arguments for freezing.

Step 5 — Add the clauses an aggregate cannot express. Suppose 12 of your 200 items are tagged critical: the required regulatory disclaimer, three refusals that must always fire, a currency calculation, and seven items derived from past production incidents. These get item-level gates:

FAIL if aggregate < 0.794
FAIL if any of the 12 critical items regresses pass -> fail   # zero tolerance
FAIL if any schema/parse assertion fails                       # deterministic, zero tolerance
FAIL if safety_violations > 0
FAIL if p95_latency > 4.0 s
FAIL if mean_cost_per_item > $0.012
WARN if aggregate > 0.864                                      # +3 SD: suspiciously good, investigate

A critical item that flips is a hard fail even though it is one item in two hundred and invisible in the aggregate — and that asymmetry is the design's most important feature, because the failures that actually hurt you are rarely distributed evenly.

Step 6 — What a run looks like when it fails. The output a developer should see:

EVAL GATE: FAILED

aggregate:  0.775  (baseline 0.829, tolerance -0.035, floor 0.794)   FAIL  -0.054
critical:   12/12 pass                                               ok
schema:     200/200 valid                                            ok
safety:     0 violations                                             ok
p95 latency: 2.9 s  (ceiling 4.0 s)                                  ok
cost/item:  $0.009 (ceiling $0.012)                                  ok

regressed items (11):
  #017 multi-hop policy question  — retrieved chunks 3/3 -> 1/3   [retrieval]
  #044 date arithmetic            — required key "2024-03-01" absent [generation]
  ... 9 more, all tagged [retrieval]

fingerprint diff vs baseline:
  chunk_size: 512 -> 1024        <-- changed in this PR
  model: gpt-x-2024-11-01        unchanged
  embedding: emb-v3              unchanged
  index_hash: a91f... -> 7c02... <-- consequence of chunk_size change

That report does the diagnosis for you: ten of eleven regressions are tagged retrieval, the fingerprint diff shows the chunk size changed, and the conclusion writes itself — the larger chunks reduced retrieval precision. The developer needs no investigation, only a decision. This is why the item-level diff and the environment fingerprint are not reporting niceties; they are the difference between a gate that helps and a gate that merely blocks.

Step 7 — The scheduled run that catches what no diff explains. The same suite runs nightly on main with no changes. One morning:

EVAL GATE: FAILED (scheduled run, no code change)

aggregate: 0.761 (baseline 0.829)   FAIL  -0.068
fingerprint diff vs baseline:
  model: provider-model-v1 -> provider-model-v1   (alias unchanged)
  model_build_id: 20241101 -> 20250115            <-- CHANGED, not by us

An empty code diff with a 6.8-point drop and a changed build id is a provider-side model update. Nothing you did, and you found out in a nightly build rather than from a customer three weeks later. This single capability is, in practice, the strongest argument for the whole apparatus, and it is the reason the scheduled unchanged-code run is not optional.

The organising principle: gate what is deterministic, what is critical, and what you have measured a tolerance for. Warn on what is noisy. Monitor what requires real users. And note the feedback loop that makes the whole system improve — monitoring finds a new failure, error analysis characterises it (09-13), it becomes a tagged item in the eval set, and from then on the gate prevents its recurrence. That loop is the mature form of LLM quality engineering, and it is why this lesson sits between A/B testing and production monitoring in the course.

This lesson serves more official objectives than anything else in the module, because it sits at the intersection of two blueprint sections. The Experimentation section (22%) supplies the experimental discipline: its scope statement covers "how to perform, evaluate, and interpret experiments, including AI model evaluation." The Software Development section (24%) supplies the engineering: its objectives explicitly cover testing and debugging LLM applications and monitoring and maintaining them. The official job-role description is even more direct — the associate "implements testing/debugging" and is responsible for "design, code, test, and debug applications", "system analysis against specifications", "integrating new AI language models into existing systems" and "assessing and resolving performance issues." A regression gate is the artifact that discharges nearly all of those responsibilities at once, and integrating a new model into an existing system without a regression suite is precisely the scenario the exam likes to describe.

The objective-numbering defect, and the honest citation. You will find a verified defect if you go looking for the Experimentation objective id that authorises this material. The objectives printed under Experimentation as 3.1–3.5 are a verbatim duplicate of the objectives printed under Data Analysis and Visualization as 2.1–2.5 — data mining, comparing models with statistical metrics, conducting data analysis under supervision, creating charts, and identifying relationships and trends. Read literally, the section explicitly scoped to experiments, model evaluation and RLHF would describe none of those things, and 22% of the exam would have no stated coverage of model evaluation at all. Because the section's own scope statement, its course objectives and its suggested-reading list all agree with each other and disagree with the printed objective text, the derived scope governs; the printed ids are traceability only.

So the honest citation for this lesson is: the Software Development objectives on testing, debugging, monitoring and maintenance apply directly and without qualification (they are printed under their own section and describe their own subject correctly), while the printed Experimentation ids 3.2 / 3.5 apply only in their generic reading — comparing systems on statistical performance metrics, and identifying factors that could affect results. Where a study resource tells you that printed objective 3.1 covers CI regression testing, it is quoting a duplicated line about data mining.

Question phrasings to expect:

Distractor families:

1. The exact-match distractor. An option proposes asserting exact equality of generated text in CI. Wrong for a stochastic system: it flakes, and flaky gates get disabled.
2. The manual-review distractor. An option proposes that an engineer reviews outputs before each release. It does not scale, it is not reproducible, and it is not a gate. Human review belongs in sampling and in eval-set construction, not in the build.
3. The monitoring-instead-of-testing distractor. An option proposes catching regressions through production monitoring alone. Monitoring is post-deploy and cannot block; the users found the bug first.
4. The public-benchmark-as-gate distractor. An option proposes gating on MMLU or GLUE. Ruled out by 10-01: wrong population, and a contaminated set has too little variance to detect your changes.
5. The auto-baseline distractor. An option has CI overwrite the baseline with each run's result. That is a ratchet with no teeth; quality drifts down invisibly.
6. The single-threshold distractor. An option gates only on an aggregate score. It cannot catch a critical single-item regression, a safety regression, or a cost blow-out.
7. The "just lower the threshold" distractor. An option resolves flakiness by loosening the gate until it stops failing, without measuring the noise floor. That is how a gate becomes decorative.

By changing what you assert, not by trying to make the model deterministic. You cannot get to determinism — temperature 0 reduces variance but does not eliminate it (09-11) — so the suite is designed around a noise floor you have measured.

The four-part answer:

Assert properties, not strings. Valid JSON matching the schema; the required entity or number present; no forbidden pattern; semantic similarity to a reference above a threshold; the retrieved chunk set containing the gold chunk. Each of these is stable under rewording while still failing when the answer becomes wrong.

Aggregate instead of asserting per item, for the overall gate. One item flipping is noise; the aggregate over 200 items moving 5 points is a signal. Statistics buy stability that individual assertions cannot.

Measure the noise, then set the tolerance above it. Run the unchanged suite repeatedly, compute the standard deviation, set the gate around 3 SD below the mean baseline. Then document the sensitivity you bought: "this gate detects regressions of about 7 items or more."

Keep a deterministic core with zero tolerance. Schema validation, forbidden strings, required keys on critical items and safety checks are genuinely deterministic. They can and should be strict. Push as much of the gate as possible into this category, because it never flakes and it never needs a tolerance argument.

The mindset shift the exam rewards: an LLM regression suite is statistical quality control, not assertion-based testing. You are running a process-control chart on your own system, and the questions are the ones a process engineer asks — what is my noise floor, what shift can I detect, and what do I do when the chart goes out of bounds.

Large enough that the aggregate is stable, small enough to run on the schedule you need — and structured so each tier answers a different question. Working guidance, with the reasoning attached rather than as numbers to memorise:

The size argument runs through sensitivity, not through a rule of thumb. On 200 items, one item is half a percentage point; a 3.5-point tolerance is 7 items. If the smallest regression you must catch is 2 items, 200 items cannot do it at any tolerance, and you need a larger set, more runs per item, or item-level gates on the specific behaviours that matter. Deciding which of those to buy is the actual design work — and item-level gates on tagged critical behaviours are usually the cheapest way to get zero-tolerance coverage of the things you truly cannot lose, without growing the whole suite.

Two composition rules matter as much as size. Stratify. Cover every input type, every difficulty band, every retrieval-dependent versus parametric-knowledge case, and every known failure mode. A 300-item set drawn entirely from easy inputs is less informative than a stratified 100-item set. Grow monotonically. Add items from incidents and error analysis; never quietly delete or replace, because comparability with the baseline is the asset you are protecting. When the set does change, re-baseline explicitly in the same reviewed commit that changed it.

Your system's behaviour changes with no commit in your repository — which is the LLM-specific failure mode ordinary CI has no concept of, and the reason a scheduled unchanged-code run belongs in the design.

Detection. Pin explicit model versions rather than floating aliases, so an update is an event you chose rather than one that happens to you. Record a full environment fingerprint with every run: model name and build identifier, embedding model version, index content hash, key library versions. Run the suite nightly on unchanged main. A failure there with an empty code diff and a changed fingerprint is unambiguous.

Triage, in order. Compare the fingerprint first — it usually names the cause immediately. Re-run to rule out a noise excursion, since a single failing run near the tolerance boundary may be nothing. Then read the item-level diff: which items regressed, and do they cluster by stage or by input type? Provider updates often produce a characteristic pattern — for example, format compliance degrading across the board while factual items hold, which points at prompt-format sensitivity rather than lost capability.

Response. If a pinned older version is still available, staying on it buys time to adapt. If the update is forced, the eval suite becomes your migration harness: you now have a measured, item-level description of exactly what changed, which is the input to fixing the prompt, adjusting retrieval, or re-tuning parameters. Once the system is back above the floor, re-baseline explicitly with a commit that says why. If the new version is genuinely better, the baseline moves up and the ratchet is tighter than before.

The thing worth internalising: your eval suite is the instrument that turns a silent vendor change into a dated, quantified, actionable event. Teams without one discover model updates from customer complaints, and by then they have neither a measurement nor a before-state to compare against.

Yes, with three constraints, and the honest answer is that a judge is a nightly-tier instrument rather than a pull-request gate.

Constraint one: the judge is non-deterministic too. You have added a second stochastic system, so the noise floor of a judge-scored metric is wider than that of a deterministic one — measure it separately rather than reusing the tolerance you computed for automatic metrics. That wider band is usually too wide to block a pull request usefully.

Constraint two: the judge version must be pinned, and it will drift. A judge model update changes your scores with no change to your system, and now you have two dependencies drifting instead of one. Pin the judge, include it in the fingerprint, and when it must be updated, re-score the baseline with the new judge before comparing anything.

Constraint three: judges carry known biases — position bias, verbosity preference, self-preference, and sensitivity to rubric wording, all of which 09-10 covers. A judge-scored gate inherits every one of them, which means it can be systematically wrong in a stable direction rather than merely noisy. Stable wrongness is worse for a gate than noise, because it does not average out.

The practical arrangement that works: deterministic checks and cheap automatic metrics gate the pull request; judge-scored rubrics run nightly and on release candidates, with a pinned judge, a fixed rubric, and a tolerance measured for that specific configuration. Treat a judge regression as a strong warning that requires a human look rather than as an automatic block — and keep a small human-reviewed sample as the periodic audit of whether the judge still agrees with people, because a judge that has silently diverged from human preference is measuring nothing you care about.

1. A regression suite asks "is this worse than last time?" — differential against a stored baseline, not absolute quality. That is what makes it useful at any quality level.
2. A gate must be able to fail the build. A report nobody can be blocked by is documentation, and documentation does not stop regressions.
3. Measure your noise floor before setting a threshold. Run the unchanged suite 5–10 times, compute the standard deviation, set the tolerance at roughly 3 SD below a baseline that is the mean of several runs, not the best one.
4. Document the sensitivity you bought. A 3.5-point tolerance on 200 items means regressions smaller than about 7 items pass silently. An undocumented gate gets mistaken for a guarantee.
5. Buy sensitivity deliberately: k runs per item cuts noise by 1/√k at k× the cost; a larger set cuts sampling variance by 1/√n. Choose with the arithmetic in hand.
6. Assert properties, not strings. Schema validity, required keys, forbidden patterns, numeric tolerance, retrieval recall and semantic similarity survive rewording; exact match on generated prose flakes.
7. Gate on a compound rule. Aggregate tolerance, zero-tolerance critical items, deterministic assertions, safety count, latency ceiling, cost ceiling — plus a warning on unexplained improvement, which is usually a leak or a metric bug.
8. Tier the suite — smoke per commit, core per pull request, full nightly — and cache on (model version, prompt hash, decoding params). A suite too slow for a pull request gets skipped.
9. Pin every version and record a fingerprint. Floating model aliases in CI turn provider updates into unattributable mysteries.
10. Run the suite on a schedule against unchanged code. A failure with an empty diff and a changed fingerprint is dependency drift, and catching it nightly instead of via a customer complaint is the apparatus paying for itself.
11. Never let CI auto-update the baseline. A self-resetting ratchet follows quality downhill in silence; baseline changes are reviewed commits with stated reasons.
12. Grow the eval set monotonically from incidents and error analysis, add rather than replace, and keep a holdout set CI never sees so you can measure your own overfitting.
13. Judge-scored metrics belong on the nightly tier, with a pinned judge, a fixed rubric and their own measured tolerance — a judge adds a second stochastic, drifting, biased dependency.
14. Regression testing, A/B testing and monitoring are complements: the gate protects what you have on every change, the A/B test proves a change helped real users, and monitoring catches what neither anticipated — then feeds it back into the suite.
15. On objective ids: the Software Development objectives on testing, debugging, monitoring and maintenance apply directly here, while the printed Experimentation ids carry the module's documented duplication defect (3.1–3.5 repeat Data Analysis 2.1–2.5) and should be cited for traceability only.

You now have the complete measurement apparatus. You can build an eval set, choose metrics, run cheap capability probes, design and read a live experiment, and enforce the whole thing automatically on every change. Everything so far has treated the model's weights as fixed — you changed prompts, retrieval, context and configuration, and measured the result. That constraint is about to come off, and the reason the measurement thread came first is that a weight change is the most expensive and least reversible thing you can do to an LLM system, and it is the one you least want to attempt without a baseline you trust and a gate that will tell you when you have made things worse.

Next: 11-01 opens the adaptation module by separating three things routinely confused — pretraining from scratch, continued pretraining on domain text, and instruction tuning for behaviour — so that when someone says "we should train the model," you can identify which of the three they mean, what each costs, and what each can and cannot change.