Offline Eval-Set Regression: Benchmarking a Deployment Against Prior Versions

[GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) states the objective directly: "continuously benchmarking deployed agents against prior versions" and specifically to "compare the current deployment/agent to previous versions on a fixed suite to catch regressions before and after rollout." Unpack the load-bearing words in that sentence, because each one is doing real work.

"Fixed suite." The evaluation set used for this comparison does not change from one run to the next. It is a frozen collection of test inputs — prompts, questions, tasks — with either a known-correct expected output or a defined grading method (exact match, a rubric, an LLM-as-judge scorer) that produces a comparable score across runs. If the suite itself changed between two comparison runs, a score difference would be ambiguous: did the deployment get worse, or did the test just get harder? Fixing the suite removes that ambiguity entirely, which is the whole reason a "fixed suite" is specified rather than "some representative sample of current traffic."

"Current deployment ... to previous versions." The comparison is always relative, not absolute. A regression benchmark does not ask "is this deployment good" in isolation — that question belongs to the initial evaluation this course's Domain 6 covers. It asks "is this deployment as good as, better than, or worse than the version it is replacing, on the same test cases." This relative framing is what makes the word "regression" appropriate: a regression is a decline relative to a known-good prior state, and you cannot detect a decline without a fixed prior state to decline relative to.

"Before and after rollout." The practice is not a single checkpoint. Run before a new version ships, the suite acts as a gate — a version that scores meaningfully worse than its predecessor should not roll out at all, the same way a build that fails its test suite should not merge. Run after a version has shipped, the same suite acts as a confirming check — did the rollout actually behave the way the pre-rollout gate predicted, once real infrastructure and load are involved. A pre-rollout-only regime misses regressions that only manifest under production conditions the gate never exercised; a post-rollout-only regime means bad versions reach users before anyone notices.

"Evaluation as continuous, not a one-time gate." This is the framing [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) states as the point of the whole practice, and it is the same idea M9-03 built toward from a different angle: a single evaluation pass cannot certify anything beyond the moment it ran, because the world keeps moving. This lesson argues the complementary half: even holding the world fixed, every new version of the deployment is a new event deserving the same rigor the original launch evaluation received, run again against the same fixed yardstick, so "did this specific change help or hurt" has a real answer instead of an assumption.

L1 — Intuition: the same test, retaken, by a new version of the same student

Picture a fixed, unchanging exam — the same twenty questions, the same grading rubric, every time. A student takes it once, scores 85%, and that score is recorded. Months later, after some combination of changes nobody fully tracked, the same student retakes the exact same exam. A new score of 91% suggests the changes plausibly helped; a new score of 68% suggests they plausibly hurt — and the fixed exam is what makes either comparison meaningful rather than a guess. Had the exam questions changed between sittings, the score difference would say nothing reliable about whether the student actually improved. An LLM deployment's fixed eval suite plays exactly the exam's role: it is the one thing that must not change, so that everything that does change (a new fine-tune, a new prompt, a new retrieval index) can be judged against a stable yardstick.

L2 — The mechanism: a comparison pipeline, not a single score

The mechanism has four moving parts, and losing track of any one of them breaks the comparison.

1. A frozen suite of test cases, each with a defined way to grade the output — exact match for factual lookups, a rubric or LLM-as-judge score for open-ended generation, a pass/fail check for a structured task. The suite is versioned itself (suite v1, v2, ...) so that if it ever must change, old comparisons are not silently mixed with new ones under the same label.
2. A run against the candidate version — the deployment being considered for rollout, or the deployment that has just rolled out — producing one score (or one score per category, for a segmented suite) on the frozen suite.
3. A run's score against the recorded baseline — the immediately prior version's score on the same frozen suite, stored from when that version was itself evaluated.
4. A comparison and a decision rule — is the candidate's score meaningfully below the baseline (a regression, blocking or flagging the rollout), meaningfully above it (an improvement, worth noting), or within the established noise band (no meaningful change, per the same noise-band discipline M9-01 and M9-02 established for operational metrics).

The comparison step is where most of the actual judgment lives, and it borrows directly from the noise-band reasoning this module has used throughout: two runs of the same version against the same suite will not produce bit-identical scores if any part of the pipeline has nondeterminism (sampling temperature, a non-zero-temperature judge model, minor infrastructure variance), so a small difference between candidate and baseline is not automatically a regression — it has to exceed whatever variance the same version shows against itself before it counts as a real signal.

L3 — The exam-relevant edge case: what a regression benchmark cannot see, and why the suite itself needs its own maintenance

A fixed suite is precisely as good as its coverage of what actually matters in production, and it is worth being honest about what this practice structurally cannot catch. A regression benchmark run on a frozen suite will not detect a quality problem confined entirely to inputs the suite never included — which is exactly the failure mode M9-03's data-drift discussion covers, a genuinely new topic the fixed suite has no test cases for at all. This is not a flaw specific to regression benchmarking; it is the same limitation any fixed-benchmark evaluation has, and it is why this lesson's practice and M9-03's drift-monitoring practice are complementary rather than redundant: regression benchmarking catches "did a specific version change break something the suite already tests for," while drift monitoring catches "has reality moved somewhere the suite never tested in the first place." A team relying on regression benchmarking alone, with a suite nobody has updated since launch, can pass every regression check indefinitely while a real quality problem grows entirely outside the suite's blind spot.

The second edge case worth naming precisely: a regression benchmark's fixed suite needs periodic, deliberate maintenance — adding cases as new capabilities or topics become relevant — without losing the property that makes old comparisons valid. The practical resolution is versioning the suite itself explicitly (as step 1 of the mechanism above does), so that a score from suite v2 is never silently compared against a score from suite v1 as if they were the same measurement, the same discipline that makes any of this comparison meaningful in the first place.

The two practices are not competitors; they target different sources of quality change — one deliberate and attributable to a specific version, one gradual and attributable to the world moving — and a mature setup runs both, since either alone leaves a real gap the other closes.

Take a constructed scenario, illustrative rather than measured from a real deployment: a team maintains a frozen, 150-item evaluation suite for a document-summarization service, graded by a rubric-based LLM-as-judge scorer producing a 0-100 score per item, averaged into one suite-level score. Version 4 (v4) is the current production deployment; version 5 (v5), which swaps in a newly fine-tuned base model intended to reduce latency, is a candidate for rollout.

Step 1 — establish the noise band for v4 against itself.
  Re-running v4 against the frozen suite three times (same version, same suite, small
  amount of sampling nondeterminism) gives scores of 82.4, 83.1, 81.9.
  -> v4's own repeat-run variance is roughly +/-0.8 points; anything within that band
     is noise, not signal.

Step 2 — run the candidate (v5) against the identical frozen suite.
  v5 scores 79.6 on the same 150-item suite.

Step 3 — compare against the recorded baseline.
  v4's recorded baseline score: 82.4 (from step 1's first run, the one used at v4's own
  rollout gate).
  v5's score: 79.6.
  Difference: -2.8 points, well outside the +/-0.8 noise band established in step 1.
  -> This is a real regression, not noise: v5 scores measurably worse than v4 on the
     identical fixed suite.

Step 4 — segment the regression by category (the suite is tagged by document type).
  Category breakdown:
    News articles (60 items):     v4 = 85.1   v5 = 84.7   (flat, within noise band)
    Legal documents (45 items):   v4 = 79.8   v5 = 71.2   (-8.6 points, large regression)
    Technical manuals (45 items): v4 = 82.0   v5 = 81.4   (flat, within noise band)
  -> The regression is concentrated almost entirely in the legal-document category.

Step 5 — decision and root-cause follow-up (using M9-02's diagnostic chain).
  Gate decision: v5 is blocked from rollout because it regresses on a real, non-noise
  category rather than shipping and hoping the drop goes unnoticed.
  Root-cause investigation (per M9-02's confirm-segment-correlate-test steps) finds the
  newly fine-tuned base model's training data under-represented legal-document
  structure relative to the model v4 used, causing it to compress legal summaries too
  aggressively and drop material clauses the rubric penalizes.

The payoff of the fixed suite and the noise-band discipline together is visible at every step: without step 1's noise-band check, the -2.8 point difference in step 3 could have been dismissed as ordinary variance or, worse, treated as a confirmed regression when it was actually within normal range — either error is possible without first knowing what "normal" repeat-run variance looks like for the same version. Without step 4's segmentation, the team would have seen only a modest-looking -2.8 point aggregate drop and might have shipped v5 anyway, missing that one entire category of real production traffic (legal documents) was regressing by more than eight points while everything else stayed flat. The fixed suite made every one of those comparisons possible in the first place, because none of them work if the test cases themselves are different between runs.

A second worked example: why "before" alone is not enough

Continue the same constructed scenario: suppose v5's fine-tuning issue in the legal-document category had been small enough — a 1.5-point drop, inside the noise band the team had established — to pass the pre-rollout gate cleanly. v5 ships to production.

Post-rollout check, run one week after v5's rollout, same frozen 150-item suite:
  Legal documents (45 items):   v5 (week 1 post-rollout) = 74.3
  Compare to pre-rollout v5 score on the same category: 78.3 (within noise band of v4's 79.8)
  Difference: -4.0 points that appeared only after real production infrastructure,
  real concurrent load, and a real retrieval index (for a RAG-augmented summarizer)
  were all engaged -- none of which the pre-rollout gate run exercised at the same scale.

A pre-rollout gate run typically executes the fixed suite in a controlled, lower-load environment; a post-rollout check runs the identical suite against the live deployment under real conditions. The two can diverge, and the gap between them is exactly what makes "before and after" — not "before, only" — the correct practice [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) specifies. A regression that only manifests under production-scale concurrency, or against a retrieval index's real, current content rather than a test-time snapshot, is invisible to a pre-rollout-only regime and would ship undetected without the post-rollout re-run confirming the gate's result still holds.

⭐ THE EARNED INSIGHT: A fixed evaluation suite is only useful as a regression detector to the exact extent that every comparison it supports holds everything else constant except the one thing actually being tested — the same test cases, the same grading method, a known noise band established from repeat runs of the same version — because the moment any of those holds loosely (a suite that quietly changed between runs, a score difference smaller than the version's own repeat-run variance, a pre-rollout-only cadence that never re-checks under real production load) the comparison stops measuring "did this version regress" and starts measuring noise, test drift, or an untested deployment condition instead, which is precisely why "continuous, before and after rollout, against a fixed suite" is stated as one indivisible practice rather than three optional add-ons to a single launch-time evaluation.

This lesson's title and scope are deliberately narrow, and the narrowness matters because a closely related practice exists under a different name in a different cert's material, and conflating the two is a real risk. "Online task-success regression: benchmarking a live agent against prior versions" is the title of a lesson covering NVIDIA's NCP-AAI (AI agent) certification material, and it owns a genuinely distinct angle: continuously comparing a live agent's task-success rate on real, in-flight production traffic against prior versions, rather than this lesson's practice of comparing scores on a fixed, frozen, offline suite. The two practices share vocabulary — both are "regression benchmarking against prior versions" — and that shared vocabulary is exactly why the boundary needs to be stated explicitly rather than left implicit.

Both practices matter, and a mature deployment runs both rather than choosing one: the offline, fixed-suite version this lesson owns catches regressions on scenarios someone thought to test for in advance, cheaply and repeatably; the online, live-traffic version catches whether an agent is actually succeeding at real tasks nobody wrote a test case for, at the cost of a comparison that is noisier and harder to make perfectly apples-to-apples given the traffic itself keeps changing. Recognizing which one a scenario is describing is mostly a matter of one question: is the comparison running against a fixed, pre-built set of test cases (this lesson), or against whatever real traffic happened to arrive (the agent-domain practice)? That single question, applied as a reading strategy rather than a cited fact, stays labelled inference and resolves the boundary in almost every scenario either lesson is likely to describe.

The table below inventories the misreadings this lesson has been built to prevent, each paired with the correction and the reason it matters for a scenario question.

Production Monitoring and Reliability is Domain 9 of the NCP-GENL blueprint, weighted at 7% [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md), and this lesson's material is explicitly numbered as objective 9.3 in the source: [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md): "Objective 9.3 calls for continuously benchmarking deployed agents against prior versions." Expect this objective in a few recurring shapes:

1. Direct identification. "Comparing a newly deployed agent against the previous version to catch regressions is called ___." [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) states this nearly verbatim as one of its own self-check items, with the keyed answer being continuous/regression benchmarking against distractors naming unrelated techniques (tokenization, quantization, beam search) from entirely different domains.
2. Continuous-vs-one-time framing. A scenario describes a team that ran an eval suite once at launch and never again, asking what risk this leaves unaddressed — the correct diagnosis names the absence of ongoing regression benchmarking specifically, distinct from the absence of drift monitoring.
3. Before-vs-after rollout reasoning. A scenario describes a regression that passed a pre-rollout check but appeared post-rollout (or the reverse), testing whether you recognize why both checkpoints are specified rather than either one alone.
4. Regression vs. drift discrimination. A scenario describes either a specific version change causing a decline, or the passage of time causing a decline with no version change, asking which practice would have caught it — testing the section 3 boundary.
5. Offline-vs-online discrimination. A question naming this lesson's fixed-suite practice alongside a live-traffic agent regression practice, testing whether the fixed-suite, offline framing is recognized as this domain's specific angle rather than the live-traffic one.

What the distractors typically look like

True to this domain's house style, the standard traps are real, nameable concepts attached to the wrong problem: offering "collect more training data" or "retrain the model" as an unconditional response to any regression finding, without first confirming the difference exceeds the noise band or segmenting to find where it is concentrated; conflating this lesson's fixed-suite, offline practice with drift monitoring's live, ongoing practice, when the two are triggered by different things and catch different failure sources; and offering a single aggregate suite score as sufficient evidence a version is safe to ship, when a real regression can be concentrated in one category the aggregate score dilutes into invisibility.

A fixed, or frozen, suite is a set of evaluation test cases — prompts, questions, or tasks, each paired with a known-correct expected output or a defined grading method — that does not change between comparison runs. The entire value of the practice depends on this: if the test cases themselves changed between two runs, a score difference could not be attributed reliably to the deployment getting better or worse, because the test itself might simply have gotten easier or harder. [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) specifies comparing versions "on a fixed suite" precisely because a fixed suite is what makes the comparison meaningful; when a suite genuinely needs updating over time, it should be versioned explicitly so that scores from different suite versions are never silently compared as if they were the same measurement.

Because a pre-rollout gate, typically run in a controlled environment at limited scale, cannot exercise everything a live production deployment will actually encounter — real concurrent load, a real and currently-changing retrieval index, real infrastructure conditions. [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) specifies catching regressions "before and after rollout" for exactly this reason: a regression that only manifests under production-scale conditions is invisible to a pre-rollout-only check, and would ship into production undetected without a post-rollout re-run of the identical fixed suite confirming that what passed the gate is still passing once real infrastructure and real load are actually in the loop.

Offline eval-set regression is triggered by a version, deployment, or configuration change and compares the current deployment's score against a prior version's score, on an identical fixed suite — it answers "did this specific change make things worse." Drift detection, covered in M9-03, is triggered by the passage of time regardless of whether anything was deployed, and compares current reality against what the model was trained or evaluated on — it answers "has the world moved away from this static model." The two catch different sources of quality decline and neither substitutes for the other: a team running only regression benchmarks can pass every check while drift grows unnoticed in inputs the fixed suite never covered, and a team monitoring only for drift can miss a regression a specific bad deploy introduced on scenarios the suite already tests for.

• Offline regression benchmarking compares a deployment to its own prior versions on an identical, fixed, frozen suite. The comparison is relative to a known-good prior state, never absolute.
• The practice runs before rollout (as a gate) and after rollout (as a check), not just one or the other. A regression that only manifests under real production conditions is invisible to a pre-rollout-only regime, as this lesson's second worked example demonstrates.
• A score difference only counts as a regression once it exceeds the version's own established repeat-run noise band. Chasing every small fluctuation wastes effort; ignoring a real, sustained difference ships a genuine regression.
• Segmenting a suite's score by category exposes localized regressions an aggregate score would dilute. The first worked example's legal-document category shows a regression that the aggregate score alone would have understated.
• This practice and drift monitoring (M9-03) catch different failure sources and neither substitutes for the other. Regression benchmarking catches a specific version's change; drift monitoring catches the world moving under a static model.
• This lesson owns the offline, fixed-suite angle specifically — not online, live-traffic task-success regression. The two share vocabulary but differ in what counts as a "prior version" and what the test cases actually are, and a related lesson in NVIDIA's agent-focused certification material owns the live-traffic version of this idea.

Next: M9-05 closes the module by asking what happens once a regression, or a drift finding, actually confirms that a model needs to change — automated model retraining, tuning, and versioning, and why versioning is about safe, traceable rollback rather than merely giving a new file a new name.

Answers

1. A team runs an evaluation suite against a candidate deployment and finds its score is 0.6 points lower than the prior version's recorded score. The prior version's own repeat-run variance is +/-1.2 points. What should the team conclude?

• A. This is a confirmed regression and the candidate must be blocked
• B. The 0.6-point difference is within the established noise band and is not yet evidence of a real regression ✅
• C. The suite itself must have changed between runs
• D. This proves the candidate is definitively better than the prior version

A score difference smaller than the version's own established repeat-run variance has not exceeded the noise band and cannot be treated as a confirmed signal either way.

2. Why must the evaluation suite used for regression benchmarking be "fixed" or "frozen" between comparison runs?

• A. So the suite never needs to be updated again
• B. So a score difference between runs can be attributed to the deployment changing rather than the test itself changing ✅
• C. Because fixed suites are cheaper to grade than dynamic ones
• D. Because only fixed suites can use an LLM-as-judge scorer

If the test cases themselves changed between runs, a score difference could reflect the test getting harder or easier rather than the deployment actually regressing.

3. A candidate version passes its pre-rollout regression check cleanly but shows a real regression one week after shipping to production. What does this demonstrate?

• A. The pre-rollout gate is unnecessary and should be skipped in the future
• B. A regression can manifest only under real production conditions the pre-rollout gate never exercised, which is why a post-rollout check is also required ✅
• C. The fixed suite must have been corrupted
• D. This can only be explained by data drift, not a version regression

Pre-rollout gates typically run at limited scale in controlled conditions; real concurrent load or a live retrieval index can surface a regression only after rollout — exactly why both checkpoints are specified.

4. How does offline eval-set regression benchmarking differ from monitoring for drift?

• A. They are the same practice under two different names
• B. Regression benchmarking is triggered by a version/deployment change and compares against a known prior version; drift monitoring is triggered by time passing and compares against launch-time expectations ✅
• C. Drift monitoring only applies to latency, while regression benchmarking only applies to quality
• D. Regression benchmarking requires no fixed suite, unlike drift monitoring

The two practices catch different sources of quality decline: a specific attributable change versus the world gradually moving under a static model.

5. Which of the following best describes the distinct scope of "online task-success regression," as owned by NCP-AAI's agent material, relative to this lesson's practice?

• A. It is identical to this lesson's practice, just under a different name
• B. It compares a live agent's task success on real, in-flight production traffic against prior versions, rather than comparing scores on a fixed, frozen offline suite ✅
• C. It only applies to latency metrics, not task success
• D. It requires no comparison against prior versions at all

The two practices share the "regression against prior versions" framing but differ in what the test cases are — a fixed offline suite here, real live traffic in the agent-domain practice.

6. A regression benchmark's aggregate suite score drops by only 1.5 points, within the noise band, but one category within the suite drops by 8 points. What should the team do?

• A. Ignore the category-level breakdown since the aggregate score is within the noise band
• B. Segment the suite's score by category before certifying the version safe, since a real regression can be concentrated in one category the aggregate dilutes ✅
• C. Conclude the suite itself is broken
• D. Automatically pass the version since the aggregate is the only score that matters

An aggregate score can mask a real, severe regression that is concentrated in one segment — segmentation is what exposes it, exactly as the lesson's first worked example demonstrates.

7. A team's fixed evaluation suite was built at launch and has never had a new test case added since. Regression checks have passed cleanly for two years. What risk does this leave unaddressed?

• A. None — a passing regression check certifies the deployment against all possible quality problems
• B. A quality problem confined to inputs the suite never included, which regression benchmarking structurally cannot catch ✅
• C. This risk is fully covered by regression benchmarking alone and needs no other practice
• D. The suite's age has no bearing on what it can detect

A fixed suite can only catch regressions on scenarios it already tests for; a stale suite develops the same blind spot as any benchmark facing a moving world, which is why drift monitoring is a separate, necessary practice.

8. What is the correct response when a candidate version's regression-benchmark score is confirmed, after checking the noise band, to be meaningfully worse than the prior version's on a real production-relevant category?

• A. Ship the version anyway since the aggregate score still looks acceptable
• B. Block or roll back the version and investigate the root cause of the regression before it reaches broad production traffic ✅
• C. Update the suite to remove the category showing the regression
• D. Wait for the next scheduled evaluation cycle before taking any action

A confirmed, non-noise regression on a real category is exactly the finding regression benchmarking exists to catch before it affects users — the correct response is to block or roll back and diagnose, not to ship or quietly remove the evidence.