Drift in Production LLMs: Data Drift vs. Concept Drift, and Why Launch Evaluation Is Not a Standing Guarantee

[GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) defines drift precisely: "a gradual decline in output quality as real-world inputs shift away from the training/validation distribution over time." Three clauses in that definition each carry real weight and are each worth pulling apart individually, because a scenario question tests exactly these clauses.

"Gradual." Drift is not a single event with a single timestamp the way a bad deploy or a dependency outage is. It accumulates — a little more mismatch between training-time inputs and today's inputs each week, each month — which means there is rarely a clean "before" and "after" the way M9-02's error-surge worked example had. This is the first reason drift resists the anomaly-detection machinery M9-02 builds: an anomaly detector is tuned to catch a sudden move outside a noise band, and a slope that creeps downward slowly enough can stay inside any reasonably-sized noise band for a long time before crossing it.

"Output quality." Drift's damage shows up specifically in whether the model's answers are still correct or useful — not in latency, not in error rate, not in uptime. This is the same distinction M9-02 draws for output-quality drops generally: none of M9-01's four operational metrics are designed to see this kind of failure, because a model serving a wrong-but-fluent answer to every request looks, from the outside, identical to a model serving correct answers. The request completed. It returned a well-formed response. It did not error out. Every operational signal reports a healthy system, while the actual content of what that system is producing has quietly stopped being trustworthy.

"Real-world inputs shift away from the training/validation distribution." This is the causal mechanism, and it is the clause that makes drift genuinely different from a bug: nothing about the model changed, and nothing about the serving infrastructure changed. What changed is the population of inputs arriving at inference time, drifting away from the population the model was trained and validated on. A model is, in a real sense, a function fit to a snapshot of the world taken at training time — and the further inference-time reality moves from that snapshot, the less that fitted function's assumptions hold, even though the function itself is bit-for-bit identical to the one that scored well at launch.

Why this is a distinct failure mode from every anomaly type M9-02 names

M9-02's latency spikes and error surges both have a mechanism that produces a detectable, often sudden signal in an operational metric: a batch-size change slows requests, a bad deploy breaks them outright. Drift has no equivalent mechanism producing a sudden signal in any operational metric, because drift's damage lives entirely in whether the content of an answer is right, and none of the four reliability metrics measure content at all. This is precisely why [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) treats drift as its own subsection rather than folding it into anomaly detection, and why detecting it needs an instrument this lesson introduces below rather than anything M9-01 or M9-02 already provides.

L1 — Intuition: the questions changed, or the right answers changed

Picture a customer-support model trained to answer questions about a product catalog as it existed a year ago. Two very different things can go wrong with it now, and they feel similar from the outside — "the model seems worse" — but the actual mechanism is opposite in each case.

In the first case, users are now asking about products that did not exist when the model was trained — new SKUs, a new pricing tier, a discontinued line nobody asks about anymore. The questions changed. Nothing about how a correct answer should be derived from a question changed; the model simply never saw training examples that look like today's questions. This is data drift: the input distribution moved.

In the second case, users are asking the exact same kinds of questions they always asked — "what is your return policy" — but the correct answer itself changed, because the company's return policy changed six months ago and the model was never retrained to reflect it. The inputs look identical to what the model saw in training. What changed is the relationship between that input and what a correct output now looks like. This is concept drift: the input-output relationship moved, while the inputs themselves may not have moved at all.

L2 — The mechanism, stated precisely

[GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) names both directly: "Data drift — the input distribution changes. Concept drift — the relationship between inputs and desired outputs changes." Stated with the notation this distinction is usually given: data drift is a change in P(X), the distribution of inputs themselves; concept drift is a change in P(Y|X), the conditional relationship between a given input and the output that input should now produce. The two are logically independent — either can occur without the other, and both can occur at once — and the source material's own framing states plainly that "both degrade live performance silently; monitoring for drift is a core reliability concern" [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md), treating them as a matched pair rather than ranking one as more important.

The independence is worth sitting with concretely. A model can face severe data drift with zero concept drift: users start asking questions in a language the model was never trained on, but for any question the model can parse, the correct-answer relationship is unchanged — the model just increasingly encounters inputs from a region of input space it has no training coverage of. A model can face severe concept drift with zero data drift: the exact same kinds of questions arrive at the exact same rate and phrasing as always, but the world's correct answers to those questions changed underneath the model, the way a return-policy or a regulatory change would. Recognizing which one is happening — or whether both are — determines what evidence would actually confirm it, which is the subject of the next section.

L3 — Why concept drift is fundamentally harder to detect than data drift

This is the exam-relevant edge case, and it is the sharpest technical point in this lesson. Data drift is detectable by watching the inputs alone — you do not need to know whether any given answer was right or wrong to notice that the distribution of incoming questions has shifted; you can measure that shift directly by comparing today's input statistics against the training-time input statistics, with no labels or ground-truth answers required at all. Concept drift has no equivalent shortcut: because it is defined as a change in the correct input-output relationship, detecting it requires knowing what the correct output for a given input actually is right now, and that is precisely the information a live production system, running without fresh ground truth, does not have on tap. You cannot notice that "the right answer to this exact question changed" by staring at the question alone — the question looks completely unremarkable. You need either a person to check the answer, an updated source of truth to compare against, or a regression-style evaluation designed specifically to surface this, none of which is available for free the way input-distribution monitoring is.

This asymmetry is the deepest reason drift, as a category, resists automatic detection the way a latency spike does not: input statistics can be logged and compared cheaply and continuously, essentially for free, off data you already have. The correct answer to a live question is not something a serving system has sitting in a log anywhere — it has to be independently established, which is exactly why later lessons in this module build a fixed evaluation suite and a human/judge review process as the actual detection mechanism for the harder half of this problem, rather than expecting an automated dashboard metric to surface it the way M9-01's latency percentiles surface a latency spike.

Read this table the way a scenario stem presents it: the described symptom almost always tells you which column applies, because the two columns diverge sharply on the "detectable without fresh labels" row — a scenario that describes catching a drift problem purely by watching input statistics is describing data drift; a scenario that requires someone to notice an answer is now wrong is describing concept drift, even if the questions being asked look completely unchanged.

Take a constructed scenario, illustrative rather than measured from a real deployment: an internal IT-helpdesk assistant, trained and evaluated eight months ago, is reported by users as "getting worse," and the team investigating has access to the last eight months of logged queries plus the original training and evaluation data.

Step 1 — check the operational metrics first (per M9-01/M9-02).
  Latency p99: flat at 900ms for 8 months.
  Error rate: flat at 0.3% for 8 months.
  -> Rules out a latency spike or error surge; whatever is happening is not
     visible to the four reliability metrics.

Step 2 — check the input distribution against training-time statistics.
  Training-time query topics (8 months ago): 40% password reset, 35% software install
    requests, 15% hardware ticket status, 10% other.
  Live query topics (last 30 days): 15% password reset, 20% software install requests,
    10% hardware ticket status, 55% "VPN access to the new remote-work portal."
  -> A topic that did not exist in training ("the new remote-work portal") now accounts
     for over half of live traffic. This is data drift: the input distribution moved
     sharply, and the model has essentially no training coverage of this new topic.

Step 3 — check whether the model's answers on the STABLE topics are still correct.
  Sample 40 live "password reset" queries (a topic present in training) and have a
  human reviewer check each answer against current internal documentation.
  Result: 38/40 correct (95%), consistent with the original evaluation score on this
  topic category eight months ago.
  -> No evidence of concept drift on this stable topic: the input-output relationship
     for password-reset questions has not changed.

Step 4 — check whether a stable-looking topic has a changed correct answer.
  Sample 40 live "software install requests" queries and check answers against current policy.
  Result: 24/40 correct (60%), down from 90%+ at launch evaluation.
  Root cause on inspection: the company changed its approved-software policy five months ago
  (a new self-service portal replaced the old ticket-based approval process), but the model
  still answers using the old ticket-based process it was trained on.
  -> This is concept drift specifically: the question "how do I get new software approved"
     looks identical to what it always looked like, and the model still gives its
     original, once-correct answer -- but the correct answer changed underneath it.

Step 5 — synthesize.
  The helpdesk assistant has BOTH data drift (a large new topic, remote-work VPN access,
  the model has no coverage of) AND concept drift (an old, stable topic, software
  install requests, whose correct answer changed via a policy update) happening
  simultaneously and independently, exactly as section 2's independence claim predicts.

The diagnostic payoff is the two different fixes this produces. The data-drift half (the VPN topic) needs new training or retrieval-corpus content covering a topic that genuinely did not exist before — there is no "old correct answer" to update, because the model never had one. The concept-drift half (the software-install topic) needs the model's existing knowledge about an existing topic corrected to match a changed policy — retraining on updated ground truth, or, if the system is RAG-based, updating the retrieved source-of-truth document rather than anything about the model's weights. Treating both problems with the same fix — for instance, assuming "more training data on current topics" alone would resolve the software-install failure — would miss that the software-install topic already had plenty of coverage; what it needed was corrected coverage, not more of the same outdated coverage.

⭐ THE EARNED INSIGHT: A model that scored well on evaluation at launch is a true statement about that model's fit to the world as it existed on the day evaluation ran, and it is never a standing guarantee about any day after that, because the model is frozen the instant training stops while the world generating its live inputs is never frozen — data drift moves the questions being asked, concept drift moves the correct answers to questions that look unchanged, and because both are gradual and both are invisible to every operational metric this module has covered so far, the only thing that stands between a silently decaying model and a customer finding the failure first is a deliberate, ongoing act of re-checking output quality against current reality, which is precisely the boundary the source material draws when it calls evaluation a before-or-at-deployment activity and monitoring a live, ongoing one — collapsing that boundary, and assuming a launch-time pass certifies anything beyond launch, is this domain's central and most consequential trap.

This is the distinction the module's guiding question exists to test, stated as directly as the source material states it: [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md): "It is distinct from Evaluation (Domain 6): evaluation measures quality before/at deployment; monitoring watches live operational health and drift afterward." And the source material's own scope note for the whole domain names the exact skill at stake: [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md): "Professional-level questions test whether you can separate monitoring from evaluation, recognize drift, and describe why versioning and regression benchmarking matter for reliability."

The trap this table exists to prevent is treating a single evaluation pass — however rigorous — as if it answers monitoring's question too. It cannot, structurally: evaluation is scored against a fixed benchmark that was itself built from a snapshot of the world, and the entire mechanism of drift is that the live world keeps moving away from that snapshot after the snapshot was taken. [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) makes exactly this point directly: "A model that passed evaluation at launch can still drift in production — evaluation is not a one-time guarantee." A model can pass every pre-deployment evaluation check available and still be actively drifting the day after it ships, because passing evaluation is a statement about fit to a fixed historical benchmark, and drift is specifically about fit to a moving target that no fixed benchmark, run once, can track.

Production Monitoring and Reliability is Domain 9 of the NCP-GENL blueprint, weighted at 7% [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md), and drift sits at the center of the domain's own stated scope note, which names separating monitoring from evaluation and recognizing drift as the two professional-level skills this domain tests most directly [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md). The source material's own self-check items make the expected question shapes explicit: [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) phrases one self-check almost exactly as "a gradual decline in output quality as real-world inputs shift over time is called: A. Drift ✅," with distractors including overfitting during pretraining, a brevity penalty, and a pipeline bubble — each a real, nameable concept from a different domain, attached to the wrong problem.

Expect this material in a few recurring shapes:

1. Direct identification. "A gradual decline in output quality as real-world inputs shift away from the training distribution is called ___." The keyed answer is drift, against distractors naming real concepts (overfitting, a brevity penalty in translation metrics, a pipeline-parallelism stall) from entirely different domains — a technique that exists, attached to the wrong problem, in the domain's characteristic distractor style.
2. Data-vs-concept discrimination. A scenario describes either a changed input distribution or a changed correct-answer relationship, and asks which type of drift it is — testing whether you can tell "the questions changed" apart from "the right answers changed" even when the described symptom is phrased ambiguously.
3. The evaluation-monitoring boundary. A scenario describes a team that evaluated thoroughly before launch and stopped measuring afterward, asking what risk this leaves unaddressed — the correct diagnosis names drift specifically, not a generic "something could go wrong."
4. Detectability reasoning. A question tests whether you know which type of drift can be caught by input-statistics monitoring alone (data drift) versus which one needs fresh ground truth (concept drift) — the asymmetry section 2's L3 tier establishes.
5. Cross-domain tie-ins. A question may pair drift with Domain 6's evaluation vocabulary specifically to test the boundary this lesson's section 5 draws, since the exam's stated intent is testing whether you can separate the two domains rather than treating monitoring as "more evaluation."

What the distractors typically look like

True to this domain's house style, the standard traps are real, nameable concepts attached to the wrong problem: offering overfitting (a training-time phenomenon, visible before deployment, per the earlier data-quality material) as an explanation for a production-time quality decline that a launch-time evaluation would already have caught; offering data drift as the answer to a scenario that actually describes a changed correct-answer relationship (concept drift), or vice versa; and offering "the model needs retraining" as a generic fix without first identifying which of the two drift types is present, which risks retraining on the wrong signal — adding more current-topic coverage when the actual failure is an outdated correct-answer relationship on an already-well-covered topic.

Data drift is a change in the distribution of inputs a model receives — the questions being asked, or the data being processed, shift toward something the model saw little or none of during training, while the correct relationship between a given input and its right output stays the same. Concept drift is a change in that relationship itself — the same kinds of inputs keep arriving, unchanged, but what counts as a correct output for a given input has shifted, typically because some real-world fact, policy, or regulation the model's training data encoded has since changed. [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) names both directly as the input distribution changing versus the input-output relationship changing, and the two are independent: a system can experience either one alone or both simultaneously, and diagnosing which is present determines whether the fix is adding coverage for a genuinely new topic or correcting outdated knowledge about an existing one.

Because evaluation is a point-in-time measurement of a model's fit to a fixed benchmark built from a snapshot of the world, and the model's weights are frozen the instant training stops while the real world generating its live inputs never stops moving. [GROUND TRUTH] (Sources/ncp-genl/domain-9-production-monitoring.md) states this as the domain's own named trap: "A model that passed evaluation at launch can still drift in production — evaluation is not a one-time guarantee." Passing a launch-time benchmark says nothing about whether the input distribution or the correct-answer relationship will still match that benchmark six months later, and because drift is gradual and invisible to every operational reliability metric, the gap between "passed evaluation once" and "is still correct today" can grow for a long time before anyone notices it without a deliberate, ongoing quality-monitoring practice in place.

Because data drift can be detected by comparing input statistics alone — you can measure that the distribution of incoming questions has shifted without needing to know whether any specific answer was right or wrong, since the shift is visible directly in the inputs themselves and needs no fresh ground truth. Concept drift has no equivalent shortcut: it is defined as a change in what counts as a correct output for a given input, and confirming that requires knowing the currently-correct answer, which is exactly the information a live system running without fresh labels does not have sitting in a log anywhere. This asymmetry is why data drift can be monitored cheaply and continuously off existing traffic data, while concept drift needs deliberately-built detection — human or judge review, or a refreshed regression suite — that a pure input-statistics dashboard cannot substitute for.

• Drift is a gradual, silent decline in output quality, invisible to every operational metric this module has covered. Latency, throughput, error rate, and uptime can all stay perfectly healthy while a model's actual answers quietly get worse.
• Data drift moves the inputs; concept drift moves the correct answers. The two are logically independent, and diagnosing which one (or both) is present determines whether the fix is new-topic coverage or corrected knowledge about an existing topic.
• Data drift is detectable from input statistics alone; concept drift is not. Concept drift requires knowing the currently-correct answer, which needs fresh ground truth that a live system does not have automatically.
• A launch-time evaluation pass is never a standing guarantee. Evaluation measures fit to a fixed, historical benchmark; drift is specifically about fit to a world that keeps moving after that benchmark was built.
• This is the domain's own named central trap: conflating evaluation with monitoring. Evaluation answers "was this good enough to ship"; monitoring answers "is it still good enough," continuously, for as long as the model serves live traffic.
• Both drift types can occur simultaneously in the same deployed system, on different topics or segments. Diagnosing each independently, per segment, is what a correct fix depends on.

Next: M9-04 builds the actual continuous-comparison instrument that catches a regression — including the kind drift causes — before and after a rollout: offline eval-set regression benchmarking, which treats evaluation as an ongoing practice against a fixed suite rather than the one-time launch gate this lesson has spent its length arguing evaluation alone can never be.

Answers

1. A support bot's users start asking heavily about a product line that launched after the model's training cutoff, while the correct-answer logic for every question type the model was trained on remains unchanged. What is this?

• A. Data drift ✅
• B. Concept drift
• C. An error surge
• D. Overfitting

The input distribution shifted toward a genuinely new topic; nothing about the input-output relationship for existing topics changed.

2. A helpdesk assistant keeps receiving the exact same kinds of questions it always received, but a company policy change six months ago means the answers it was trained to give are now factually wrong. What is this?

• A. Data drift
• B. Concept drift ✅
• C. A latency spike
• D. Benchmark contamination

The inputs look unchanged; what changed is what counts as a correct output for those inputs — the defining signature of concept drift.

3. Which type of drift can be detected purely by comparing live input statistics against training-time input statistics, without needing any fresh ground-truth labels?

• A. Data drift ✅
• B. Concept drift
• C. Neither type is detectable without labels
• D. Both types require identical detection methods

Data drift is visible directly in the inputs themselves; concept drift requires knowing the currently-correct answer, which input statistics alone cannot reveal.

4. A model passes every pre-launch evaluation check. Six months later its answers have become systematically wrong on several topics, with no code or configuration change in between. What does this demonstrate?

• A. The evaluation process itself must have been flawed
• B. Passing evaluation at launch is not a standing guarantee against later drift ✅
• C. This can only be an error surge, not drift
• D. Evaluation and monitoring are the same activity and one implies the other

This is the domain's own named central trap: evaluation is a point-in-time measurement; drift can degrade a model afterward with no code change at all.

5. Why do latency, throughput, error rate, and uptime all fail to detect drift?

• A. Because drift only affects models that have never been evaluated
• B. Because drift's damage is in output quality/content, which none of those four operational metrics measure ✅
• C. Because drift always causes a latency spike eventually
• D. Because drift is only detectable through GPU utilization

A model producing wrong-but-fluent, well-formed, on-time responses looks operationally identical to one producing correct responses — the four reliability metrics cannot see content quality at all.

6. A system shows both a large new, previously-unseen topic in its live traffic AND an old, stable topic whose correct answer has changed due to a policy update. What does this illustrate?

• A. This is impossible; a system can only experience one type of drift at a time
• B. Data drift and concept drift are independent and can occur simultaneously in the same system, on different segments ✅
• C. This is a single combined phenomenon requiring one fix
• D. This must be measurement error, since drift is always one type or the other

The two drift types are logically independent — either, or both, can be present at once, and each needs its own diagnosis and fix, as the worked example's helpdesk scenario shows.

7. What distinguishes drift from a latency spike or error surge, in terms of how it appears in monitoring data?

• A. Drift always causes a sudden move outside an established noise band, identical to a latency spike
• B. Drift is gradual and accumulates slowly, which can keep it inside an anomaly detector's noise band for a long time before crossing it ✅
• C. Drift only occurs immediately after a deployment
• D. Drift is detected using the same threshold-based alerting as an error surge

Anomaly detection (per M9-02) is tuned to catch sudden moves; drift's gradual accumulation is a structurally different signal shape that the same threshold-based alerting can miss for a long time.

8. How does Evaluation (Domain 6) differ from Monitoring (this domain) with respect to drift specifically?

• A. They are the same activity; evaluation performed thoroughly makes ongoing monitoring unnecessary
• B. Evaluation measures quality against a fixed benchmark before or at deployment; monitoring continuously checks whether that quality still holds as the world moves afterward ✅
• C. Monitoring is only needed if evaluation was skipped
• D. Evaluation is continuous and monitoring is a one-time gate

This is the domain's stated boundary: evaluation is a point-in-time check against a historical snapshot; monitoring is the ongoing practice that catches drift precisely because that snapshot stops matching reality over time.