Structured Feedback and the Data Flywheel: How Captured Signal Drives Refinement

Identity statement: structured feedback is a captured signal about an agent's performance — a rating, a correction, an escalation — recorded in a defined, consistent form that a refinement process can systematically read and act on, as opposed to an ad hoc comment that exists only in someone's memory or an unlogged conversation. [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md): "Feedback (ratings, corrections, escalations) is captured in a structured form and fed back into refinement."

The word "structured" is doing real work in that identity statement, and it is worth separating from its opposite to see exactly what it rules out. A user who mutters "that wasn't very helpful" to a colleague, never captured anywhere the agent's team can see, is real feedback in the colloquial sense and is worthless to a refinement process, because nothing about it can be systematically read, counted, or acted on — it simply does not exist as far as the system is concerned. A user who taps a thumbs-down button, which logs a rating tied to the specific interaction that prompted it, has produced structured feedback: a defined signal, in a defined form, attached to a specific, retrievable record of what the agent actually said. The difference is not how strongly the user felt — it is whether the feedback was captured in a form the system can do anything with.

Three named forms of structured feedback

The source material names three specific forms, and distinguishing them matters because each one carries a different kind of signal and typically triggers a different kind of downstream action.

Each form is structured in the sense this lesson cares about — captured consistently, tied to a specific interaction, readable by a downstream process — while carrying genuinely different information. A rating tells you that something went wrong, in aggregate, across many interactions. A correction tells you what the right answer actually was for one specific case. An escalation tells you the agent hit the edge of what it could handle at all. Treating all three as interchangeable "feedback" collapses information a refinement process could otherwise use more precisely.

The same feedback vocabulary appears again from the human-oversight side

This course's Human-AI Interaction and Oversight material names the identical three feedback forms — approvals, corrections, ratings — feeding into the identical data flywheel concept, framed from a different angle: oversight as a source of learning signal, not merely a gate that approves or blocks an action before it happens. The overlap is not a coincidence or a duplication to worry about; it reflects that structured feedback genuinely serves two purposes an agent's design has to satisfy at once. From this module's angle, feedback is one of several inputs a refinement process uses to decide what to investigate next, always confirmed against the fixed benchmark before a change ships. From the oversight angle, the same captured signal is also evidence that a human reviewer's judgment is being incorporated into the system rather than exercised once and discarded — a distinct concern about accountability and trust that sits alongside, rather than inside, the refinement question this lesson is answering. The mechanism captured in Section 1's table is genuinely shared infrastructure; what each module does with it differs.

L1 — Intuition

A flywheel, in its original mechanical sense, is a heavy wheel that, once set spinning, stores momentum and makes the next push easier than the last — each turn feeds the next, and the system as a whole accelerates rather than needing to be restarted from scratch every cycle. NVIDIA's data flywheel borrows that image directly: real usage of a deployed agent generates data (the structured feedback from Section 1), that data feeds a refinement process that produces an improved version of the agent, and the improved agent, by being genuinely better, attracts or retains more usage — which generates more data, which feeds the next refinement, and the cycle continues. The flywheel's whole appeal is that it is self-reinforcing in a way a one-time evaluation effort is not: it keeps generating its own fuel as long as the loop stays closed.

L2 — Mechanism

Mechanically, the flywheel has four connected stages, and it is worth tracing them in order because a break at any single stage stalls the whole cycle, not just the stage where the break occurred. Usage — real interactions happen, generating real queries the agent has to handle. Structured feedback capture — ratings, corrections, and escalations from those real interactions get recorded in the defined forms Section 1 described, rather than lost. Refinement — the captured feedback informs a change: a prompt adjustment, a new example added to a training or evaluation set, a newly identified capability gap addressed. Improved usage — the refined agent, now measurably better at whatever the feedback revealed, serves the next round of real interactions better, which can mean more usage, more trust, or simply better outcomes on the existing volume of usage, any of which restarts the cycle by generating a fresh round of structured feedback to capture.

The flywheel's value compounds specifically because each stage feeds the next one directly, without requiring a separate, disconnected initiative to restart the loop. A one-time customer survey is not a flywheel, even if it produces useful feedback, because nothing about running the survey once creates a mechanism for the next round of feedback to arrive automatically from ongoing usage. The flywheel language signals something more specific than "we listen to users" — it signals a designed loop where usage itself is the fuel supply for the next iteration, continuously, rather than periodically.

L3 — The exam-relevant edge case: a flywheel that is spinning but generating the wrong kind of data

A subtlety worth naming precisely: a flywheel can be genuinely spinning — usage generating feedback, feedback informing changes, changes shipping — while still not actually improving anything, if the captured feedback is systematically skewed toward a narrow, unrepresentative slice of real usage. A team that only captures feedback from users who bother to tap a rating button, for instance, is drawing structured feedback disproportionately from users motivated enough to react — often the most frustrated or the most delighted, rather than the median user's typical experience — and refining against that skewed sample can optimize the agent for the vocal minority's concerns while leaving the median experience untouched or even quietly worse. A scenario question describing a flywheel that appears active by every mechanical measure (feedback is being captured, changes are shipping) but is not improving the fixed evaluation benchmark's score is testing exactly this: a spinning flywheel is not automatically a correctly-calibrated one.

⭐ THE EARNED INSIGHT A flywheel's momentum comes from the loop closing, not from the feedback inside it being representative or correct — which means a flywheel can accelerate in a genuinely wrong direction just as readily as a right one, entirely on the strength of its own mechanical design. The discipline that keeps a flywheel honest is not spinning faster; it is checking, on a fixed and representative measure, whether the direction it has been spinning actually matches the direction the product needs — which is precisely the job the next section explains feedback alone cannot do.

[GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) states this directly and it is the single most important sentence in this lesson: "Feedback informs refinement; it does not replace benchmarks — you still need repeatable evaluation to know a change helped rather than just shifted behavior." Unpacking why requires returning to what M3-01 established about what makes a comparison trustworthy at all.

Structured feedback, even captured perfectly, is not a fixed, representative task set scored by defined criteria — it is an uncontrolled, self-selected sample of whatever real users happened to react to, in whatever way they happened to react, at whatever volume happened to occur. That is exactly the profile M3-01 warned would produce an incomparable result: the population of users providing feedback on version 1.0 is not the same population providing feedback on version 1.1, the volume and mix of query types is not held constant, and there is no defined scoring criterion applied consistently to every piece of feedback the way a rubric is applied to every task in a fixed benchmark. Feedback trending more positive after a change is suggestive, but it carries none of the structural guarantees that make a benchmark's before-and-after comparison attributable.

"A change helped rather than just shifted behavior" names the specific risk directly. A change motivated by negative feedback on how-to questions might genuinely fix how-to questions — or it might change the agent's behavior in a way that happens to reduce complaints on how-to questions while quietly introducing a new problem elsewhere that has not yet generated enough feedback volume to be visible. Feedback trending positive after a change is consistent with both stories; only re-running the fixed evaluation benchmark across every task category, not just the one feedback flagged, can distinguish "this change helped" from "this change moved the problem somewhere feedback has not caught up to yet."

This is the same relationship M3-02 established between profiling and evaluation, applied to a third instrument. A profiler's timing data and structured feedback's ratings, corrections, and escalations are both genuinely useful, evidence-based signals about specific things worth investigating — a slow step, a category generating complaints — and both share the identical limitation relative to the fixed benchmark: neither one, by construction, checks the categories it was not already looking at. The fixed benchmark is the one instrument in this module built specifically to check everything, every time, on the identical footing, which is exactly why it is the step none of the others can substitute for, however good the signal that motivated a change in the first place.

A support agent's structured feedback log shows a spike in negative ratings and escalations specifically tied to how-to questions over a two-week window — a pattern the team traces, via a sample of the underlying corrections users submitted, to the agent frequently citing an outdated version of the product's setup documentation.

Structured feedback summary, how-to category, two-week window
Ratings:      62% negative (up from a typical 15-20% baseline)
Corrections:  14 submitted, 11 of which specifically flag "outdated
              instructions" or "this doesn't match the current app"
Escalations:  9 (up from a typical 2-3 per two-week window)

Refinement made: retrieval index updated to deprioritize the outdated
setup-documentation version and prioritize the current one.

This is a constructed scenario, illustrative rather than drawn from a real support log. The feedback signal here is genuinely useful and correctly interpreted — it identified a real, specific problem (a stale document in the retrieval index) that the team would have had no other easy way to notice, since a fixed evaluation benchmark built before the documentation went stale would have no way of knowing to test for staleness it did not yet know existed. This is exactly the kind of gap structured feedback is positioned to catch that a benchmark, however well-designed, cannot: feedback surfaces problems from real, unpredictable usage; a benchmark only tests what it was built in advance to test.

Re-evaluation on the fixed 50-ticket benchmark, before and after the fix
                          Before fix    After fix
How-to:                      81%           91%
Billing:                     82%           82%    [unchanged]
Access:                      94%           94%    [unchanged]
Aggregate:                   86%           89%

Here the fixed benchmark plays its irreplaceable role: it confirms the feedback-driven fix actually improved how-to performance (81% to 91%) without degrading the two categories the feedback said nothing about. Had the benchmark instead shown how-to improving to 91% while billing quietly dropped from 82% to 70% — an effect the retrieval-index change might plausibly have on an unrelated category sharing the same index — the feedback log alone, which was only tracking how-to complaints, would never have surfaced that cost. The feedback found the problem; only the benchmark, run across every category rather than just the one feedback flagged, confirmed the fix was actually net-positive.

A second team runs the identical support agent, but only surfaces the ratings prompt to users after an escalation — that is, the rating button only appears once a conversation has already been flagged as needing human review. Over several refinement cycles, the team tunes the agent heavily against the corrections submitted through this channel, since it is the only structured feedback source they built.

Feedback captured over three refinement cycles (escalation-triggered ratings only)
Cycle 1: 40 corrections submitted, all from escalated (i.e., already-failing) conversations
Cycle 2: 35 corrections submitted, all from escalated conversations
Cycle 3: 31 corrections submitted, all from escalated conversations

Each cycle's refinement: retrain/adjust against the submitted corrections,
ship, repeat.

This is a constructed scenario, illustrative rather than measured. By the mechanical definition in Section 2, this flywheel is genuinely spinning: usage is producing structured feedback, feedback is driving refinement, and refinements are shipping every cycle. What the mechanism cannot see, from inside itself, is that every single piece of feedback it has ever received came from conversations that had already failed badly enough to escalate — the flywheel has no channel at all for hearing from the large majority of conversations that succeeded, or that failed only mildly and never reached escalation.

Fixed 50-ticket benchmark, run after three refinement cycles
Billing:    14 / 17 passed  (82%)   [unchanged across all three cycles]
Access:     15 / 17 passed  (88%)   [down from 94% before cycle 1]
How-to:     10 / 16 passed  (63%)   [unchanged across all three cycles]

Aggregate: 39 / 50 passed  (78%)    [down from 86% before cycle 1]

Three cycles of a visibly active, spinning flywheel produced an aggregate score that went down, not up, and the category that quietly regressed — access questions — never generated a single escalation, so the feedback channel had no way of ever flagging it. This is Section 2's L3 edge case made concrete: activity in the flywheel (feedback captured, changes shipped) is not the same claim as improvement, and only the fixed benchmark, checked across every category regardless of which categories were generating feedback, revealed that this particular flywheel had been tuning the agent against an unrepresentative slice of its real usage the whole time.

Evaluation and Tuning carries 13% of the NCP-AAI blueprint, and Objective 3.3 names collecting and integrating structured user feedback for iterative improvement as its own distinct skill, separate from Objective 3.1's evaluation pipelines and Objective 3.5's results analysis. [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) frames the data flywheel as "the core of NVIDIA's data flywheel idea: real usage generates data that improves the next iteration, which improves usage, and so on" — vocabulary specific enough to expect direct recall, alongside the sharper, more consequential distinction this lesson has built around: that feedback informs but does not replace repeatable evaluation.

Expect the question shape to describe a team that ships a change based on positive-trending feedback alone, without re-running a fixed benchmark, and ask what is missing from their process — the keyed answer names the missing repeatable evaluation step, and distractors typically offer "nothing is missing, the feedback confirms the change worked" as the tempting wrong answer. A second recurring shape names the three feedback forms — ratings, corrections, escalations — directly, or asks what "the data flywheel" specifically refers to, testing whether the term is understood as usage-generates-data-that-improves-the-system-that-drives-more-usage rather than as a loose synonym for "continuous improvement."

Structured feedback is captured in a defined, consistent form — a rating, a correction, an escalation flag — tied to a specific, retrievable interaction, so that a refinement process can systematically read and act on it. Unstructured feedback, by contrast, might be just as genuine a reaction from a user, but if it is never logged in a form the system can process — a verbal comment, an unrecorded impression — it cannot feed a refinement loop at all, regardless of how informative it might have been. The distinguishing factor is not the feedback's sincerity or accuracy; it is whether it was captured in a form a downstream process can actually use.

Because feedback, even captured perfectly, is an uncontrolled, self-selected sample rather than the fixed, representative, consistently-scored comparison a repeatable benchmark provides — it lacks the structural guarantees M3-01 established as necessary for an attributable result. A positive trend after a change is consistent with the change genuinely helping, but it is equally consistent with the change shifting a problem to an area feedback has not yet caught up to reporting on, which Section 4's worked example makes concrete: a fix confirmed as net-positive only because the fixed benchmark checked every task category, not just the one the feedback happened to flag.

This does not make feedback a weaker or less valuable signal than a benchmark score — it makes feedback a different kind of signal, suited to a different job. Feedback is unmatched at telling you where to look, because it comes directly from real usage and can surface a problem no one thought to write a benchmark task for in advance, exactly as Section 4's outdated-documentation example showed. A fixed benchmark is unmatched at telling you whether a change actually worked, because it applies the identical, representative, consistently-scored test before and after, regardless of which category happened to generate the loudest feedback. Neither instrument does the other's job well, and the discipline this lesson is building is using each one for the job it is actually suited to.

1. A user tells a colleague an agent's response "wasn't very helpful," but this comment is never logged anywhere the agent's team can see. Is this structured feedback?
   • A. Yes, because the user's reaction was genuine.
   • B. No, because it was never captured in a defined, retrievable form a refinement process can act on.
   • C. Yes, as long as the colleague remembers to mention it later.
   • D. No, because only escalations count as feedback.
2. What does "the data flywheel" specifically refer to?
   • A. Any general effort to improve a product over time.
   • B. A self-reinforcing loop where usage generates structured feedback, feedback drives refinement, and the refined system drives further usage.
   • C. A single, one-time survey of user satisfaction.
   • D. The evaluation pipeline's fixed task set.
3. A team ships a change based on a feedback trend turning more positive, without re-running the fixed evaluation benchmark. What is missing from their process?
   • A. Nothing; a positive feedback trend is sufficient confirmation the change helped.
   • B. Confirmation, via the fixed benchmark, that the change helped rather than just shifted the problem to an area feedback has not yet surfaced.
   • C. A louder announcement of the change to users.
   • D. A second round of the identical feedback collection.
4. In Section 5's worked example, why did the aggregate benchmark score decline even though the flywheel was actively capturing feedback and shipping refinements every cycle?
   • A. The benchmark itself was broken.
   • B. Feedback was only ever captured from already-escalated conversations, so an entire category regressing quietly was invisible to the feedback channel.
   • C. The team stopped shipping refinements after cycle one.
   • D. Ratings are never a valid feedback form.
5. What distinguishes a rating from a correction as feedback forms?
   • A. A rating states the specific correct answer; a correction is a coarse quality judgment.
   • B. A rating is a coarse quality judgment on a response; a correction is the user's own statement of what the correct answer should have been.
   • C. They are interchangeable and carry identical information.
   • D. Only corrections can be captured in structured form.

Answers

1. B. Whatever the user's sincerity, the comment was never captured in a form the system can read and act on, so it cannot function as structured feedback regardless of how real the reaction was.
2. B. The flywheel is a specific, self-reinforcing mechanism, not a general synonym for continuous improvement or a one-time initiative.
3. B. Feedback trending positive is consistent with a change helping, but also consistent with the change shifting a problem elsewhere; only the fixed benchmark, checked across categories, distinguishes the two.
4. B. The feedback channel only ever heard from conversations that had already escalated, so a category regressing without generating an escalation was structurally invisible to that channel, even though the flywheel was mechanically active.
5. B. A rating aggregates a quality signal across interactions; a correction supplies the specific right answer for one interaction — different information, different downstream uses.

• Structured feedback — ratings, corrections, escalations — is feedback captured in a defined, consistent form a refinement process can systematically act on, distinct from an unlogged, informal reaction that a system cannot use.
• The data flywheel is a specific, self-reinforcing loop: usage generates data, data drives refinement, refinement improves the system, and the improved system drives further usage — not a loose synonym for continuous improvement.
• A flywheel can be mechanically spinning — feedback captured, changes shipping — while not actually improving the fixed benchmark's score, especially if the captured feedback is skewed toward a narrow, self-selected slice of users.
• Feedback informs refinement; it does not replace a repeatable benchmark, because only the fixed evaluation pipeline can confirm a change helped rather than shifted the problem somewhere feedback has not yet surfaced.
• The three feedback forms carry different information — a rating signals an aggregate trend, a correction supplies a specific right answer, an escalation flags a capability gap — and conflating them loses precision a refinement process could otherwise use.
• The correct process is feedback-informed hypothesis, then benchmark-confirmed result: use feedback to decide what to investigate, and the fixed benchmark from M3-01 to confirm the resulting change was actually net-positive.

Confirming that a feedback-driven change genuinely helped, as M3-01's fixed benchmark does, is still only half of what closing the loop on structured feedback actually requires — the other half is knowing how to read a set of evaluation results, feedback-driven or otherwise, and turn them into the next specific, targeted change, rather than a vague sense that "something in the how-to category" needs attention.

Next: M3-06 covers analyzing results to guide targeted optimization — categorizing failures, finding the highest-impact weakness, and changing exactly one thing before re-evaluating on the fixed set, the same experimental discipline as an A/B test, applied to the agent itself, and the module's closing lesson.