HITL Techniques: Supervised Learning, RLHF, and Active Learning

[GROUND TRUTH] (Sources/ncp-aai/domain-10-human-ai-interaction-oversight.md) Human input is embedded in AI workflows through several distinct techniques, and the source material names three explicitly: supervised learning, where humans label training data; RLHF, where human feedback trains a reward model to align behavior; and active learning, where humans weigh in specifically on the model's low-confidence cases. Each of the three moves a human's judgment into the system through a different channel, and each produces a different kind of artifact as a result — a labeled dataset, a trained reward model, or a targeted correction on an uncertain prediction.

The table's middle column is the one worth reading closely, because it is where the three techniques actually diverge: labeling is an act of authorship (the human supplies the correct answer directly), RLHF is an act of comparison (the human judges which of several already-produced outputs is better, without necessarily being able to author a better one from scratch), and active learning is an act of selective attention (the human's effort is spent only where the model's own signal says it is most needed). Confusing any two of these — treating a preference ranking as though it were a label, or assuming active learning reviews everything rather than a targeted subset — is exactly the kind of category error a scenario question is built to catch.

L1 — Intuition: a human writes down the right answer, and the model learns to imitate it

Supervised learning is the most direct of the three techniques: a human looks at a piece of data — a support ticket, an image, a passage of text — and assigns it the answer that is considered correct, whether that answer is a category, a numeric value, or a full response the model should learn to produce. The model is then trained to reproduce, as closely as possible, the labels humans assigned. There is no comparison step and no separate scoring model in this technique; a human simply authors the ground truth directly, one example at a time, and the model imitates the resulting dataset.

L2 — Mechanism: what makes labeling expensive and what makes it foundational anyway

Every other technique in this lesson, and much of the rest of this certification's material on evaluation and tuning, depends on labeled data existing somewhere in the pipeline — a reward model in RLHF still needs an initial dataset to bootstrap from, and active learning's low-confidence cases still get resolved, ultimately, by a human supplying a label for that specific case. Labeling is therefore not a technique that gets replaced by the other two; it is the base case they both still rely on, applied either to the whole dataset upfront (plain supervised learning) or to a much smaller, carefully chosen subset (active learning, covered in section 4). The expense that makes supervised labeling the least scalable of the three techniques on its own is exactly what motivates the other two: if every example an agent will ever need to handle correctly had to be individually labeled by a human before the system could learn from it, an agent's training data requirements would grow linearly with every new case type it needed to cover, with no mechanism for the system to tell you which cases actually need that attention and which do not.

L3 — The exam-relevant edge case: labeling quality is a ceiling, not a floor

A subtlety worth holding onto is that a model trained via supervised learning cannot exceed the quality of the labels it was trained on — if human labelers disagree with each other, or apply inconsistent standards across examples, the model inherits that inconsistency as noise in its training signal, not as a correction to it. This is the same human-error-and-inconsistency drawback M10-01 named for HITL generally, showing up here in its most literal form: a labeling process is only as good as its labelers, and unlike a reward model (section 3) which is explicitly built to model aggregate preference across possibly-disagreeing raters, plain supervised learning has no built-in mechanism for reconciling disagreement beyond whatever quality-control process — multiple labelers per example, adjudication of disagreements — a team chooses to build around it.

L1 — Intuition: it is easier to say which is better than to write the best one

RLHF asks a fundamentally different question of a human than supervised learning does. Instead of "what is the correct answer here," RLHF asks "which of these candidate answers do you prefer" — a comparison, not an act of authorship. [GROUND TRUTH] (Sources/ncp-aai/domain-10-human-ai-interaction-oversight.md) Humans provide feedback that trains a reward model to align behavior, and that reward model is what actually shapes the agent's future outputs, rather than the human's individual comparisons being applied one at a time.

L2 — Mechanism: why a reward model exists as a separate artifact at all

The reason RLHF introduces a reward model rather than simply collecting a pile of human comparisons and stopping there is scale: a single agent, during training, needs to evaluate an enormous number of candidate outputs to know which direction to adjust its behavior in, far more than a human panel could realistically compare one at a time in real time. The reward model exists to generalize a finite set of human comparisons into a scoring function that can be applied automatically, at scale, to outputs the human panel never actually saw — it is a learned stand-in for the panel's judgment, trained on their comparisons and then queried far more often than the panel itself could be. This is precisely the mechanism [GROUND TRUTH] (Sources/ncp-aai/domain-10-human-ai-interaction-oversight.md) names as training a reward model to align behavior: the human input does not act on the agent directly, it acts on the reward model, and the reward model then acts on the agent.

L3 — The exam-relevant edge case: RLHF versus plain supervised labeling is the domain's named trap

[GROUND TRUTH] (Sources/ncp-aai/domain-10-human-ai-interaction-oversight.md) states this confusion directly as a common exam trap: don't confuse RLHF, a reward model built from human feedback, with plain supervised labeling. The distinction that resolves nearly every scenario question testing this boundary is the one drawn in section 1's table: supervised labeling is a human authoring a correct answer directly; RLHF is a human comparing outputs and having that comparison trained into a separate scoring model, which is then what actually drives the behavior change. A question that describes humans "labeling data" is describing supervised learning, full stop, even if the labeling happens to be about which response is preferable — the moment that preference signal gets used to train a reward model that then optimizes a policy, the technique has crossed into RLHF territory, and calling it "just labeling" at that point is the specific mistake this domain flags.

L1 — Intuition: don't review what the model already gets right

Active learning starts from an observation the other two techniques do not use: a model's own confidence in a given prediction is itself useful information. [GROUND TRUTH] (Sources/ncp-aai/domain-10-human-ai-interaction-oversight.md) Humans weigh in on low-confidence cases the model is unsure about — not on every case, and not on a random sample of cases, but specifically on the ones where the model's own signal indicates it is most likely to be wrong. This is the technique that answers the scalability drawback M10-01 named most directly: if human review is expensive per item, the cheapest way to buy the most improvement per reviewer-hour is to point reviewers at exactly the cases where their judgment is most likely to change an outcome, rather than spreading that same reviewer-hour budget evenly across cases the model was already confident, and likely correct, about.

L2 — Mechanism: what "low-confidence" means and how a system decides to route a case

A model's confidence in a prediction typically comes from a score the model itself produces alongside its output — a probability, a margin between the top two candidate answers, or an explicit uncertainty estimate, depending on what kind of model is producing the prediction. A system implementing active learning sets a threshold on that confidence score: predictions above the threshold proceed automatically, and predictions below it get routed to a human for review or correction. This is structurally the same review-rate lever M10-01's worked examples sized in dollar and latency terms, except that here the criterion for who gets routed to review is not a fixed percentage of volume but the model's own per-case uncertainty — which means the review rate active learning produces is not set directly by a policy decision, it is an emergent consequence of how well-calibrated and how confident the model happens to be on the specific traffic it is currently seeing.

L3 — The exam-relevant edge case: active learning is a targeting strategy, not a scale of review

The trap worth naming explicitly here is the reverse of the RLHF-versus-labeling confusion: active learning is sometimes mistaken for "human review of everything," when its entire point is the opposite — routing to a human is the exception, triggered by low confidence, not the default. A workflow that reviews 100% of an agent's outputs regardless of confidence is not active learning under this technique's definition, however similar it might look on the surface; it is simply comprehensive human review, which M10-01's worked examples costed out directly and which active learning exists specifically to avoid paying for on every case. Recognizing that active learning's defining feature is the selectivity of what gets routed to a human — not the presence of a human in the loop at all, which all three techniques share — is what separates it cleanly from the other two in a scenario question.

Constructed scenario, invented for teaching, describing three separate teams' workflows without naming which technique each one is using.

Workflow A: A team hires annotators to read 50,000 customer support transcripts
and assign each one a category — "billing," "technical," "account access," or
"other" — which becomes the training set for a ticket-routing model.

Workflow B: A team shows annotators pairs of draft responses their agent
generated for the same prompt and asks, for each pair, "which response would
you rather receive." Those comparisons train a scoring model, and the agent
is then tuned to produce responses that score highly against it.

Workflow C: A deployed classification agent flags any prediction where its
own confidence score falls below 70% and routes only those flagged cases to
a review queue; predictions at or above 70% confidence proceed automatically.

Step 1 — identify what the human is doing in each workflow. In Workflow A, annotators are directly assigning a category to each transcript — an act of authorship, producing a label. In Workflow B, annotators are comparing two already-generated drafts and expressing a preference — an act of comparison, not authorship. In Workflow C, humans are reviewing only the subset of cases a confidence threshold flagged — an act of targeted, threshold-triggered attention.

Step 2 — match each action to its technique. Workflow A is supervised learning: a human labels data directly, and that labeled data becomes the training set, with no comparison step and no reward model involved. Workflow B is RLHF: the human comparisons train a scoring model (the reward model), which is what then shapes the agent's tuned behavior — the human never authors a response directly, they only compare. Workflow C is active learning: the routing criterion is the model's own confidence score, and only cases below the threshold reach a human; cases above it never do.

Step 3 — check for the trap each workflow is designed to test. Workflow A could be mistaken for active learning if a reader assumes "labeling" always implies selectivity — it does not; nothing in Workflow A restricts which transcripts get labeled, so it is plain supervised learning at full coverage. Workflow B could be mistaken for supervised learning if a reader focuses only on "annotators provide input" and misses that the input is a comparison feeding a separate scoring model, which is the detail that makes it RLHF rather than labeling. Workflow C could be mistaken for a form of supervised learning applied narrowly, but the defining active-learning detail — routing is triggered by the model's own uncertainty, not by a fixed sampling rule set in advance — is what places it in its own category rather than being "supervised learning on a subset chosen arbitrarily."

Constructed scenario, invented for teaching, illustrative figures. A document-classification agent processes 10,000 documents per day, and its confidence score is well-calibrated enough that the fraction of documents falling below any given threshold is known from historical data.

Constructed scenario, invented for teaching — illustrative figures, not measured data.

Historical confidence distribution for this agent's daily volume of 10,000 documents:
  Below 50% confidence:  2% of volume  ->  200 documents/day
  Below 70% confidence:  8% of volume  ->  800 documents/day
  Below 90% confidence: 22% of volume  -> 2,200 documents/day

Reviewer throughput: 60 documents reviewed per hour, 8-hour shift = 480/day/reviewer

Step 1 — reviewer headcount needed at each candidate threshold.

Threshold 50%: 200 / 480  = 0.42 reviewers -> round up to 1
Threshold 70%: 800 / 480  = 1.67 reviewers -> round up to 2
Threshold 90%: 2,200 / 480 = 4.58 reviewers -> round up to 5

Step 2 — compare against a supervised-learning-at-full-coverage baseline for the same volume.

Full-coverage labeling (all 10,000 documents reviewed, the supervised-learning baseline):
  10,000 / 480 = 20.8 reviewers -> round up to 21

Step 3 — read what the threshold choice actually buys. Moving the active-learning threshold from 50% to 90% multiplies reviewer headcount roughly elevenfold (1 to 5) while still requiring barely a quarter of the full-coverage baseline's 21 reviewers — active learning at even its most conservative threshold in this scenario is dramatically cheaper than reviewing every case, which is exactly the technique's value proposition stated in reviewer-headcount terms rather than in the abstract. The choice of threshold is still a real design decision, not a free dial: a 90% threshold catches far more of the agent's uncertain cases than a 50% threshold, at meaningfully higher reviewer cost, and the right setting depends on how expensive an undetected low-confidence error actually is downstream — the same cost-versus-benefit arithmetic M10-01 applied to review rate generally, now applied specifically to where an active-learning threshold gets set.

Step 4 — see why active learning's cost curve behaves differently from supervised learning's. The full-coverage baseline's reviewer count (21) does not change no matter how confident the agent becomes over time, because it reviews every document regardless of confidence. The active-learning configurations, by contrast, get cheaper automatically as the agent improves and its confidence distribution shifts — a better-calibrated, more accurate agent produces fewer low-confidence cases at any given threshold, which means the same threshold routes fewer documents to review next month than it does this month, with no change to the threshold itself required. This is a structural advantage active learning has that plain supervised-learning-at-full-coverage does not: its cost is coupled to the model's own improving performance, while full coverage's cost is coupled only to volume.

It is worth walking through a single agent's lifecycle end to end to see that these three techniques are not competing choices a team picks one of — they typically show up at different points in the same system's life, each solving a different problem at that point.

At initial build time, before an agent has produced a single real output, there is nothing yet for a reward model to compare and no confidence signal worth trusting, because the model has not yet learned anything about the task. Supervised learning is the only one of the three techniques that works here: a team collects and labels an initial dataset — the categories, the correct responses, the ground truth — and trains the first working version of the model against it. Neither RLHF nor active learning has a role at this stage, because both depend on a model that can already produce candidate outputs (for RLHF's comparisons) or a confidence signal worth thresholding (for active learning's routing), and neither of those exists yet.

Once an initial model exists and can generate candidate responses, RLHF becomes available as a refinement step: a team collects human comparisons between candidate outputs, trains a reward model from those comparisons, and optimizes the policy against it to shape qualities — tone, helpfulness, calibrated hedging — that are easier for a human to recognize by comparison than to author directly as a label. This stage is where a system moves from "produces plausible outputs" to "produces outputs humans actually prefer," and it depends on the supervised-learning stage having already produced a model capable of generating something worth comparing.

Once the refined model is deployed and running on live traffic, active learning becomes the ongoing mechanism for catching what neither of the first two stages fully resolved: cases the model is genuinely unsure about, in production, that were never part of the original training or comparison data. This stage never really ends — new low-confidence cases keep appearing as the traffic distribution shifts — and the corrections it produces can themselves become new supervised-learning data for the next training cycle, closing a loop back to the first stage rather than existing as a one-time step.

Reading the three techniques this way — build-time labeling, then preference-based refinement, then ongoing targeted correction — makes clear why a real production system is very unlikely to use exactly one of the three and call it done: each solves a problem specific to a different moment in the system's life, and a mature deployment accumulates all three over time rather than choosing permanently among them.

Objective 10.1 of Domain 10 covers human-in-the-loop broadly, and this lesson's three techniques are the domain's concrete answer to "how does a human's input actually get into the system." [VENDOR SPEC] (Sources/ncp-aai/domain-10-human-ai-interaction-oversight.md) The source material's own scope note names RLHF-versus-supervised-labeling directly as a common exam trap, which signals where scenario questions concentrate: not on defining any one technique in isolation, but on distinguishing it from its closest neighbor.

How the question tends to be phrased

Expect a short workflow description — annotators doing something with model outputs — followed by a request to name which HITL technique it represents, with the four options spanning supervised learning, RLHF, active learning, and a distractor describing something else entirely (unsupervised clustering, or fully automated processing with no human at all). A second common shape asks specifically what distinguishes RLHF from supervised labeling, with the keyed answer naming the reward model as the artifact that makes RLHF distinct. A third shape describes a confidence-threshold routing mechanism and asks which technique it is, testing whether "low-confidence cases only" is recognized as active learning's defining feature.

What the distractors typically look like

The standard trap collapses RLHF into supervised learning by describing the reward-model training step vaguely enough that it reads as ordinary labeling — recognizing the reward model as the tell is the fix. A second distractor offers "reviewing every case" as a definition of active learning, omitting the selectivity that is the technique's entire point. A third distractor presents a workflow using more than one technique across different stages and asks for a single answer, when the correct read is that a real pipeline can use more than one technique for different parts of the same system.

Supervised learning has a human directly author the correct answer for a piece of data, and the model is trained to reproduce those answers. RLHF has a human compare candidate outputs and express a preference, and that preference is used to train a separate reward model, which then shapes the agent's behavior through policy optimization. The human never authors an answer in RLHF the way they do in supervised learning — they only judge which of several already-produced outputs is better, and a downstream model turns that judgment into a scoring function applied at scale.

No — active learning specifically routes only the model's low-confidence cases to a human, leaving high-confidence predictions to proceed automatically. The technique's entire value proposition depends on this selectivity: it exists to concentrate a limited amount of expensive human attention on the cases where that attention is most likely to change an outcome, rather than spreading it evenly across every case regardless of how confident the model already was.

• Three distinct mechanisms, not three names for the same thing: supervised learning is authorship, RLHF is comparison feeding a reward model, active learning is targeted attention triggered by low confidence.
• The reward model is what makes RLHF distinct from plain labeling — a human's comparative judgment is trained into a separate scoring artifact, which then drives behavior change, rather than the human directly authoring the correct answer.
• Active learning's defining feature is selectivity, not the presence of a human — routing only low-confidence cases to review is the entire point, and a 100%-review workflow is not active learning by this definition.
• A model's quality under supervised learning cannot exceed its labels' quality — labeler disagreement and inconsistency become training noise with no built-in correction mechanism.
• Real systems commonly combine all three — initial data labeled via supervised learning, behavior refined via RLHF, and live uncertain cases routed via active learning, each addressing a different stage.
• Lowering an active-learning confidence threshold is not a free accuracy gain — it routes more volume to human review, reintroducing the same scalability cost M10-01 sized directly.

⭐ THE EARNED INSIGHT

The three techniques are really three different answers to the same constraint — human attention is the scarce, expensive resource M10-01 costed out, and each technique is a different strategy for getting the most value out of a fixed budget of that attention: spend it broadly across a full dataset (supervised learning), spend it narrowly on comparisons that a reward model can then generalize far beyond what any human panel reviewed directly (RLHF), or spend it only where the model itself signals it is most needed (active learning). None of them make human attention free; they just change what you are buying with it.

Knowing which technique embeds a human's input is only half of designing oversight into a system — the other half is what happens to that input once it is captured, and whether it actually changes anything downstream. Next: M10-03 takes that up directly — captured approvals, corrections, and ratings are not just a gate, they are a learning signal that feeds the same improvement flywheel this certification's evaluation domain builds on, and structuring that feedback loop well is a distinct design problem from choosing which technique captures the input in the first place.