Evaluation Pipelines and Task Benchmarks: Building a Repeatable Agent Harness

Identity statement: an evaluation pipeline is a repeatable harness that runs an agent over a fixed, representative set of tasks and scores the resulting outputs against defined criteria, producing a result that is comparable to the result of running the same pipeline again later. [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md): "An evaluation pipeline runs the agent over a fixed, representative set of tasks and scores the outputs on defined criteria, so results are repeatable and comparable run to run." Every clause in that sentence is load-bearing, and each one rules out a specific shortcut that looks like evaluation but is not.

"Fixed" rules out picking a fresh, different handful of examples each time you check the agent — if the tasks change between runs, any difference in the score is entangled with which tasks happened to be chosen, not with whether the agent actually got better or worse. "Representative" rules out a task set so narrow or so easy that a high score tells you nothing about how the agent handles the range of situations it will actually face in production — ten trivial questions the agent has never once failed is not a benchmark, it is a formality. "Scores the outputs on defined criteria" rules out an evaluator's private, undocumented sense of "that response felt about right," because a criterion nobody wrote down cannot be applied consistently by the same person twice, let alone by two different people, let alone by an automated scorer standing in for either. And "repeatable and comparable run to run" is the payoff every other clause exists to deliver: a number you can trust to mean the same thing today as it meant last month, because nothing about how it was produced was allowed to drift.

A task benchmark is the standardized task suite an evaluation pipeline runs the agent over — the actual list of prompts, scenarios, or multi-step problems, held fixed, that stands in for the range of things the agent needs to be good at. The pipeline is the process; the benchmark is the fixed content the process runs over. Confusing the two is a common early mistake: a team that changes its benchmark every time it evaluates has a process (the scoring mechanics) but no actual pipeline, because the fixed-task-set half of the definition has quietly dropped out.

Why "comparable run to run" is the property that makes everything downstream possible

Every later lesson in this module assumes you already have a trustworthy number to work with. Profiling versus evaluation (M3-02) assumes you can tell a real quality change from noise. The accuracy-versus-latency trade-off assumes you can measure both sides of the trade honestly. Structured feedback and targeted optimization both assume you can re-run the same measurement after a change and trust that any difference reflects the change, not a shifted task set. None of that is possible without the property this section names directly: a pipeline whose output means the same thing twice. Skip building that property in first, and every later lesson in this module is standing on a foundation that was never poured.

⭐ THE EARNED INSIGHT An evaluation pipeline's real product is not the score it prints — it is the license to compare two scores at all. A single number, produced once, tells you almost nothing on its own; the entire value of the harness described in this lesson is that it lets you run the identical measurement again later and trust that a changed number means the agent changed, not that the measurement did. Build the harness loosely, and every later lesson in this module — profiling, the accuracy-latency trade-off, feedback, targeted optimization — inherits a number nobody can actually trust.

L1 — Intuition

Think of an evaluation pipeline the way you would think of a standardized test given to different students at different times: the questions do not change between sittings, the grading rubric does not change, and a score from this year's sitting can be honestly compared to a score from last year's, because the only thing that varied is the student — or, in this case, the agent version — being tested. Strip away everything mechanical about how the test is administered, and the intuition is this: hold the test fixed, hold the grading fixed, let the thing being tested vary, and only then does a score difference mean anything.

L2 — Mechanism

Mechanically, an evaluation pipeline has three moving parts that all have to be present for the identity statement in Section 1 to hold. First, the fixed task set — a concrete list of inputs (prompts, multi-step scenarios, tool-use situations) that does not change between runs of the same evaluation. Second, the agent under test, invoked exactly the way it would be invoked in production, so the evaluation measures the real system rather than a simplified stand-in for it. Third, a scoring mechanism — a defined criterion, applied consistently, that turns each raw output into a number or a category: correct/incorrect, a numeric quality score, a pass/fail against a rubric, or an LLM-as-judge verdict against written criteria. The pipeline's job is to run every task in the set through the agent, apply the scoring mechanism to every resulting output, and aggregate the per-task scores into a result you can report and, later, compare against a different run's result.

The aggregation step deserves its own attention, because it is where a pipeline can quietly produce a technically-correct but practically-useless number. A single aggregate score — "82% pass rate" — tells you the overall shape of performance but hides exactly which tasks failed and why, which is precisely the information the later lesson on analyzing results to guide targeted optimization needs. A well-built pipeline therefore preserves the per-task breakdown underneath the aggregate, not just the headline number, because the headline number is what you report and the per-task breakdown is what you act on.

L3 — The exam-relevant edge case: a fixed task set does not by itself guarantee a fair comparison

A subtlety the exam tests directly: keeping the task set fixed is necessary but not sufficient for a trustworthy comparison. If the scoring criteria change between two runs — a rubric gets stricter, an LLM-judge prompt gets rewritten, a human grader starts applying a different standard — the task set being identical does not save the comparison, because the thing that changed is now the yardstick, not the tasks. A scenario question that describes an unchanged benchmark but a changed scoring method is testing exactly this: "fixed tasks" is one of three requirements, not the whole requirement, and a comparison that gets the task set right while letting the scoring criteria drift is just as broken as one that changes both.

Objective 3.1 names implementing evaluation pipelines and task benchmarks as a skill, and the NeMo Agent Toolkit gives you a specific, named way to implement one rather than requiring you to build the mechanics described in Section 2 from scratch every time. [VENDOR SPEC] (Sources/ncp-aai/domain-3-evaluation-tuning.md): "The NeMo Agent Toolkit ships a built-in evaluation system, so you can score workflows without wiring up bespoke harnesses." The comparison below lays out what that buys you against writing your own.

None of this makes the built-in evaluation system a substitute for the judgment Section 1 described — choosing a representative task set and defined scoring criteria is still work only you can do, because only you know what your agent is actually supposed to be good at. What the toolkit removes is the mechanical cost of wiring a harness together correctly, which is exactly the "without wiring up bespoke harnesses" the source material calls out directly.

Consider a customer-support agent, version 1.0, evaluated on a fixed benchmark of 50 support tickets drawn to represent the mix of question types the agent actually receives in production: billing questions, account-access questions, and product how-to questions, roughly a third each.

Evaluation pipeline run — Agent v1.0
Fixed task set: 50 tickets (17 billing, 17 access, 16 how-to)
Scoring criterion: resolved-without-escalation (pass/fail), graded by a written rubric

Billing:    14 / 17 passed  (82%)
Access:     16 / 17 passed  (94%)
How-to:     10 / 16 passed  (63%)

Aggregate: 40 / 50 passed  (80%)

This is a constructed scenario — the exact numbers are illustrative, not measured from a real support system — built to make the mechanics of Section 2 concrete. The 80% aggregate is the number you would report as "agent v1.0's benchmark score," but notice what it hides on its own: how-to questions are the agent's weakest category by a wide margin, at 63% against billing's 82% and access's 94%. A team that only tracked the aggregate would see "80%, reasonably good" and miss that one entire category is dragging the average down. This is exactly the per-task breakdown Section 2 insisted a well-built pipeline preserve — the aggregate is what you report, the breakdown is what tells you where to look next, a thread M3-06 picks up directly when it turns this same kind of per-category breakdown into a targeted fix.

Now suppose the team ships a change intended to improve how-to answers and re-runs the identical fixed task set three weeks later:

Evaluation pipeline run — Agent v1.1 (same 50-ticket fixed task set)
Scoring criterion: identical rubric, unchanged

Billing:    14 / 17 passed  (82%)
Access:     16 / 17 passed  (94%)
How-to:     13 / 16 passed  (81%)

Aggregate: 43 / 50 passed  (86%)

Because the task set and the scoring criterion are both held fixed between the two runs, the 6-point aggregate improvement — and, more specifically, the how-to category's jump from 63% to 81% — can be honestly attributed to the change the team made, rather than to a different, easier set of tickets happening to be sampled the second time. This is the entire payoff of Section 1's "comparable run to run" property, made concrete: without the fixed task set and unchanged scoring, this comparison would be exactly the kind of unattributable result the module's later lesson on targeted optimization exists to warn against.

Objective 3.2 explicitly names comparing agent performance across tasks and datasets, not just tracking one benchmark over time. Consider the same support agent, now evaluated on two distinct task suites in the same evaluation cycle: the 50-ticket support benchmark from Section 4, and a separate 30-item benchmark of multi-step account-remediation tasks that require the agent to look up an account, apply a policy, and take a corrective action across several tool calls.

Same evaluation cycle, two separate fixed task suites

Suite A: Support-ticket resolution (50 tickets, single-turn, pass/fail rubric)
  Aggregate: 43 / 50 passed  (86%)

Suite B: Multi-step account remediation (30 tasks, 3-6 tool calls each, pass/fail rubric)
  Aggregate: 18 / 30 passed  (60%)

A team that only ran Suite A would report "86%, agent is performing well" — a claim that is true of single-turn ticket resolution and silently false of anything requiring multi-step tool coordination. A team that only ran Suite B would report "60%, agent is struggling" — true of multi-step remediation and silently unfair to the agent's genuinely strong single-turn performance. Neither number alone characterizes the agent; only running both suites and reporting them separately does, which is exactly why Objective 3.2 names comparison across tasks and datasets as its own distinct skill rather than treating a single benchmark's score as sufficient. This same principle extends directly to retrieval-heavy agents: for a RAG-grounded agent specifically, evaluation extends further still, into whether retrieved context is relevant to the query and whether the generated answer is actually grounded in that context — RAG-specific quality dimensions that a general task-pass/fail rubric does not capture on its own, and that this course's knowledge-integration material develops from its own angle.

Evaluation and Tuning carries 13% of the NCP-AAI blueprint, the third-largest domain after the tied 15% domains of Agent Architecture and Agent Development, and it pairs tightly with Run, Monitor, and Maintain — the domain that continues this same measurement discipline into live production. [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) states the domain's own scope note directly: "the recurring professional theme is targeted optimization — use measurement (evaluation) and instrumentation (profiling) to decide what to change, rather than guessing." Objective 3.1, implementing evaluation pipelines and task benchmarks, is where that theme opens, because every later objective in this domain — comparing across tasks, tuning the accuracy-latency trade-off, analyzing results for targeted optimization — presumes you already have a trustworthy number in hand, and this lesson is where that number comes from.

Expect the question shape to test the identity statement in Section 1 by description rather than by definition-matching: a scenario describes a team re-running "an evaluation" with a different set of test cases each time, or comparing two scores produced under two different scoring rubrics, and asks whether the comparison is valid. The keyed answer traces back to which of the three requirements — fixed task set, fixed criteria, held-constant comparison basis — the scenario violated. A second recurring shape tests Objective 3.2 directly: a scenario reports one benchmark's score and asks what conclusion is safe to draw about the agent generally, with the keyed answer noting that a single task suite's result does not generalize to task types it never tested.

A repeatable evaluation result is one where running the identical pipeline again, on the identical fixed task set, with the identical scoring criteria, against an unchanged agent, produces the same score — and where running it again after a deliberate agent change produces a score difference that can be honestly attributed to that change rather than to some other variable that also happened to shift. Repeatability is not about the agent always scoring the same; an agent that genuinely improves should score differently. It is about the measurement process holding still, so that when the score does change, you know the change reflects something real about the agent rather than something that drifted in how you measured it.

A single overall pass rate collapses every task category into one number, which hides exactly the information a targeted fix needs: which category of task is actually weak, and by how much. Two agents can share the identical 80% aggregate pass rate while one is uniformly mediocre across every task type and the other is excellent everywhere except one badly failing category — a distinction the aggregate alone cannot express, but the per-task breakdown described in Section 2 makes visible immediately. Objective 3.2's requirement to compare across tasks and datasets exists precisely because an aggregate, taken alone, systematically understates how uneven an agent's real performance usually is.

• An evaluation pipeline needs three things held constant to produce a trustworthy result: a fixed task set, defined scoring criteria, and an unchanged comparison basis — drop any one and the resulting score cannot be honestly compared to another run's.
• A task benchmark is the fixed content the pipeline runs over; the pipeline is the repeatable process that runs and scores it. The two are related but distinct, and a team that changes its benchmark every time it evaluates has lost the fixed-task-set half of the definition.
• The NeMo Agent Toolkit ships a built-in evaluation system specifically so teams do not need to wire up bespoke scoring harnesses from scratch for every workflow.
• An aggregate score hides per-task or per-category variation that a targeted fix later in this module needs; a well-built pipeline preserves that breakdown rather than discarding it.
• Comparing performance across multiple task suites, not just one, is its own named skill (Objective 3.2) — a single benchmark's score does not generalize to task types the benchmark never tested.
• For retrieval-heavy agents, evaluation extends further, into whether retrieved context is relevant and whether the answer is actually grounded in it — a RAG-specific dimension layered on top of the general pipeline concept.

1. A team evaluates an agent on Monday using 40 tasks, then evaluates it again on Friday using a different set of 40 tasks drawn from the same general pool. What is the strongest concern with comparing the two scores?
   • A. Forty tasks is too small a sample regardless of anything else.
   • B. The two runs did not use the identical fixed task set, so any score difference is entangled with which tasks were sampled.
   • C. Evaluations should never be run more than once per week.
   • D. The agent cannot have changed meaningfully between Monday and Friday.
2. What does the NeMo Agent Toolkit's built-in evaluation system primarily provide?
   • A. A replacement for choosing your own task set and scoring criteria.
   • B. A way to score toolkit workflows without building a bespoke evaluation harness from scratch.
   • C. A guarantee that any agent will pass its evaluation.
   • D. A profiler for measuring token counts and latency.
3. An agent scores 90% on a single-turn FAQ benchmark and is described in a report as "performing well overall." What is missing from that conclusion?
   • A. Nothing; a 90% score is sufficient evidence of overall performance.
   • B. The report should have used a stricter pass/fail threshold.
   • C. The score describes only the FAQ task type and does not establish performance on other task types the agent was never benchmarked against.
   • D. Single-turn tasks are inherently invalid for evaluation.
4. Two evaluation runs use the identical fixed task set, but the second run's LLM-judge prompt was rewritten to be stricter between the two runs. Is the resulting comparison valid?
   • A. Yes, because the task set stayed fixed.
   • B. Yes, as long as both runs used an LLM judge rather than a human grader.
   • C. No, because the scoring criteria changed, which breaks the comparison even though the tasks did not.
   • D. No, because LLM-as-judge scoring is never valid.
5. Why does a well-built evaluation pipeline preserve the per-task score breakdown rather than reporting only the aggregate?
   • A. Aggregates are computationally expensive to calculate.
   • B. The breakdown reveals which specific task category is weak, which the aggregate alone hides.
   • C. Regulations require per-task reporting.
   • D. The aggregate score is always inaccurate.
6. What does Objective 3.2, comparing agent performance across tasks and datasets, add beyond Objective 3.1's evaluation pipeline?
   • A. A requirement to use exactly one benchmark for all evaluation.
   • B. A requirement to run the same pipeline over multiple distinct task suites and report them separately, so uneven strength across task types is visible.
   • C. A requirement to always prefer harder benchmarks over easier ones.
   • D. Nothing; 3.1 and 3.2 describe the identical skill.
7. A RAG-grounded agent scores well on a general task-pass/fail benchmark. What does this evaluation not yet establish?
   • A. Whether the retrieved context was relevant and whether the answer was actually grounded in it.
   • B. Whether the agent used any tools at all.
   • C. Whether the benchmark had a fixed task set.
   • D. Nothing; a general pass/fail score fully characterizes RAG quality.
8. A team's evaluation "harness" is a script one engineer wrote and no one else has read or can independently verify. What risk does this most directly create?
   • A. The evaluation will always run too slowly.
   • B. A subtly broken scoring integration could silently miscount or misalign results without anyone noticing.
   • C. The agent will fail every task automatically.
   • D. The task set will change every time the script runs.

Answers

1. B. Comparability requires the identical fixed task set; drawing a different sample from the same pool still breaks the requirement, because the specific tasks compared are not the same ones.
2. B. The toolkit's built-in evaluation system removes the mechanical cost of building a scoring harness; it does not choose your task set or criteria for you, and it is not a profiler.
3. C. A single task suite's score characterizes performance on that suite only; generalizing it to "overall performance" is exactly the failure Objective 3.2 exists to correct.
4. C. Even with the task set fixed, a changed scoring rubric or judge prompt moves the yardstick, which is enough on its own to invalidate a direct comparison between the two runs.
5. B. The aggregate can look acceptable while one category is failing badly; only the per-task or per-category breakdown surfaces that unevenness.
6. B. Objective 3.2 is specifically about running the pipeline over more than one task suite and reporting each separately, not about picking a single "best" benchmark.
7. A. RAG-specific quality — retrieval relevance and groundedness — is a distinct dimension a general pass/fail rubric does not automatically capture, even when the general score is high.
8. B. An unverifiable, unaudited harness is exactly where a scoring bug can hide, producing plausible-looking numbers that do not actually reflect what the agent did.

Building the harness that produces a trustworthy score is only half of Objective 3.1's real job — the other half is knowing what that score can and cannot tell you about why the agent performed the way it did, which is a completely different question from a completely different instrument. M3-02 takes up that distinction directly: evaluation asks how good the outputs are, and a separate discipline called profiling asks where the time and tokens actually went, and conflating the two — optimizing one without checking the other — is the most testable trap in this domain.

Next: M3-02 covers profiling versus evaluation as two different questions, and why optimizing a profiler metric like latency without re-checking the evaluation score you built here can silently degrade quality without anyone noticing until it is measured directly.