Profiling vs. Evaluation: Two Different Questions About the Same Agent

Identity statement: evaluation measures output quality against defined criteria; profiling measures resource consumption — time and tokens — against the code path that produced it. Neither one is a more detailed or more advanced version of the other. They are answers to categorically different questions, applied to the same underlying agent run.

[GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) frames the two side by side directly, and the framing is worth reproducing exactly because the exam draws on this precise contrast: evaluation asks "how good are the outputs," answered by benchmarks, LLM-as-judge scoring, or human review — the same mechanisms M3-01 built a harness around. Profiling asks "where do time and tokens go," answered by the toolkit's profiler, which surfaces token counts, timings, and bottlenecks. [VENDOR SPEC] (Sources/ncp-aai/domain-3-evaluation-tuning.md): "The toolkit's profiler surfaces token counts, timings, and bottlenecks — the raw material for tuning." Neither question can stand in for the other. A profiler trace can tell you that a particular tool call consumes 40% of a turn's total latency, and it will tell you nothing whatsoever about whether that tool call's output was correct. An evaluation score can tell you an agent answers 91% of benchmark questions correctly, and it will tell you nothing about which single step inside a correct-scoring turn happened to burn most of the turn's token budget.

Why "the toolkit does both" does not blur this line

The NeMo Agent Toolkit ships both capabilities — its built-in evaluation system from M3-01 and its profiler here — as part of the same "batteries included" bundle, alongside its observability integrations. That the same toolkit ships both is a convenience, not a conceptual merger: shipping two different instruments from the same vendor does not make them the same instrument. A scenario question that describes the toolkit's profiler and asks whether it "evaluates" the agent's output quality is testing precisely this — the correct answer is no, regardless of how convenient it is that both tools live in the same toolkit.

L1 — Intuition

Picture an agent's single turn as a relay race with several runners — a retrieval call, a reasoning step, a tool call, a final generation call — each handing off to the next. A stopwatch at each handoff point tells you exactly how long each runner took and where the baton spent the most time in transit. That is what a profiler does: it times each leg of the relay and tells you which leg is slow. What the stopwatch cannot tell you is whether the team actually won the race — whether the final output crossing the finish line was the right one. That is a completely separate judgment, made by someone watching the finish line, not the stopwatch.

L2 — Mechanism

Concretely, the toolkit's profiler instruments a workflow's execution and reports, per step or per call, the timing (how long that step took) and the token count (how many input and output tokens that step consumed), and aggregates these into a picture of where the bottlenecks are — the steps consuming a disproportionate share of the turn's total time or token budget. This is the "raw material for tuning" the source material names: once you know that, say, a reasoning step before the final answer consumes 60% of a turn's tokens while contributing little to the eventual output's correctness, you have a specific, evidence-backed target for optimization rather than a guess about which of an agent's many steps might be the expensive one.

What the profiler's output does not, and structurally cannot, contain is any judgment about correctness. A profiler trace showing a tool call took 800 milliseconds and consumed 200 tokens is a complete, accurate description of that call's resource cost, and it carries zero information about whether the tool call retrieved the right document, whether the reasoning step that followed drew a valid conclusion from it, or whether the final answer was actually correct. Resource cost and output quality are two independent axes; a step can be fast and wrong, slow and right, fast and right, or slow and wrong, and the profiler alone can only ever place a step on one of those two axes.

L3 — The exam-relevant edge case: a profiler improvement that a quality regression is hiding behind

The edge case the exam cares about most is a change that looks unambiguously good by profiler evidence alone. Suppose a team removes a verification step from an agent's reasoning chain — a step that had the agent double-check a retrieved fact against a second source before answering. The profiler afterward shows a clear win: turn latency drops by 35%, and token consumption drops by nearly as much, because an entire step's worth of model calls simply stopped happening. Read the profiler alone, and this change is a straightforward success story. What the profiler cannot show is that the verification step the team removed was catching a meaningful fraction of retrieval errors before they reached the user, and removing it may have quietly raised the rate at which the agent now answers confidently from a wrong source. Only re-running the evaluation pipeline from M3-01 on the fixed benchmark surfaces that cost — the profiler has no mechanism for detecting it at all, because detecting it was never the question the profiler was built to answer.

⭐ THE EARNED INSIGHT A profiler cannot lie to you, and that is exactly what makes it dangerous used alone: every number it reports about timing and token consumption is real and accurate, which makes a profiler-only success story feel fully verified when it has only been half verified. The half that is missing — whether the faster, cheaper version still produces outputs as good as the slower, more expensive one — is not a smaller or optional half. It is the half evaluation exists to check, and skipping it is how a genuine performance win quietly becomes a genuine quality loss that nobody notices until a user does.

[GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) names this trap in exactly these terms: "Optimizing a profiler metric (e.g., latency) without re-checking evaluation can silently degrade quality — and vice versa." Both directions of that warning matter, and it is worth separating them, because the exam tests both and they are not symmetric in how they usually show up in practice.

Optimizing latency/tokens without re-checking evaluation. This is the more common direction in practice, because profiler-driven changes are seductive — the profiler hands you a specific, concrete, measurable win (35% faster, 30% fewer tokens) that is easy to report and easy to feel good about, while the quality cost, if any, is invisible unless you go looking for it with the other instrument. The fix the trap implies is procedural, not clever: any change made because the profiler flagged a bottleneck gets the fixed evaluation benchmark re-run against it before anyone calls the change done.

Optimizing evaluation score without re-checking profiling. The reverse direction is less commonly the trap's headline example but is symmetric in structure: a team chasing a higher evaluation score adds a self-verification loop, a second retrieval pass, or a longer chain-of-thought, and the score does improve — while latency and token cost silently balloon in a way nobody measured until a user complained about a slow response or a bill arrived larger than expected. M3-04's accuracy-versus-latency trade-off develops this second direction in full; the point to hold onto here is narrower and more mechanical: whichever instrument you optimized against, the other instrument is the one that tells you what that optimization cost you on the axis you weren't looking at.

Why "silently" is the operative word

Both halves of the trap share the word "silently" for a specific reason: neither instrument raises an alarm about the dimension it does not measure. A profiler will never report "quality dropped," because quality is not a quantity it tracks. An evaluation pipeline will never report "latency increased," because latency is not a quantity it scores. The silence is not a bug in either tool; it is a direct consequence of each tool doing exactly the one job it was built to do and no other. The professional discipline this section is naming is entirely about compensating for that silence deliberately, by always running the instrument that would otherwise stay quiet.

Return to the support agent from M3-01's worked example, evaluated on the same fixed 50-ticket benchmark, with an 86% pass rate at version 1.1. A profiler trace on that same version surfaces a specific bottleneck.

Profiler trace — Agent v1.1, averaged over the 50-ticket benchmark
Step                          Avg. time (ms)   Avg. tokens   Share of turn
Retrieval call                    180              90            8%
Draft-answer generation           620              340           28%
Self-critique pass                980              510           44%
Final-answer generation           450              260           20%
                                 -----            -----
Total per turn                  2,230            1,200          100%

Bottleneck flagged: self-critique pass — 44% of latency and tokens,
the single largest share of the turn.

This is a constructed scenario, built to make the profiler-versus-evaluation contrast concrete rather than a measured result from a real system. The self-critique pass — a step where the agent reviews its own draft answer before finalizing it — is clearly the profiler's flagged bottleneck, and removing it is the obvious profiler-suggested optimization: cut the single most expensive step, and the turn should get substantially faster and cheaper.

Agent v1.2 — self-critique pass removed
Profiler result: total per turn drops from 2,230ms/1,200 tokens
                 to 1,250ms/690 tokens (~44% faster, ~42% fewer tokens)

Evaluation result (identical fixed 50-ticket benchmark, unchanged rubric):
Billing:    14 / 17 passed  (82%)   [unchanged from v1.1]
Access:     16 / 17 passed  (94%)   [unchanged from v1.1]
How-to:      9 / 16 passed  (56%)   [down from 81% in v1.1]

Aggregate: 39 / 50 passed  (78%)    [down from 86% in v1.1]

The profiler's numbers are unambiguously good: v1.2 is dramatically faster and cheaper than v1.1, and if latency and token cost were the only two numbers anyone looked at, v1.2 would be reported as a clean win. Re-running the fixed evaluation benchmark tells a different story: the aggregate pass rate dropped 8 points, concentrated almost entirely in the how-to category, where the self-critique pass had apparently been catching and correcting a meaningful share of first-draft mistakes on the agent's hardest task type. Removing the bottleneck the profiler flagged did exactly what the profiler predicted on the axis it measures, and exactly what the trap in Section 3 warns about on the axis it does not. The two instruments, read together, tell the complete story that either one alone would have missed half of.

Section 3 named a second, less commonly discussed direction the same trap runs: an evaluation-driven change that quietly costs latency and tokens nobody measured. Consider the same v1.1 agent, this time with a change motivated purely by the evaluation side — a team notices the how-to category is the weakest at 81% and adds a second retrieval pass specifically for how-to questions, re-querying the knowledge base with a reformulated question whenever the first retrieval's confidence score is low.

Agent v1.3 — second-pass retrieval added for low-confidence how-to retrievals

Evaluation result (identical fixed 50-ticket benchmark, unchanged rubric):
Billing:    14 / 17 passed  (82%)   [unchanged from v1.1]
Access:     16 / 17 passed  (94%)   [unchanged from v1.1]
How-to:     14 / 16 passed  (88%)   [up from 81% in v1.1]

Aggregate: 44 / 50 passed  (88%)    [up from 86% in v1.1]

Read on its own, this is a clean win: the aggregate pass rate improved, and specifically the weakest category improved the most, exactly where the team was targeting the fix. A team that stopped here — evaluation improved, ship it — would have checked only one of the two instruments, which is the same partial verification Section 4's example warned against, just running in the opposite direction.

Profiler trace — Agent v1.3, averaged over the 50-ticket benchmark, how-to tasks only
                                v1.1 (single retrieval)   v1.3 (second-pass added)
Avg. time per how-to turn            1,850ms                   3,400ms   (+84%)
Avg. tokens per how-to turn            980                      1,720   (+76%)

This is a constructed scenario, illustrative rather than measured, built to make the reverse direction of Section 3's trap concrete. The second retrieval pass nearly doubled both latency and token cost specifically on how-to questions — the exact category the fix targeted — and nothing about the 88% evaluation score would have surfaced that cost on its own, because evaluation was never built to measure it. Whether this trade is worth making depends entirely on the product's latency and cost requirements, which is M3-04's subject directly; the point this lesson is making is narrower: the team cannot make that call responsibly without first running the profiler, because the evaluation score alone gives no indication the cost even exists.

A profiler-flagged bottleneck and an evaluation-confirmed quality drop, discovered together as in Section 4's worked example, are not a contradiction to resolve — they are the two halves of a single, complete measurement, and the correct response is to treat the change as not yet safe to ship rather than to pick whichever number looks more favorable. The table below sketches the space of outcomes when a profiler-guided change is checked against evaluation, and what each combination implies.

Only the first row is an unambiguous win. The second row — Section 4's scenario — is the one this lesson exists to make sure you catch, because it is the row a profiler-only view cannot distinguish from the first.

Evaluation and Tuning carries 13% of the NCP-AAI blueprint, and [GROUND TRUTH] (Sources/ncp-aai/domain-3-evaluation-tuning.md) names the profiling-versus-evaluation distinction directly as "the most testable distinction in the domain" — a rare degree of explicitness in how the source material flags its own most important trap. The domain's scope note frames the professional theme running through every lesson here as targeted optimization: measurement before guessing, and knowing which of two measurement instruments answers which question is the precondition for that discipline actually working.

Expect the question shape to present a scenario where a team makes a change based on one instrument's evidence and describe the outcome only in that instrument's terms — latency dropped, or accuracy rose — then ask what conclusion is safe to draw, or what the team should check next. The keyed answer is almost always "re-check the other instrument before concluding anything," because the scenario as described has deliberately withheld exactly the information (the other axis) needed to know whether the change was actually a net win. A second common shape names the profiler's specific outputs — token counts, timings, bottlenecks — and asks whether the profiler measures output quality; the keyed answer is no, and the distractors typically offer the profiler as measuring "how good the agent is" in some paraphrased form, testing whether you have actually internalized which instrument answers which question rather than just memorized that two instruments exist.

The toolkit's profiler measures resource consumption attributed to specific steps in an agent's execution: how long each step took (timings), how many tokens each step consumed, and which steps, taken together, account for a disproportionate share of a turn's total time or token budget (bottlenecks). It does not score, grade, or otherwise judge whether the outputs those steps produced were correct, complete, or high-quality in any sense — that judgment is evaluation's job, performed by a separate instrument built around benchmarks, LLM-as-judge scoring, or human review, not by the profiler.

Because "better" on the profiler's numbers only ever answers half the question a real optimization decision needs answered. The profiler can confirm a change made an agent faster or cheaper with complete accuracy, and it has no mechanism at all for detecting whether that same change also made the agent's outputs worse — that is a different axis, measured by a different instrument. Section 4's worked example shows exactly this: a 44%-faster, 42%-cheaper change that the profiler alone would call an unambiguous win was, on re-evaluation, an 8-point quality regression concentrated in the agent's hardest task category. Calling a profiler-only result a success without checking evaluation is declaring victory on one axis while leaving the other axis completely unmeasured.

• Evaluation and profiling answer categorically different questions: how good are the outputs, versus where did the time and tokens go. Neither substitutes for the other, and the source material names this as the domain's single most testable distinction.
• The NeMo Agent Toolkit's profiler surfaces token counts, timings, and bottlenecks — real, accurate resource-cost data that carries zero information about output correctness or quality.
• The named trap runs in both directions: optimizing latency/tokens without re-checking evaluation, and optimizing evaluation score without re-checking profiling, can each silently degrade the dimension you were not measuring.
• A profiler-flagged bottleneck that, once removed, makes the profiler's numbers look better is not yet a confirmed win — it is a hypothesis that the fixed evaluation benchmark from M3-01 has to confirm or refute before the change ships.
• When profiling and evaluation disagree after a change — faster but worse, say — treat the change as not yet safe to ship, not as a result to be resolved by picking the more flattering number.
• Both instruments shipping together in the same toolkit as part of its "batteries included" bundle is a convenience, not evidence that they measure the same thing.

1. A profiler trace shows a tool call consuming 500ms and 300 tokens. What can you conclude about that tool call's output?
   • A. It is very likely correct, since it completed quickly.
   • B. Nothing about correctness — the profiler measures resource cost, not output quality.
   • C. It is very likely incorrect, since 500ms is slow.
   • D. It must have failed, since profiling only runs on failed steps.
2. A team removes a step the profiler flagged as the largest bottleneck, and latency drops sharply. What should happen before the change ships?
   • A. Nothing further; a profiler-confirmed latency win is sufficient on its own.
   • B. Re-run the fixed evaluation benchmark to check whether removing that step cost any quality.
   • C. Re-run the profiler a second time to confirm the same result.
   • D. Announce the win and move to the next bottleneck immediately.
3. An evaluation score improves after a team adds a second retrieval pass. What has this change not yet demonstrated?
   • A. Whether the retrieval pass used the correct embedding model.
   • B. Whether the added latency and token cost are acceptable, since evaluation does not measure either.
   • C. Whether the evaluation benchmark was fixed.
   • D. Whether the agent has any tools at all.
4. Which pair of instruments, read together, would have caught the quality regression in Section 4's worked example that the profiler alone missed?
   • A. Two separate profiler runs on the same version.
   • B. The profiler and the fixed evaluation benchmark from M3-01, run before and after the change.
   • C. A single evaluation run with no profiler at all.
   • D. Increasing the benchmark's sample size without re-running evaluation.
5. Why does the source material call optimizing latency without re-checking evaluation a "silent" risk?
   • A. Because profilers do not print any output.
   • B. Because neither instrument raises an alarm about the dimension it does not measure — a profiler will not report a quality drop, and an evaluation pipeline will not report a latency increase.
   • C. Because evaluation always takes longer to run than profiling.
   • D. Because the toolkit disables logging during profiler-guided changes.
6. Two versions of an agent are compared: version A is faster with an unchanged evaluation score; version B is slower with an improved evaluation score. What does this tell you about which version to ship?
   • A. Always ship the faster version.
   • B. Always ship the version with the higher evaluation score.
   • C. Neither is automatically correct; the choice depends on the product's accuracy-versus-latency requirements, the subject of M3-04.
   • D. Ship whichever version was tested first.

Answers

1. B. Timing and token counts describe resource cost only; the profiler carries no signal about whether the call's output was correct.
2. B. A profiler-confirmed win only verifies the resource-cost axis; the evaluation benchmark is the instrument that verifies the change did not cost quality.
3. B. Evaluation scores quality, not cost — an improved score says nothing about whether the added retrieval pass increased latency or token spend, which only the profiler would show.
4. B. Running both instruments before and after the change is what would have surfaced both the latency win and the quality cost; either instrument alone shows only one axis.
5. B. Each instrument does exactly its own job and no other, so the dimension it does not measure changes without any warning from that instrument.
6. C. Once profiling and evaluation move in opposite directions, choosing between them is a product decision about the acceptable trade-off, not a fact either instrument can settle on its own.

Knowing that a bottleneck exists and that removing it cost quality, as Section 4's worked example showed, still leaves open the question of how you actually saw, step by step, what happened inside that self-critique pass to understand why removing it hurt the how-to category specifically rather than some other category. Aggregate profiler numbers and an aggregate evaluation score both tell you that something changed; neither one, by itself, shows you the individual run-by-run trace that explains why.

Next: M3-03 covers observability for evaluation — the step-level tracing that platforms like Phoenix, Weave, Langfuse, and OpenTelemetry provide, which is the raw material this lesson's aggregate profiler and evaluation numbers both depend on when you need to trace a single run step by step rather than read only the summary statistics.