Profiling Performance and Reliability Under Distributed Load

Identity statement: distributed system behavior differs from single-node behavior because three specific mechanisms — contention, network latency, and tail effects — only manifest once multiple replicas, multiple machines, and concurrent real traffic are all present simultaneously, none of which a single-node benchmark includes by construction. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): "Distributed behavior differs from single-node behavior: contention, network latency, and tail effects only appear at scale." The same source names the standing trap directly: "Assuming single-node benchmarks predict distributed behavior. Load-test the distributed system before committing to a scaling plan."

It is worth being precise about why a single-node test cannot, structurally, surface these three mechanisms — not because the test was run poorly, but because the conditions that produce them do not exist in a single-node setup at all. Contention is competition for a shared resource — a GPU, a network link, a database connection pool — among multiple concurrent consumers; a single replica running alone, even under a synthetic load generator sending many requests, is still only one consumer of whatever resources it uses on its own machine, so it never experiences another replica's simultaneous demand for the same shared GPU memory, the same shared network bandwidth, or the same shared downstream dependency's connection limit. Network latency between replicas, between a replica and a load balancer, or between a replica and a shared dependency (a database, a vector store, a NIM endpoint it calls) simply does not exist in a single-node test where every component runs on the same machine with no network hop between them at all — a localhost call and a cross-machine call are not the same measurement, and a benchmark that only ever exercised the former has no data at all about the latter. Tail effects — the behavior of the slowest few percent of requests, not the typical request — require enough concurrent requests and enough replicas for a slow outlier to actually occur and to actually matter to an aggregate metric; a single-node test run against a light, sequential load has comparatively few opportunities for the rare slow case to appear, and even when it does appear, there is no fleet-level aggregate percentile for it to distort, because there is only one node's own timing to look at.

L1 — Intuition

Picture a kitchen with one chef, timed cooking one dish start to finish with nothing else happening around them — that is a single-node benchmark. Now picture the same kitchen at dinner service with ten chefs sharing one stove, one walk-in fridge, and one dish-washing station, orders coming in continuously and unevenly, and one chef occasionally slower than the rest for reasons that have nothing to do with their skill on any single dish. The first measurement tells you how fast one dish can be cooked under ideal, uncontested conditions. The second measurement tells you something the first one structurally cannot: how long an order actually takes when ten chefs are competing for the same stove (contention), when an order has to travel from the dining room to the kitchen and back (network latency the single chef's bench test never had to cross), and when the one unusually slow chef occasionally becomes the reason a whole table's order is late even though the other nine chefs finished on time (a tail effect distorting an aggregate outcome).

L2 — Mechanism

Contention shows up mechanically as queueing: when concurrent demand for a shared, finite resource exceeds what that resource can serve immediately, requests wait, and that wait time is additional latency the resource's own raw processing speed never included. A GPU serving one NIM replica's requests one at a time has no queueing to speak of; the same GPU shared across several replicas' worth of concurrent requests, or a single replica receiving far more concurrent requests than its configured batch size and concurrency limit can absorb at once, produces queueing delay that grows nonlinearly as demand approaches the resource's ceiling — a property a single-node, single-request-at-a-time benchmark has no way to expose, because it never generates the concurrent demand that produces the queue in the first place.

Network latency shows up mechanically as the added, non-zero time for a request or a response to cross a network boundary between machines — a load balancer to a replica, a replica to a shared vector store, a replica to a downstream NIM endpoint on a different node — where a single-node test's equivalent call was a local, effectively-zero-latency function call or a loopback network call with none of a real network's variability, packet loss retransmission, or cross-availability-zone hop cost. Even a genuinely fast network link adds latency that was simply absent from a co-located, single-machine measurement, and that latency compounds every time a distributed request's path crosses another network boundary — a multi-step agent turn touching several distributed services can accumulate several such hops in a single user-facing request, none of which a single-node benchmark's architecture ever had to cross.

Tail effects show up mechanically in percentile statistics: the p50 (median) latency describes the typical request, while p95, p99, and higher percentiles describe the slower few percent — and at scale, with enough concurrent requests hitting enough replicas, some small fraction of requests will always be slower than typical, for reasons ranging from a momentary GPU contention spike on one specific replica, to a garbage-collection pause, to a network blip on one specific hop. A single-node benchmark run against light, sequential load rarely generates enough concurrent requests for this small fraction to appear at all, and even when a slow outlier does occur in a single-node test, there is no fleet of concurrently-served requests for that one outlier to sit inside as a percentile — the single-node number is just "sometimes it was slow," not a measured p99 across a real distribution of concurrently served traffic.

L3 — The exam-relevant edge case: why the three mechanisms compound rather than simply add

The edge case worth holding precisely is that contention, network latency, and tail effects do not merely coexist independently at scale — they interact, and the interaction is often worse than any one of them measured in isolation would suggest. A replica under GPU contention runs slower than its unloaded benchmark suggested; that slower replica is now also more likely to be the one still processing a request when a network timeout on a downstream call fires, converting a merely-slow response into an outright failure; and because that one contended, slow replica is disproportionately represented among the slowest few percent of all requests across the fleet, it drags the aggregate p99 latency upward by more than "one replica's slowdown divided across all replicas" would predict, because percentile statistics are sensitive to exactly this kind of concentrated, non-uniform slowness rather than an evenly distributed one. A profiling plan that measures contention, network latency, and tail effects as three separate, isolated line items risks understating how bad the combined, real-world distributed picture actually is — which is precisely why the discipline is to load-test the distributed system as a whole, under realistic concurrent traffic, rather than to measure each mechanism in a clean, isolated test and simply sum the results.

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): "Tools like Nsight Systems (profiling) and DGX Cloud Benchmarking (workload performance) help you measure before you scale." The parenthetical labels in that source sentence are doing real work, and conflating the two tools' jobs is a common, avoidable mistake.

Nsight Systems is a profiler — a tool for looking inside a running system's timeline and seeing exactly where time is being spent at a fine-grained level: which kernel is executing on the GPU at a given moment, how long a specific operation takes, where a pipeline is stalling waiting on something else to finish. Profiling in this sense answers "why is this specific thing slow," at the level of what the hardware and software are actually doing, moment to moment — it is a microscope pointed at one running process or one machine's timeline.

DGX Cloud Benchmarking, by contrast, measures workload performance — how a given deployment performs under a realistic, generated load, producing the kind of aggregate throughput and latency numbers (including the percentile statistics tail effects show up in) that describe the system's behavior from the outside, under the conditions load-testing is meant to simulate. Benchmarking in this sense answers "how does this deployment actually perform under realistic traffic," at the level of the system as a whole responding to load — it is closer to a stopwatch and a traffic generator pointed at the deployed system's external behavior.

The two are complementary, not competing: a benchmark run (DGX Cloud Benchmarking) is what surfaces that p99 latency is unexpectedly high under realistic concurrent load in the first place; a profiler run (Nsight Systems), pointed at one of the specific replicas or GPUs implicated in that slow tail, is what then explains why — a specific kernel taking longer than expected, a specific queueing pattern, a specific stall. Reaching for a profiler before a benchmark has identified where to look wastes the profiler's fine-grained detail on a part of the system that may not be the actual bottleneck; reaching for a benchmark alone, without ever profiling the specific slow component it identifies, leaves the "why" unanswered and the fix undirected.

The reason this table is worth building explicitly is that M3-02's core distinction — evaluation asks how good the outputs are, profiling asks where the time went — extends cleanly into distributed territory, but gains a third question in the process: not just "is it correct" and "where did the time go for one run," but "how does the whole fleet behave under real concurrent load," which is a question neither a correctness evaluation nor a single-run profile can answer on its own. Skipping straight from single-run profiling to a scaling decision, without ever running the distributed workload benchmark, is exactly the gap this domain's named trap describes.

Constructed scenario, illustrative only. A team runs a workload benchmark against their ten-replica, load-balanced deployment from M4-02's worked example, generating realistic concurrent traffic that ramps from 50 to 400 concurrent requests over the test window.

Single-node benchmark (measured earlier, one replica, no concurrent
traffic, no other replica present):
  p50 latency: 850 ms
  p99 latency: 920 ms
  (p99 is barely above p50 -- almost no tail at all)

Distributed load test (ten replicas, load-balanced, ramping to 400
concurrent requests):
  p50 latency: 910 ms   <- close to the single-node number, typical
                            case is only slightly worse
  p95 latency: 1,340 ms <- noticeably higher than p50
  p99 latency: 3,600 ms <- nearly 4x the p50, and 4x the single-node p99

Diagnosis:
  The MEDIAN case degraded only modestly under distributed load --
  contention and network overhead added roughly 60 ms typically.
  The TAIL degraded dramatically -- p99 is 4x the median, which the
  single-node test's near-flat p50/p99 gap never hinted at, because
  the single-node test never had enough concurrent requests or
  enough replicas for a tail this pronounced to occur at all.

Next step: point Nsight Systems at whichever specific replica(s) are
over-represented in the slowest 1% of requests during the load test
window, to find the specific mechanism (GPU contention on a specific
node, a slow downstream network hop, a queueing spike) driving the
p99 number, rather than guessing.

The number worth sitting with is the ratio, not the absolute figures: a p99 four times the median is a materially different reliability story than a p99 barely above the median, and a team that only ever looked at the median (or only ever ran the single-node test, whose own p50/p99 gap was nearly flat) would report this deployment as "basically fine" right up until enough real users hit the unlucky 1% for it to become a visible, repeated complaint. This is precisely the shape of problem M8-01's monitoring dashboards are built to keep surfacing continuously in production, but the point of profiling before scaling is catching it here, in a controlled load test, rather than discovering it live.

Constructed scenario, illustrative only. The same team, investigating the p99 spike above, wants to know whether the tail is being driven primarily by GPU contention on overloaded replicas or by network latency to a shared downstream dependency (a vector store all ten replicas call for retrieval) — two different causes that would call for two different fixes.

Test A -- isolate GPU contention:
  Run the distributed load test again, but replace the ten replicas'
  actual GPU inference calls with a fixed, artificial delay (no real
  GPU work), keeping every other component (network, load balancer,
  vector store) identical.
  Result: p99 drops from 3,600 ms to 1,050 ms.
  Interpretation: a large share of the tail was GPU-contention-driven,
  since removing real GPU inference load removed most of the spike.

Test B -- isolate downstream network latency:
  Run the distributed load test again, restoring real GPU inference,
  but replace the shared vector-store retrieval call with a fixed,
  local, near-zero-latency stub, keeping GPU inference identical.
  Result: p99 drops from 3,600 ms to 3,050 ms.
  Interpretation: the shared vector store's network latency is a much
  smaller contributor to the p99 tail than GPU contention was.

Diagnosis: the tail is primarily a GPU-contention problem on the
replicas themselves, not a downstream-network problem on the shared
retrieval dependency -- pointing the fix toward replica sizing or
batching configuration (a `M7-02`-style throughput-tuning question)
rather than toward the vector store's network path.

The technique this second example demonstrates — substituting a stub for one component at a time and re-measuring — is a controlled way of attributing an aggregate symptom to a specific cause among several plausible candidates, rather than guessing which of contention, network latency, or some third factor is actually responsible. Without this kind of isolation, a team facing a p99 spike has no principled way to choose between "add more replicas," "tune the GPU batching configuration," or "speed up the shared retrieval dependency" — three very different fixes that only one of which, in this constructed example, would have actually addressed the measured problem.

⭐ THE EARNED INSIGHT A single-node benchmark is not a smaller, cheaper version of a distributed load test — it is a measurement of a different system entirely, one with no contention because nothing else shares its resources, no network latency because nothing crosses a network boundary, and no tail because too few concurrent requests exist for a rare slow case to occur and to register in an aggregate. Trusting that measurement to predict scaled behavior is not a matter of degree, extrapolating a small number up to a big one — it is applying a measurement from a system that structurally cannot exhibit the three failure mechanisms that matter most at scale, to a system built specifically to exhibit all three at once.

The tools and disciplines covered above are easy to reach for in the wrong order, or to substitute for one another when neither actually substitutes. The table below is the decision aid worth internalizing before a scaling decision, not after a reliability incident forces the question.

Reading the table's right-hand column as a whole, the recurring theme is scope mismatch: every wrong reach in this table is a case of using a measurement whose scope (one run, one node, a single component) does not match the question actually being asked (the whole fleet's behavior under real concurrent traffic). Matching scope to question, every time, is most of what separates a profiling discipline that catches problems before they ship from one that merely produces numbers nobody can act on correctly.

[GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md): profiling under distributed load is objective 4.3 within Deployment and Scaling, which carries 13% of the NCP-AAI blueprint. The domain's own framing states the trap this lesson is built around as its own explicit warning: "Assuming single-node benchmarks predict distributed behavior. Load-test the distributed system before committing to a scaling plan." That ordering — profile first, commit to a scaling plan second — is itself testable independent of the mechanism: a scenario describing a team that scaled based on single-node numbers alone, and then hit unexpected reliability problems, is describing exactly this trap in narrative form.

How the question tends to be phrased

Expect a direct mechanism-recall item: "why profile a multi-agent system under distributed load before scaling?" with contention, network latency, and tail effects as the keyed answer against distractors claiming distributed behavior always matches single-node behavior, or that profiling is only relevant to training. [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) states this exact self-check item. Expect also a tool-identification item distinguishing Nsight Systems' profiling role from DGX Cloud Benchmarking's workload-performance role, testing whether a candidate can place each tool correctly rather than treating "measurement tooling" as one undifferentiated category.

What the distractors typically look like

The house style favors an appealing but structurally wrong shortcut: treating a single-node benchmark's numbers as sufficient evidence a scaled deployment will behave the same way; describing profiling as a training-only concern with no deployment relevance; and collapsing Nsight Systems and DGX Cloud Benchmarking into interchangeable synonyms for "a performance tool," rather than recognizing their distinct fine-grained-versus-aggregate roles.

A single-node benchmark, by construction, never generates the conditions that produce the three mechanisms this lesson names: it has no other replica competing for the same shared GPU or network resource (so no contention), it typically runs everything co-located on one machine (so no cross-machine network latency), and it rarely generates enough concurrent requests for a rare slow outlier to occur and to matter to an aggregate percentile (so no meaningful tail effect). A number produced under those conditions describes a system that simply does not exist once that same code is one of several replicas serving real, concurrent, distributed traffic — which is why [GROUND TRUTH] (Sources/ncp-aai/domain-4-deployment-scaling.md) names "assuming single-node benchmarks predict distributed behavior" as a standing trap rather than a minor caveat.

Compare the median (p50) latency to a high percentile (p95 or p99) from the same load test. If the median is close to what an unloaded or single-node baseline predicted, but a high percentile is dramatically higher — several times the median, as in this lesson's worked example — the problem is concentrated in the slowest few percent of requests rather than affecting typical requests broadly, which is the signature of a tail effect. If instead the median itself has shifted substantially higher, the problem is a typical-case slowdown (often contention-driven, since contention affects most concurrent requests roughly proportionally) rather than a tail-specific one, and the two different shapes point toward different fixes: a tail-specific problem often points to isolating and fixing whatever occasionally makes one replica or one request unusually slow, while a broad median shift often points to insufficient replica capacity or resource headroom overall.

• Single-node benchmarks cannot predict distributed behavior — contention, network latency, and tail effects are all structurally absent from a single-node test and only appear once real, concurrent, multi-replica traffic exists.
• Contention is queueing for a shared resource; network latency is a real cross-machine hop; tail effects concentrate in high percentiles — three distinct, nameable mechanisms, not one vague "scale is different" phenomenon.
• Nsight Systems profiles fine-grained timing inside one process or machine; DGX Cloud Benchmarking measures aggregate workload performance across the whole distributed system — complementary tools answering different questions, not interchangeable synonyms.
• Always check high percentiles (p95/p99), not just the median — a severe tail-latency problem can hide entirely behind an unremarkable p50.
• Benchmark first to find where a problem is, then profile that specific component to find why — reversing the order wastes fine-grained profiling effort on a possibly-uninvolved component.
• Isolate candidate causes (stub substitution) before committing to a fix — the three mechanisms interact and can be mistaken for one another without deliberate isolation.
• The standing trap is load-testing order: profile the distributed system before committing to a scaling plan, not after discovering a reliability problem in production.

Measuring where headroom and hotspots actually sit under real distributed load is a diagnostic step — it says nothing yet about the deliberate discipline of turning that measurement into a repeatable delivery and governance process, so a fix, once found, actually ships safely and stays auditable. That is where this module goes next: M4-04 covers MLOps and governance — CI/CD, monitoring, and audit as a first-class deployment concern, not something bolted on after the profiling and scaling work here is done.

Next: M4-04 covers MLOps and governance — CI/CD, monitoring, and audit — the repeatable, automated delivery process and the governance controls that make a fix found through profiling like this one safe to ship and traceable afterward.