Profiling and Troubleshooting GPU Bottlenecks with Nsight

Identity statement: profiling is the diagnostic step that determines which of a training or inference run's possible bottlenecks is actually binding — kernel occupancy, memory-bound versus compute-bound execution, or a CUDA memory allocation issue — so that the fix selected afterward addresses a measured cause rather than an assumed one.

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): "Nsight profiling tools locate kernel and memory bottlenecks: occupancy, memory-bound vs compute-bound kernels, and CUDA memory allocation issues," and, as the domain's own named trap, "guessing at bottlenecks wastes time" — objective 7.4 is explicitly about identifying and fixing bottlenecks with CUDA profiling, not about reciting what any single lever does in isolation.

The reason this ordering matters is not merely procedural tidiness. Every one of this module's five levers has a real, nonzero cost to apply and a real chance of doing nothing for a symptom it was never built to address. Switching from FP16 to BF16 to fix an underflow problem that was never actually happening wastes an engineering cycle and changes nothing about a run that was actually memory-bandwidth-limited the whole time. Adding tensor parallelism to a model whose real problem is an unbalanced data loader starving the GPU of work wastes GPU-hours reconfiguring a distributed topology that was never the bottleneck. Profiling first is what converts "try things and see what helps" into "measure, diagnose, then apply exactly one lever with a stated reason to expect it will work" — a categorically faster path to a fixed run, and the one the exam's objective 7.4 tests directly.

Identity statement: occupancy measures how much of a GPU's available parallel execution capacity — the number of concurrently resident, actively scheduled threads relative to the hardware's maximum — a given kernel is actually using while it runs.

A GPU's throughput advantage over a CPU comes from running enormous numbers of threads concurrently, hiding the latency of any single operation behind the sheer volume of others proceeding at the same time. A kernel with low occupancy is not using that capacity fully: too few threads are resident and schedulable at once, so the GPU's hardware sits partially idle even while the kernel is nominally "running." Low occupancy has several possible causes worth distinguishing rather than treating as one undifferentiated "the GPU is slow" symptom — a kernel configured with too few threads per block or too few blocks for the problem size, a kernel whose per-thread resource usage (registers, shared memory) is high enough to limit how many threads the hardware can schedule concurrently, or a kernel whose launch configuration simply does not match the problem it is solving. Nsight's profiling output surfaces occupancy directly, as a measured percentage of a kernel's actual resident-thread count against the hardware's theoretical maximum for that kernel's resource profile, which is the number that tells you whether "make this kernel use more of the GPU's parallelism" is even a relevant fix to reach for, as distinct from a fix aimed at what that kernel is fundamentally limited by once it is running.

Identity statement: a compute-bound kernel is limited by how fast the GPU's arithmetic units can perform the operations the kernel requires; a memory-bound kernel is limited by how fast data can move between memory and those arithmetic units — and a profiler distinguishes the two by comparing a kernel's measured arithmetic throughput and memory-bandwidth utilization against the hardware's respective ceilings for each.

L1 — Intuition: two different things to run out of

Every kernel needs both arithmetic (multiplications, additions, and so on) and data (the operands those arithmetic operations consume, and the results they produce) to complete its work. A kernel can be limited by either resource, and the limiting resource determines what kind of fix actually helps. A kernel that is compute-bound is one where the arithmetic units are the scarce resource — they are running close to their maximum throughput, and the kernel would finish faster only if it needed fewer or faster arithmetic operations. A kernel that is memory-bound is one where the arithmetic units are, comparatively, sitting idle waiting for data to arrive from or depart to memory — the bottleneck is bandwidth, not arithmetic throughput, and giving the kernel faster arithmetic units would not speed it up at all, because arithmetic was never what it was waiting on.

L2 — Mechanism: reading the profiler's own comparison

A profiler like Nsight measures, for a given kernel, its achieved arithmetic throughput (commonly expressed relative to the hardware's peak FLOP rate) and its achieved memory-bandwidth utilization (relative to the hardware's peak bandwidth), and the ratio between how close each of those two measurements sits to its respective ceiling is the signal that classifies the kernel. A kernel running near its hardware's peak arithmetic throughput while its memory-bandwidth utilization sits well below its own ceiling is compute-bound: the arithmetic units are the saturated resource. A kernel running near its hardware's peak memory bandwidth while its arithmetic-throughput utilization sits well below its own ceiling is memory-bound: the data movement is the saturated resource, and the arithmetic units are waiting on it. This comparison — not a single number in isolation, but the relative distance of each measurement from its own ceiling — is what a roofline-style profiling view is built to show directly, and it is the single most load-bearing read a profiler produces for deciding which of this module's levers is even relevant.

L3 — Why the wrong lever for the wrong bound changes nothing

A concrete instance of the trap this whole lesson exists to prevent: mixed precision, this module's M7-04, delivers a genuine speedup for a compute-bound kernel, because Tensor Cores directly increase arithmetic throughput on 16-bit inputs — precisely the scarce resource a compute-bound kernel is waiting on. That same switch to a 16-bit format delivers a much smaller, and sometimes negligible, speedup for a kernel that is memory-bound, because halving the bytes per value does reduce the volume of data that must move (a genuine, if secondary, benefit), but it does nothing to increase arithmetic throughput, which was never the memory-bound kernel's limiting resource in the first place. A team that profiles a memory-bound kernel, misreads it as a compute problem, and reaches for mixed precision purely for its throughput story will be disappointed by how little the change actually helps — not because the technique is broken, but because it was aimed at the wrong bound.

⭐ THE EARNED INSIGHT

"Compute-bound" and "memory-bound" are not properties of a model or a training job in the abstract — they are properties of a specific kernel, measured under specific conditions, and a single training step can contain kernels of both kinds sitting right next to each other. A large, wide matrix multiplication is frequently compute-bound; a kernel that gathers scattered values from memory with comparatively little arithmetic per byte moved is frequently memory-bound; and a real profile of a real training step almost always shows a mix, not a single verdict for the whole run. The discipline this lesson teaches is not "learn whether my model is compute-bound or memory-bound" as a one-time fact — it is "profile the specific kernel that dominates wall-clock time, and read its specific bound," because that is the granularity at which the classification is actually true and actually actionable.

Occupancy and compute-versus-memory-bound classification both describe a kernel that is running, just running inefficiently along one axis or another. A separate class of problem Nsight surfaces is one where memory allocation itself is the failure mode — a run that crashes with an out-of-memory error, or one that runs but suffers from allocator fragmentation, where memory nominally freed by one part of the program remains unavailable to satisfy a new allocation because the memory allocator's cached blocks do not match the shape of the new request. [GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md) names "CUDA memory allocation issues" as one of the three specific things Nsight's tools locate, distinct from occupancy and from the compute/memory-bound distinction — a run can have perfectly healthy occupancy and a well-understood compute/memory bound on its kernels and still fail because of how memory is being requested, held, and released across the program's lifetime, which is a lifecycle problem rather than a kernel-execution-efficiency problem. Recognizing this as a third, separate category — not a variant of "the GPU is too slow" but a distinct "the GPU ran out of usable memory, or memory management itself is the overhead" — is part of what makes Nsight's diagnostic value broader than a single occupancy number or a single bound classification.

Take a constructed profiling readout and walk it to a diagnosis and a specific lever, in the shape the exam's scenario questions favor. Constructed scenario — every figure below is illustrative, authored to isolate the diagnostic reasoning, not measured from a real profiling run.

Profiled training step, dominant kernel: the feed-forward matrix
multiplication inside each transformer block.

Nsight readout (illustrative):
  Achieved arithmetic throughput:   88% of this GPU's peak FLOP rate
  Achieved memory-bandwidth usage:  31% of this GPU's peak bandwidth
  Occupancy:                        76% (healthy, not the limiting factor)
  CUDA memory errors:               none reported

Read the readout in order. Occupancy at 76% with no reported memory errors rules out both section 2's and section 4's failure modes as the binding constraint here — the kernel is well-scheduled and memory allocation is not the problem. The remaining comparison is arithmetic throughput against memory-bandwidth usage: 88% of peak arithmetic throughput against only 31% of peak memory bandwidth is the signature section 3 defines as compute-bound — the arithmetic units are the resource close to saturated, and memory bandwidth has considerable headroom left unused. Given that diagnosis, the lever this module actually predicts as effective is mixed precision (M7-04): a compute-bound kernel is exactly the case where Tensor Core throughput on 16-bit inputs delivers its full benefit, because arithmetic throughput was the scarce resource this kernel was actually waiting on. Reaching instead for a change that primarily addresses memory bandwidth or memory capacity — a different 16-bit-format choice made purely for its byte-count savings, or memory sharding via M7-03 — would not address this kernel's actual bottleneck, because bandwidth was never the constraint the profile identified.

Take a second profiled step with a different signature, to show the same workflow producing a different, equally specific answer. Constructed scenario.

Profiled training step, dominant kernel: an attention computation
over a very long sequence.

Nsight readout (illustrative):
  Achieved arithmetic throughput:   22% of this GPU's peak FLOP rate
  Achieved memory-bandwidth usage:  91% of this GPU's peak bandwidth
  Occupancy:                        68% (adequate)
  CUDA memory errors:               none reported, but peak memory
                                     usage is close to the device limit

Here the comparison inverts: 91% of peak memory bandwidth against only 22% of peak arithmetic throughput is the signature of a memory-bound kernel — the arithmetic units are sitting comparatively idle, waiting on data movement, which mixed precision's Tensor Core throughput benefit would do little to relieve, because arithmetic was never the scarce resource here. The additional detail that peak memory usage is close to the device limit points toward this module's memory-oriented levers rather than its throughput-oriented one: if the sequence length itself, rather than optimizer-state redundancy, is driving that memory pressure, M7-01's context parallelism is the specific fit, since it splits the sequence dimension across all layers regardless of tensor-parallel configuration — precisely the lever aimed at a long-sequence memory constraint. If the profile instead showed the memory pressure concentrated in optimizer state rather than activations, M7-03's distributed optimizer sharding would be the better-targeted fix. The profile's specific readout — which term is actually near its ceiling — is what decides between those two memory-oriented levers, rather than a general impression that "memory is tight" pointing vaguely at either.

The complete discipline this lesson teaches is a loop, not a single measurement: profile the run, identify which kernel dominates wall-clock time, classify that kernel's bound (occupancy, compute/memory-bound, or a memory-allocation issue), select the one lever from M7-01 through M7-05 that this module's earlier lessons establish as the correct fit for that specific bound, apply it, and then profile again to confirm the change produced the expected shift in the readout. Skipping the final re-profile step converts a measured, evidence-based fix back into an unverified guess about whether the change actually worked — a training run that "feels faster" without a re-measured profile is exactly the kind of unverifiable claim this domain is built to discourage. A successful fix for a compute-bound kernel should show arithmetic-throughput utilization dropping toward memory-bandwidth utilization (both nearer their respective ceilings, or the kernel now genuinely faster in wall-clock terms) on re-profile; a successful fix for a memory-bound kernel's sequence-length pressure should show peak memory usage falling and, ideally, memory-bandwidth utilization easing as well. If the re-profile does not show the expected shift, the original diagnosis was wrong, or a second bottleneck was hiding behind the first one the profile initially surfaced — in either case, the loop runs again rather than declaring victory on faith.

This "second bottleneck hiding behind the first" possibility deserves its own emphasis, because it is a common way a single profile-fix-done cycle produces a disappointing result even when the diagnosis and the lever were both correct at the time. Relieving the dominant bottleneck a first profile identifies frequently shifts wall-clock time onto whatever kernel was the second-largest contributor, which the first profile's readout may not have highlighted at all if the original bottleneck was dominating the picture. A team that fixes a compute-bound feed-forward kernel with mixed precision and then re-profiles may discover that a previously minor memory-bound attention kernel is now the new largest contributor to wall-clock time, simply because the first bottleneck no longer masks it. This is not a sign the original fix failed — the compute-bound kernel genuinely got faster — it is a sign that the loop in this section is a loop for a reason, and a single pass through it should not be assumed to be the last one a real optimization project needs.

Domain 7, GPU Acceleration and Optimization, is 14% of the NCP-GENL blueprint, the second-largest domain behind Model Optimization's 17%, and [GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md) states these two domains together account for 31% of the exam — explicitly named as "where to invest" study effort. Objective 7.4 is specifically about identifying and fixing bottlenecks with CUDA profiling, and the domain's own scope note frames the whole domain's objectives as asking candidates to configure multi-GPU training and fix bottlenecks — a framing that makes profiling the connective tissue between every other lesson in this module: M7-01 through M7-05 each supply one lever, and this lesson supplies the discipline that tells you which lever a real, measured symptom actually calls for.

How the question tends to be phrased

Expect a scenario presenting profiler-style readouts — arithmetic throughput, memory-bandwidth utilization, occupancy, or a reported memory error — and asking which of several named techniques (drawn from this module's earlier lessons) is the correct fix, in the shape of sections 5 and 6's worked examples. Expect a companion shape testing the ordering principle directly: a scenario describing a team changing batch size, precision, or parallelism configuration before profiling, asking what they should have done first, keyed to profiling. And expect definitional items distinguishing compute-bound from memory-bound kernels, or asking what Nsight specifically surfaces, keyed to the three named categories in section 1's ground-truth citation: occupancy, the compute/memory-bound distinction, and CUDA memory allocation issues.

What the distractors typically look like

Expect a lever from earlier in the module — mixed precision, a specific parallelism family, gradient accumulation — offered as the fix for a bottleneck whose profiled signature actually points at a different one, testing whether the reader matched the lever to the measured bound or merely to a vaguely-plausible-sounding association. Expect "increase batch size" or "add more GPUs" offered as a generic fix without reference to what the profile actually showed, which is a real intervention attached to no stated diagnosis at all — exactly the guessing this domain's own trap warns against. And expect occupancy, the compute/memory-bound distinction, and CUDA memory issues presented as though they were the same underlying problem measured three different ways, when they are in fact three distinct categories of thing that can independently go wrong.

Yes, and this is the common case rather than the exception, which is exactly why section 3's earned insight insists on profiling the specific dominant kernel rather than characterizing an entire run with one label. A single transformer block contains several distinct kernels — the query/key/value projections, the attention score computation, the feed-forward matrices, normalization operations — and each one can sit at a different point on the compute-versus-memory-bandwidth spectrum depending on its own arithmetic-to-data ratio. A large feed-forward matrix multiplication, with a great deal of arithmetic performed per byte of data moved, is frequently compute-bound. A normalization operation, which performs comparatively little arithmetic on each value it touches, is frequently memory-bound, because moving the data dominates the small amount of arithmetic applied to it. A profiler's per-kernel breakdown is what makes this visible: rather than reporting one aggregate number for the whole training step, it reports each kernel's own throughput and bandwidth utilization, so the engineer can see that, say, 70% of wall-clock time in a step is spent in a handful of compute-bound matrix multiplications while the remaining 30% is spent in memory-bound normalization and elementwise operations — and can then prioritize whichever lever addresses the larger share of that wall-clock time first, rather than optimizing a kernel that was only ever a small fraction of the total.

No, and this is a common misreading worth correcting directly. Occupancy answers a narrow question — is the GPU's available thread-level parallelism being used — and a kernel can have excellent occupancy while still being severely memory-bound, severely compute-bound relative to a better algorithm, or otherwise inefficient in ways occupancy does not measure at all. A kernel running at 95% occupancy that is memory-bound is still limited by memory bandwidth; high occupancy tells you the hardware's scheduling capacity is being used well, not that the kernel's underlying algorithm or its memory-access pattern is optimal. This is precisely why section 1's three named categories — occupancy, compute/memory-bound classification, and CUDA memory allocation issues — have to be read together rather than treated as redundant signals of the same underlying health: a kernel can score well on one and poorly on another, and the correct fix depends on which specific category is actually the problem, not on a single combined "is this kernel healthy" score that does not exist in a profiler's output.

• Profile before touching batch size, precision, or parallelism configuration. Guessing at a bottleneck wastes time; Nsight measures where it actually is.
• Nsight surfaces three distinct categories: occupancy, compute-bound-versus-memory-bound classification, and CUDA memory allocation issues — treat them as separate diagnoses, not interchangeable symptoms of "the GPU is slow."
• Compute-bound and memory-bound are properties of a specific kernel, measured by comparing its arithmetic-throughput and memory-bandwidth utilization each against its own hardware ceiling — not a single verdict for an entire training run.
• Match the lever to the measured bound. Mixed precision (M7-04) targets compute-bound kernels; context parallelism (M7-01) or optimizer-state sharding (M7-03) target different flavors of memory-bound symptoms; gradient accumulation (M7-05) targets a batch-size-under-a-memory-ceiling problem specifically.
• The workflow is a loop, not a single measurement: profile, diagnose, apply one lever, re-profile to confirm the expected shift actually happened.
• This lesson is the domain's synthesis, not a sixth independent lever — M7-01 through M7-05 supply the fixes, and profiling supplies the discipline for choosing correctly among them.
• Domain 7 is 14% of the NCP-GENL blueprint, second only to Model Optimization's 17%; together they are 31% of the exam, and objective 7.4's bottleneck-fixing focus is exactly where this module's five levers and this lesson's diagnostic habit meet.

This module has covered the full arc of getting a model trained efficiently on the hardware available: which parallelism axis to reach for, how to shard memory without inventing a new parallelism axis, which numeric format to compute in and what each one risks, how to reach a target batch size under a memory ceiling, and now, how to verify any of those choices against a measured profile rather than a guess. None of that yet addresses what happens once a trained, optimized model needs to actually serve requests in production — a different set of constraints, tools, and tradeoffs that the next module picks up. Next: Domain 8, Model Deployment, begins with how NVIDIA's serving stack — Dynamo-Triton and NIM among them — takes a model that training and GPU-acceleration work has already made efficient, and turns it into a running service.