Pruning and Structured 2:4 Sparsity for LLM Inference

Identity statement: pruning deletes weights, neurons, or entire structural units — channels, filters, attention heads — that a trained network's output barely depends on, shrinking the network's footprint and the compute a forward pass costs while holding onto nearly all of the accuracy the full network had.

When it matters: whenever a scenario describes shrinking or speeding up an already-trained checkpoint by removing parts of its existing structure, as opposed to re-encoding the numeric precision those parts are stored in (quantization, M4-01) or growing an entirely separate, smaller network from scratch guided by a teacher (distillation, M4-02).

Most converged networks finish training holding onto capacity they never really needed for the task at hand: a meaningful fraction of weights end up with a magnitude close to zero, or with a measurable contribution to the output that is negligible for the inputs the model actually sees in production. Pruning's job is finding and deleting exactly those weights. The simplest selection rule ranks every weight by absolute value and deletes the smallest ones, but low magnitude is a proxy, not the underlying property being sought — what actually justifies deleting a weight is that the output barely moves without it, and magnitude merely correlates with that rather than guaranteeing it. Sharper selection criteria measure a weight's real, observed effect on the output directly rather than relying on magnitude as a stand-in, though magnitude-based ranking stays the default baseline because it requires no extra forward passes to compute.

Deleting a weight the model trained alongside changes the arithmetic every surviving neighbor now participates in — those neighbors settled into their final values on the assumption that the deleted weight's small contribution would still be there on every forward pass, and once it is gone that assumption is simply wrong. A brief resumed-training pass at a lowered learning rate on the pruned network gives the survivors a chance to re-settle around the gap, usually recovering most of whatever accuracy the deletion cost. Calling the job finished the moment weights are deleted, with no such recovery pass, is a frequent and avoidable way a pruning result underdelivers relative to what the same deletion could have achieved with one more training step attached.

L1 — Intuition: emptying seats scattered across a stadium versus closing whole sections

Picture a stadium where every seat corresponds to one weight. Unstructured pruning walks the entire stadium seat by seat and removes whoever is judged least essential, wherever that person happens to sit — row 4, seat 12 here, row 88, seat 3 there, with no pattern to where the empty seats end up. The stadium ends up genuinely less crowded, but every row and section is still standing, still staffed, still operated exactly as before — nothing about running the venue gets any simpler just because attendance dropped in an irregular, spread-out way.

L2 — Mechanism: a matrix multiply has no built-in way to notice an empty seat

Concretely, an importance score is computed per weight, and whichever weights score lowest get set to exactly zero, with no constraint on where in the matrix those zeroed positions land. The matrix's dimensions never change — the same number of rows and columns exist after pruning as before — only some fraction of the entries inside it are now zero rather than some small nonzero value. Because nothing constrains where those zeros fall, this approach can push the zero fraction higher than a pattern-constrained method typically reaches, simply by having complete freedom to zero whichever entry scores lowest regardless of its neighbors.

That freedom is also exactly what breaks the hardware story. A standard dense matrix-multiply kernel walks every position in the matrix and multiplies it, one at a time or in fixed-size parallel batches — it has no logic built in to notice "this particular entry happens to be zero, skip it," so a zero costs the same cycle a nonzero value would have cost. [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "Unstructured pruning saves parameters but may not speed up dense hardware." Turning the zero fraction into an actual latency win needs hardware or software purpose-built to detect and bypass zeros wherever they happen to fall — a capability that is far less universally available than the ordinary dense-matmul path every GPU ships with by default.

L3 — The exam-relevant edge case: a real percentage that buys nothing on the wrong hardware

Here is the trap this subsection is built to test: a headline sparsity number — say, 80% of a matrix's weights zeroed — sounds like it should translate one-to-one into an 80%-ish speed or memory win, and it is tempting to read it that way. Whether it does depends entirely on a separate fact the sparsity percentage itself says nothing about: does the hardware actually running this model know how to detect and skip a zero wherever it lands? On hardware without that capability, an 80%-sparse unstructured matrix runs in the same time a fully dense matrix would, because the multiply-by-zero cycles still happen. The model genuinely occupies less disk space in a sparse-aware storage format — that part of the win is real and unconditional — but the runtime speed win is conditional on hardware the sparsity percentage does not guarantee exists.

L1 — Intuition: closing exactly half of every section, on a fixed schedule

Return to the stadium, but change the removal rule: instead of walking every seat individually, divide the whole venue into blocks of four adjacent seats, and in every single block, close exactly two and keep exactly two — never three closed, never one, always precisely half, in every block, everywhere. That rule sounds more restrictive than picking whoever scores lowest across the whole stadium, and it is — but it buys something the free-form approach cannot: a venue operator can now build one fixed staffing plan around "half of every block is always empty" and rely on it holding in every single block, everywhere, permanently.

L2 — Mechanism: why a fixed 2-of-4 ratio is what dedicated hardware can commit to

[GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "Structured 2:4 sparsity (two nonzeros per group of four) maps directly to Tensor Core sparse acceleration." Slice a weight matrix's values into non-overlapping runs of four consecutive entries, and within every run, keep only the two highest-scoring values, zeroing the other two — every run, no exceptions. Because the ratio never varies — precisely half, in every group, guaranteed — a sparse Tensor Core datapath can be engineered around exactly that guarantee: fetch the two surviving values per group along with a small marker recording which two positions they occupy, skip the multiply for the other two entirely, and finish the group's contribution to the output in less work than a full four-value dense pass would need.

Unstructured pruning's zero pattern offers no such guarantee to build hardware around — one four-value run might have all four values survive, the next might have three zeroed, and a fixed datapath has no single rule that covers every case reliably. The 2:4 pattern trades away some of the accuracy headroom an unconstrained ranking would have (a matrix free to zero whichever individual weight scores lowest, anywhere, generally preserves more signal at a matched zero-count than one forced into a rigid per-group ratio) in exchange for a guarantee specific hardware can be, and has been, built to exploit every single time.

L3 — The exam-relevant depth: why pairing 2:4 with INT8 is a documented TensorRT path

[GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "Combining INT8 with 2:4 sparsity is a documented TensorRT acceleration path." This combination is worth sitting with because it stacks two genuinely independent levers rather than doubling up on the same one. INT8 quantization (M4-01 covers PTQ's calibration mechanics in full) reduces how many bits represent each surviving weight; 2:4 sparsity reduces how many of those weights survive at all, in a hardware-exploitable pattern. Neither one substitutes for the other — a matrix can be dense and INT8, sparse and FP16, or both sparse and INT8 at once — and TensorRT's documented path applies both together, compiling a model that is simultaneously lower-precision per surviving weight and structurally halved in nonzero count per group, with both properties independently contributing to the resulting speedup. This is the same "independent, compounding levers" relationship M4-02 established between distillation and quantization, applied here between pruning's sparsity pattern and quantization's precision reduction.

⭐ THE EARNED INSIGHT

Sparsity and speed are two separate outputs that pruning can produce, and only one specific pattern reliably ties them together on today's hardware. A weight matrix's sparsity percentage is a real, measurable property regardless of pattern — but converting that percentage into an actual inference speedup requires the pattern to match what the underlying hardware was specifically built to exploit. Unstructured pruning maximizes the first output and leaves the second to chance; structured 2:4 sparsity accepts a more constrained first output in exchange for guaranteeing the second, because the pattern itself is the thing Tensor Core sparse acceleration was designed around.

The row worth returning to under time pressure is the fourth from the bottom: unstructured pruning can be more accurate at a matched sparsity level, because it is unconstrained in what it removes, while structured 2:4 sparsity is more reliably fast, because its removal pattern is exactly what dedicated hardware expects. Neither pattern dominates the other on both axes simultaneously — the choice between them is a choice about which property (accuracy headroom or guaranteed speedup) matters more for the deployment target described.

Take an illustrative weight matrix slice of 4,096 values — small enough to count exactly, large enough to show the pattern clearly — and prune it two different ways to the same overall sparsity target.

Starting point: 4,096 weight values, dense, FP16 (2 bytes each)
Dense memory: 4,096 x 2 bytes = 8,192 bytes

Target: 50% sparsity (half the weights zeroed)

Unstructured pruning to 50% sparsity, ranking all 4,096 weights by magnitude and zeroing the smallest half, wherever they fall:

Nonzero weights: 4,096 x 0.50 = 2,048
Zeros are scattered: some groups of 4 adjacent weights might have
0, 1, 2, 3, or 4 zeros among them — no fixed pattern.

If stored dense (all 4,096 positions still allocated): 8,192 bytes, unchanged
If stored sparse (value + index per nonzero, ~4 bytes each on a typical scheme):
  2,048 x 4 bytes = 8,192 bytes  (no net storage win at this overhead ratio)

Inference on dense-only hardware: unchanged — every position still multiplied
Inference on hardware with generic sparse-matmul support: proportional
  to the 50% nonzero fraction, if the support can handle arbitrary patterns

Structured 2:4 sparsity to the same 50% target, partitioning the same 4,096 weights into 4,096 / 4 = 1,024 groups of four and keeping exactly two nonzero values per group:

Groups: 1,024
Nonzero weights: 1,024 groups x 2 nonzero per group = 2,048  (identical count to the unstructured case)
Pattern: every single group has exactly 2 nonzero, 2 zero — fixed, predictable

Tensor Core sparse acceleration: reads 2 nonzero values + their in-group
  positions per group, skips the 2 zeros with no wasted multiply-cycle
Inference: real, hardware-realized speedup on any GPU with sparse
  Tensor Core support, guaranteed by the fixed pattern rather than
  contingent on how the zeros happened to fall

Both approaches reach exactly 2,048 nonzero weights out of 4,096 — the same 50% sparsity number. The difference that matters is entirely about where those zeros land: unstructured pruning's zeros could concentrate unevenly across groups (a group with all four values important survives untouched; a neighboring group might have three or four zeros), while 2:4 sparsity guarantees exactly two zeros in every group, no exceptions, no variance. That guarantee is precisely what lets Tensor Core hardware build a fixed, always-applicable fast path around it — a benefit unstructured pruning's equally-sized but irregular sparsity pattern cannot offer, because there is no fixed rule the hardware could rely on holding for every group.

Take a 7-billion-parameter decoder model's weights, stored at FP16, and apply the documented combination — structured 2:4 sparsity plus INT8 quantization — to see how the two levers' savings combine.

Baseline, dense FP16:
  7e9 params x 2 bytes = 14 GB

Step 1 — apply structured 2:4 sparsity (50% of weights zeroed, in-pattern):
  Nonzero weights: 7e9 x 0.50 = 3.5e9
  Still at FP16, with sparse storage (value + position per nonzero,
  roughly 2.25 bytes per nonzero on a typical structured-sparse encoding):
  3.5e9 x 2.25 bytes ≈ 7.875 GB

Step 2 — quantize the surviving nonzero weights to INT8:
  3.5e9 nonzero weights x (1 byte + small position overhead, ~1.25 bytes total) ≈ 4.375 GB

Treat these bytes as an illustration, built on assumed encoding overhead rather than pulled from a measured deployment — real sparse-storage overhead differs by implementation. What the arithmetic is meant to show, independent of the exact byte counts, is that the two savings stack instead of overlapping: 14 GB drops to roughly 7.875 GB from sparsity alone, and drops again to roughly 4.375 GB once INT8 is layered on top, and neither of those two drops happens without the other lever also being applied. M4-01 already covers how INT8 PTQ derives its per-tensor scale factors from a calibration pass, and none of that mechanic changes here. The new wrinkle in this example is that the calibration and quantization step is now scoped to the surviving nonzero weights inside an already-sparsified matrix, rather than to every weight in an untouched dense one — exactly the pairing [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md) documents.

The latency side of this combination follows the same compounding logic. Sparse Tensor Core acceleration delivers a real speedup from skipping the zeroed half of each group; INT8's lower per-weight bit-width additionally reduces the memory bandwidth spent moving the surviving weights through memory during the matrix multiply. Neither effect displaces the other — one addresses how much data moves, the other addresses how much of that data is real versus skippable — which is exactly why TensorRT documents them as a paired, not competing, optimization.

Model Optimization is Domain 4 of the NCP-GENL blueprint at 17% — the single largest domain on the exam — and [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md) names pruning and sparsity as a distinct subsection with its own explicitly stated trap: unstructured pruning's parameter savings do not automatically translate into a dense-hardware speedup, and structured 2:4 sparsity is the pattern that does. Objective 4.2's demand to measure an accuracy tradeoff applies here exactly as it does to quantization — pruning is a bargain, memory or latency for some accuracy risk, and the exam rewards knowing which pruning shape actually delivers the latency half of that bargain on real hardware rather than only the memory half.

Expect the question to arrive in these recognizable shapes:

• A pattern-identification item. "Which sparsity pattern maps directly to Tensor Core sparse acceleration?" with structured 2:4 sparsity as the keyed answer against distractors like random unstructured pruning, dropping the embedding layer, or an unrelated technique such as temperature scaling.
• A scenario item naming a hardware constraint. A description states a deployment target has confirmed sparse Tensor Core support and asks which pruning approach to use, or states general dense-only hardware and asks the same — the keyed reasoning traces back to whether the described hardware can exploit the specific pattern produced.
• A combination-recognition item. "Which precision technique is documented as pairing with 2:4 sparsity in TensorRT?" with INT8 as the keyed answer.
• A recovery-step item. A pruning workflow description omits fine-tuning, and the question asks what is missing or what the omission risks.

What the distractors typically look like

Expect unstructured pruning's scattered zeros described as "hardware-friendly" or "the accelerated pattern" — backwards, since it is the fixed, regular 2:4 shape that dedicated hardware is built to exploit, not an arbitrary scatter of zeros however dense. Expect a sparsity percentage offered as if it were already a speed number on its own, trading on the intuitive but false assumption that more zeros automatically means faster inference regardless of pattern or hardware. Expect fine-tuning omitted from a described pruning workflow as if pruning alone, with no recovery step, were the complete and standard procedure. And expect FP16 or another precision format offered in place of INT8 as "the" documented TensorRT pairing with 2:4 sparsity, when INT8 specifically is the combination named.

They are related but sit at different granularities, and mixing them up is a common source of confusion. Structured pruning, in its broader sense, deletes whole predefined units — an entire channel, filter, or attention head — leaving behind a matrix with smaller dimensions and no zeros scattered inside it, so ordinary dense hardware handles the result at full speed without any special support. Structured 2:4 sparsity is a narrower, finer-grained pattern operating inside the matrix itself: within every fixed run of four consecutive weight values, exactly two survive, with no regard for where a channel or filter boundary happens to fall. A model can have whole attention heads deleted at the channel level and, separately, have its surviving weight matrices reduced to a 2:4 pattern — two compatible interventions at two different granularities, not one technique described twice.

Yes, and section 6's worked example is exactly this combination in the specific, documented TensorRT form: structured 2:4 sparsity plus INT8 quantization. The two techniques attack different kinds of redundancy — pruning (in either its channel-level or 2:4 form) removes or zeroes weights the network uses least, while quantization represents the weights that remain, or survive pruning, with fewer bits. They do not simply add their savings without measurement, though: the combined result needs its own accuracy re-evaluation on a held-out eval set, because each technique changes the starting point the other operates on, exactly as M4-01's discipline around measuring quantization's accuracy tradeoff requires independent of whether pruning happened first.

Work through each item and reason out an answer before checking the key underneath. Each wrong choice names something that is true of some pruning scenario — pick the one that fits the specific setup described, rather than ruling out anything as nonsense on its face.

1. A weight matrix is sliced into runs of four consecutive values, and every run keeps exactly two survivors. What is this pattern called, and what hardware feature does it target?
   • A. Magnitude pruning; it targets CPU cache lines.
   • B. Structured 2:4 sparsity; it targets sparse Tensor Core acceleration.
   • C. Knowledge distillation; it targets a smaller student architecture.
   • D. QAT; it targets fake-quant nodes.
2. A deployment reports a large sparsity percentage on an unstructured-pruned checkpoint, yet the measured latency is identical to the original dense model. What best explains this outcome?
   • A. The reported sparsity figure must be wrong.
   • B. The target GPU has no mechanism for locating and bypassing zeros wherever they land in this irregular pattern.
   • C. Sparsity never affects latency under any circumstance.
   • D. The checkpoint was actually quantized, not pruned.
3. Which precision format does the source material name as the documented TensorRT pairing for 2:4 sparsity?
   • A. FP32.
   • B. INT8.
   • C. BF16.
   • D. FP64.
4. A pruned network's accuracy on the eval set sits below expectations immediately after weights are deleted. What is the standard next step, and what does it accomplish?
   • A. Re-run inference on the same weights; nothing changes without an additional step.
   • B. Resume training briefly at a lowered learning rate, letting survivors adjust to the removed neighbors' absence.
   • C. Increase the sparsity percentage further.
   • D. Switch to FP32 for all remaining weights.
5. Two matrices reach the identical 50% zero fraction — one via unconstrained per-weight ranking, one via a fixed four-value-run rule. Only one reliably speeds up on sparse-aware hardware. Why?
   • A. The unconstrained version always has fewer total zeros in practice.
   • B. Only the fixed-rule version guarantees the same ratio in every run, which is the property a fixed hardware datapath can commit to exploiting every time.
   • C. Sparse hardware cannot process a 50% zero fraction at all.
   • D. The unconstrained version is always more accurate, which disqualifies it from acceleration.
6. Whole attention heads are deleted from a model, shrinking its dimensions with no leftover zeros inside the surviving matrices. Which technique is this?
   • A. Structured 2:4 sparsity.
   • B. Channel/unit-level structured pruning.
   • C. Post-training quantization.
   • D. Streaming attention.
7. Which statement correctly separates what pruning changes from what quantization changes?
   • A. Both change the same property: total parameter count.
   • B. Pruning deletes weights or structural units outright; quantization keeps every weight but re-encodes its numeric precision.
   • C. Quantization deletes weights; pruning re-encodes precision.
   • D. Neither technique is measurable for an accuracy cost.
8. A team's deployment target has no confirmed sparse-hardware support and cares most about disk footprint rather than inference speed. Which pruning approach best fits, and why?
   • A. Structured 2:4 sparsity, because it is always the superior choice regardless of hardware.
   • B. Unstructured pruning with a sparse storage format, because its storage savings do not depend on the hardware's ability to bypass zeros at runtime.
   • C. Neither approach helps without a training budget.
   • D. Distillation, because it is the only lever that reduces disk footprint.

Answers

1. B. Section 3 names this pattern and its target directly: a fixed two-of-four survivor ratio is exactly what sparse Tensor Core hardware is engineered to exploit.
2. B. This is section 2's L3 point: a real sparsity percentage does not by itself guarantee a speedup — the hardware must be able to locate and skip zeros in the specific pattern produced.
3. B. INT8 is the precision [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md) names as pairing with 2:4 sparsity, worked through arithmetically in section 6.
4. B. This is section 1's recovery mechanism: a brief resumed-training pass lets survivors re-settle around the gap left by deleted neighbors.
5. B. Section 3's L2 mechanism: a guaranteed, unvarying per-run ratio is what a fixed hardware datapath can be built around; an unconstrained pattern offers no such guarantee.
6. B. This is the broader, unit-level sense of structured pruning distinguished from the finer-grained 2:4 pattern in the "Is structured 2:4 sparsity the same thing as structured pruning?" discussion above.
7. B. This is the module's running axis distinction: pruning removes what exists; quantization changes how what exists is represented, unchanged from M4-01 and M4-02's framing.
8. B. Section 4's comparison table states this directly: unstructured pruning's storage win, unlike its speed win, does not require any particular hardware capability to be real.

• Pruning deletes existing weights or structural units from an already-trained checkpoint — a distinct axis from quantization's precision re-encoding (M4-01) and distillation's separate-architecture training (M4-02).
• A large unstructured sparsity percentage is a real, measurable property of the matrix, but it converts into a latency win only if the target hardware can locate and bypass zeros in whatever irregular pattern resulted.
• The 2:4 rule — exactly two survivors in every run of four — is what lets sparse Tensor Core hardware commit to a fixed, always-applicable bypass strategy, which an unconstrained zero pattern cannot offer.
• Pairing INT8 with 2:4 sparsity is TensorRT's documented acceleration path, and the two savings stack because one governs bit-width per surviving weight while the other governs which weights survive at all — genuinely independent axes.
• A brief resumed-training pass is the normal follow-up to any pruning step, letting the surviving weights compensate for whatever the deleted ones used to contribute.
• Model Optimization is 17% of the NCP-GENL blueprint, the single largest domain, and the gap between a real sparsity number and a guaranteed hardware speedup is one of this subsection's most directly tested single facts.

Pruning and quantization both change properties of the model itself — how many weights it has, or how many bits represent each one — before a single inference request is ever served. The next lever in this module shifts the frame entirely, from "what does the model look like" to "what happens at the moment it runs": how attention is computed and bounded during a very long generation, and how a compiler tunes the whole execution for the specific GPU serving it.

Next: M4-05 covers streaming attention and TensorRT runtime optimization — how sliding-window attention bounds memory for very long sequences, and how TensorRT compiles, fuses kernels, and auto-tunes a model for a target GPU, distinct from the serving role Triton plays in Domain 8.