PTQ vs QAT vs GPTQ: Quantization Approaches Compared

Three techniques get compressed into one word — "quantization" — and the exam's single most reliable trap is asking you to tell them apart under a scenario rather than a definition. Post-training quantization (PTQ) takes a finished model and calibrates scale factors against sample data, with no retraining at all. Quantization-aware training (QAT) inserts fake-quantization nodes into the model and retrains it so the weights themselves learn to tolerate the coming precision cut. GPTQ is a third, separate thing again: a one-shot, post-training, weight-only method that uses approximate second-order information to solve for low-bit weights directly, without touching a single gradient of the original training objective.

All three exist to answer the same underlying question — how do you make a model smaller and faster without wrecking it — but they answer it in incompatible ways, with incompatible costs, and the exam rewards knowing exactly which one a described situation is asking for. This is also the highest-leverage lesson in the highest-leverage domain on the exam: [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md) states Model Optimization is Domain 4, and at 17% of the blueprint it is the single largest domain NVIDIA tests, larger than any other topic area on the NCP-GENL exam. Quantization is the first and most heavily traded concept inside it, and [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md) frames objective 4.2 as explicitly requiring a measured accuracy tradeoff for every optimization technique in this domain — never an assumed one.

1. What quantization actually does to memory and compute

Identity statement: quantization is the act of representing a value that was stored in a high-precision numeric format (typically FP32 or FP16) using a lower-precision format (INT8, FP8, or INT4), in order to cut the memory a model occupies and the compute a GPU spends serving it.

When it matters: whenever a model does not fit on the GPU you have, whenever inference is too slow for the latency budget, or whenever a scenario item on the exam offers "quantize it" as one of four answer choices and you need to know whether that is actually the fix being described.

The mechanism is a byte-cost change, and it obeys the same rule as any other precision decision: weight memory is approximately parameter count times bytes per parameter.

Precision	Bytes per parameter	Weight memory for a 70B model
FP32	4	~280 GB
FP16 / BF16	2	~140 GB
FP8 / INT8	1	~70 GB
INT4	0.5	~35 GB
INT3 (3-bit, packed)	~0.375	~26 GB

Read the table as a lever, not a curiosity. Every step down halves the weight footprint again, and because Tensor Cores on modern NVIDIA GPUs run lower-precision arithmetic faster than higher-precision arithmetic, the same step usually buys latency too — not just storage. That is the entire economic case for quantization: you are trading numeric precision, which costs you accuracy you must measure, for memory and speed, which you can bank immediately.

But "quantization" as a single word hides a critical branch: it does not tell you how the lower-precision values were chosen. That is exactly the branch PTQ, QAT, and GPTQ split on, and it is the branch this lesson exists to make automatic. Note also what quantization is not: it is never a technique for improving accuracy. It reduces memory and latency, full stop. The best any quantization method does is protect accuracy while cutting cost — a distinction the exam tests directly, and one worth carrying as a standing check on every quantization claim you read for the rest of this lesson.

2. Post-training quantization: calibrating after the fact

Identity statement: post-training quantization (PTQ) takes an already-trained, full-precision model and quantizes it using a small calibration dataset to observe activation distributions and derive per-tensor scale factors — with zero retraining and zero gradient updates.

PTQ's whole value proposition is speed of turnaround. You have a finished checkpoint, you feed it a representative batch of unlabeled data — a calibration set, not a training set, and critically not a labeled one — and you record how activations actually behave at each layer: their typical range, their outliers, their distribution shape. From those observations you compute a scale factor per tensor (or per channel, in finer-grained variants) that maps the original floating-point range onto the target low-precision integer or float range with the least distortion. Once the scales are fixed, every future forward pass through that layer uses the quantized weights and the calibrated scale. No backward pass ever runs.

Why calibration data matters more than it sounds

The quality of PTQ's result depends entirely on how representative the calibration set is of real production inputs. If calibration data under-samples the range of activations the model will actually see in deployment, the derived scale factors clip or waste precision on values that never show up, and accuracy degrades in ways that only appear once the model is live. This is why PTQ pipelines emphasize calibration set curation even though the set itself is small — often a few hundred to a few thousand examples — and unlabeled.

Where PTQ runs in the NVIDIA stack

TensorRT is the layer where PTQ calibration is most commonly executed in an NVIDIA deployment pipeline: it observes per-tensor activation ranges on representative data during a calibration pass, computes the resulting scales, and folds them into the compiled engine. [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "NVIDIA ModelOpt / NeMo PTQ supports FP8, INT8 SmoothQuant, and INT4 AWQ" — named recipes rather than raw calibration mechanics. INT8 SmoothQuant redistributes quantization difficulty between weights and activations before calibrating, specifically to handle the outlier activations that plain INT8 calibration struggles with, and INT4 AWQ (Activation-aware Weight Quantization) weighs which weights matter most to preserve based on the activations they interact with. All three are PTQ variants: all three quantize an already-trained checkpoint using calibration data, and none of them retrain.

PTQ's failure mode is precision-dependent, not universal. At INT8 and FP8, PTQ is routinely good enough for production — the accuracy loss is typically small and often within noise for many tasks. The gap widens as precision drops further. At INT4 and especially FP4, calibration alone increasingly struggles to find scale factors that preserve the model's behavior, because the representable value space has become too coarse for a fixed set of retroactively-chosen scales to cover the range the model actually needs. That widening gap at low precision is the exact reason QAT exists as a separate technique rather than PTQ simply being pushed harder.

3. Quantization-aware training: learning to tolerate the cut

Identity statement: quantization-aware training (QAT) inserts quantize/dequantize (QDQ) nodes into the model's computation graph and then retrains — or continues training — the model with those nodes active, so the weights themselves adapt to the reduced precision rather than being quantized after the fact.

A QDQ node simulates the effect of quantization during the forward pass — it quantizes a tensor down to the target low-precision representation and immediately dequantizes it back to a higher-precision type for the rest of the computation, a pattern often called "fake quantization." The forward pass therefore sees the same rounding and clipping errors the final deployed quantized model will experience. Because training is running through these QDQ nodes, gradients flow back through them (using a straight-through estimator to route gradients around the technically non-differentiable rounding step), and the optimizer adjusts weight values specifically to compensate for that rounding and clipping. The model, in effect, learns weight values that are robust to being quantized — not weight values that happen to survive quantization by luck, as in PTQ.

Why QAT wins where PTQ loses

The gap between PTQ and QAT is not constant across precisions — it widens specifically at very low precision. At FP16 or INT8, a well-calibrated PTQ pass is often nearly as good as QAT, because the representable value space is still fine enough that post-hoc scale factors capture most of what the model needs. At INT4 and FP4, the value space is coarse enough that where a weight lands relative to its quantization bins starts to matter for accuracy in a way calibration cannot fully compensate for after training has already finished. QAT resolves that by baking the compensation into training itself: the model does not merely tolerate coarse bins, it adjusts its own weight values so the bins it lands in produce the right output.

What QAT costs that PTQ does not

That accuracy advantage is not free. QAT requires a training or fine-tuning run — meaning labeled or task-appropriate training data, GPU-hours for gradient updates, and the full engineering overhead of a training job (checkpointing, hyperparameter selection, validation) rather than a lightweight calibration pass. TensorRT supports both the PTQ and the QAT path: it can either calibrate an already-trained network directly, or ingest a network that already has QDQ nodes baked in from a QAT-aware training framework and compile that into an efficient low-precision engine. The choice between the two is a build-time decision about how much retraining budget you have, not a runtime one.

4. GPTQ: a distinct third approach, not a QAT variant

Identity statement: GPTQ is a one-shot, post-training, weight-only quantization method that uses approximate second-order (Hessian) information about the model's loss surface to solve for low-bit weight values directly, layer by layer, without any gradient-based retraining.

This is the point where the exam's trap sharpens, because GPTQ sounds like it should be a flavor of QAT — it produces accuracy that rivals or beats plain PTQ at very low bit-widths, which is exactly the territory QAT is supposed to own — and yet GPTQ is not QAT at all. It never runs a training loop. It never touches the original loss function's gradients. It runs once, after training has already finished, and it never updates the model's behavior on a task; it only ever solves the narrower problem of "what low-bit values best approximate this already-trained weight matrix, given how the model actually uses it."

How the Hessian trick works, at the level the exam expects

GPTQ quantizes one layer's weight matrix at a time. For each layer, it uses a small calibration set (again: unlabeled, not a training set, exactly like PTQ) to estimate how sensitive that layer's output is to small changes in each weight — an approximation of the Hessian, the matrix of second derivatives of the reconstruction error with respect to the weights. It then quantizes weights column by column, and after quantizing each column it adjusts the remaining, not-yet-quantized weights in that same layer to compensate for the error just introduced — using the Hessian information to decide exactly how much compensation each surviving weight needs. That per-column, error-compensating update is what lets GPTQ push down to 3-4 bits per weight — and, pushed further, to 2 bits in extreme regimes — while a naive round-to-nearest quantization at the same bit-width would collapse accuracy. [GROUND TRUTH] (Sources/ncp-genl/domain-4-model-optimization.md): "It can quantize a 175B-parameter GPT model to 3-4 bits per weight in a few GPU hours with negligible accuracy loss, enabling that model to run for generative inference on a single GPU" — the signature GPTQ result this lesson's worked example (§6) walks through in full.

Why "weight-only" is a load-bearing phrase

GPTQ quantizes weights and leaves activations in their original higher-precision format at inference time, dequantizing weights on the fly for the matrix multiply. This is different from full quantization schemes like SmoothQuant or plain INT8 PTQ, which quantize both weights and activations. Weight-only quantization is specifically well suited to memory-bound serving scenarios — cases where the bottleneck is fitting the model and moving its weights through memory, rather than the arithmetic itself — which is precisely the situation a 175B-parameter model on a single GPU represents.

The two traps to hold onto

Two mistakes recur often enough to name explicitly. First, GPTQ is not a QAT variant: it needs no labeled data, no training loop, and no loss-function gradient descent on a task — it needs only a small unlabeled calibration set and a few GPU-hours of a fundamentally different, layer-local optimization procedure. Second, GPTQ is not interchangeable with generic PTQ, even though both are post-training and both use calibration data: PTQ (as TensorRT or ModelOpt typically implement it) derives per-tensor scale factors for both weights and activations from observed distributions, while GPTQ solves a per-column, error-compensating optimization problem for weights only, using second-order information that plain calibration-based PTQ does not use at all. They are close cousins that produce similar-sounding claims and answer different exam questions.

5. PTQ vs QAT vs GPTQ: the three-way comparison

This is the table to memorize in shape, because a large share of Domain 4's scenario items resolve directly against it.

	PTQ	QAT	GPTQ
Retraining required?	No	Yes	No
Calibration data needed?	Yes, unlabeled	No calibration set — uses labeled/task training data instead	Yes, unlabeled
Uses gradients on the task loss?	No	Yes	No (uses second-order info for layer-local error compensation, not task-loss gradients)
Accuracy at INT8/FP8	Good	Good to slightly better	Not its target regime
Accuracy at INT4/FP4/3-bit	Degrades noticeably	Best of the three, purpose-built for this regime	Strong — its signature result is exactly here
Speed to produce a quantized model	Fastest — a calibration pass	Slowest — a full (re)training run	Fast — a few GPU-hours, no training loop
What gets quantized	Weights and activations (typically)	Weights and activations, learned jointly with the rest of training	Weights only
Typical use case	Quick deployment win at moderate precision (INT8/FP8)	Maximum accuracy at aggressive precision when a training budget exists	Extreme compression of very large models for single-GPU inference, with no training budget

Notice the two axes that actually separate these three, because distractor answers on the exam are built by scrambling them. The first axis is retraining: only QAT retrains; PTQ and GPTQ are both strictly post-training. The second axis is what gets quantized: PTQ and QAT typically quantize both weights and activations, while GPTQ is weight-only. A GPTQ description that mentions activations being quantized, or a PTQ description that claims it needs labeled data, or a QAT description with no training step at all — each is a scrambled version of one row in this table, and each is a wrong answer choice built exactly that way.

6. Worked example: taking a 175B model to 3-4 bits with GPTQ

Take a 175-billion-parameter GPT-scale model — the canonical example in the GPTQ literature — stored at FP16.

FP16 weight memory:  175e9 params × 2 bytes = 350 GB

That footprint alone rules out serving on a single high-end GPU; even the largest single-GPU memory configurations available fall well short of 350 GB, before any KV cache or activation memory is added. Now apply GPTQ's headline result: quantizing to 3-4 bits per weight.

GPTQ at 4 bits: 175e9 params × 0.5 bytes = 87.5 GB
GPTQ at 3 bits: 175e9 params × 0.375 bytes = 65.6 GB

Read that as an engineer, not an arithmetician. Going from FP16 to 4-bit weights is a 4x reduction in raw byte count — 2 bytes down to 0.5 bytes per parameter — and it takes the weight footprint from a multi-GPU-only 350 GB down to a range that a single high-end GPU with substantial memory can plausibly hold, with headroom left for the KV cache and activations that any serving deployment also needs. Going one bit lower, to 3-bit packed weights, squeezes further still, to roughly 65.6 GB.

The number that makes this result remarkable is not just the memory ratio — it is the cost and the accuracy loss it took to get there. GPTQ's reported result for a model at this scale is a few GPU-hours of one-shot, layer-by-layer processing (not a training run measured in GPU-days or GPU-weeks) to produce that 3-4 bit checkpoint, with negligible accuracy loss relative to the FP16 original. Layer this against the reported end-to-end inference speedup: on high-end GPUs, GPTQ-quantized models have shown roughly 3-4.5x throughput improvement over FP16, on top of the memory win, because low-bit weight formats move less data through memory bandwidth-bound matrix multiplies and can leverage faster Tensor Core paths once dequantized on the fly.

Contrast the type of cost this required against what PTQ or QAT would have cost for the same target. A comparable PTQ pass at 4-bit would have been similarly fast — a calibration pass, not a training run — but the GPTQ literature's entire point is that plain calibration-based PTQ degrades substantially more at 3-4 bits than GPTQ's Hessian-guided, error-compensating approach does. A QAT pass targeting the same 3-4 bit result could plausibly match or exceed GPTQ's accuracy, but would require a full retraining run on a 175B-parameter model — GPU-days to GPU-weeks of cost, not GPU-hours, and a labeled or task-appropriate training corpus GPTQ never needed. That cost delta is the entire reason GPTQ is treated as its own category rather than as "PTQ, but better" or "QAT, but cheaper": it occupies a cost-accuracy point neither of the other two techniques can reach on its own.

7. Worked example: choosing a technique under a stated constraint

Numbers alone do not resolve a scenario item; the constraint does. Walk through three variations of the same underlying question — "we need a 40B-parameter model to run faster and smaller" — because each variation points at a different one of the three techniques, and the exam builds its distractors by scrambling exactly this kind of constraint.

Variation A: no training budget, no labeled data, target is INT8 for a quick win. This is PTQ's exact niche. A calibration pass against a few hundred unlabeled representative prompts, run once, produces per-tensor scale factors, and the model deploys the same day. At INT8 the accuracy gap against the FP16 baseline is typically small enough that no further work is justified. Memory drops from 40e9 × 2 bytes = 80 GB at FP16 to 40e9 × 1 byte = 40 GB at INT8 — a model that previously needed two high-memory GPUs now plausibly fits on one, with no retraining cost at all.

Variation B: a training budget exists, labeled data exists, and the target is INT4 for maximum accuracy. This is QAT's exact niche. The team inserts QDQ nodes, retrains (or continues training) the model so its weights adapt to INT4 rounding and clipping, and accepts the GPU-days-to-GPU-weeks cost of a real training run in exchange for the best achievable INT4 accuracy. Memory drops to 40e9 × 0.5 bytes = 20 GB, matching PTQ's memory result at the same bit-width — the difference between the two is not the memory outcome, it is the accuracy retained at that outcome and the cost paid to get there.

Variation C: no training budget, no labeled data, but the target is still INT4 and accuracy loss must stay negligible. Neither Variation A's PTQ nor Variation B's QAT cleanly fits — PTQ alone tends to degrade too much at INT4 for this bar, and QAT is unavailable because there is no training budget or labeled data. This is GPTQ's exact niche: an unlabeled calibration set, a one-shot Hessian-guided layer-by-layer pass, a few GPU-hours, and a memory result identical to plain INT4 PTQ or QAT (20 GB) but an accuracy result close to what QAT would have delivered, without QAT's cost.

The pattern to extract: the memory arithmetic is identical across all three variations at a given bit-width, because memory is a function of bit-width alone. What separates PTQ, QAT, and GPTQ is never the memory number — it is the accuracy achieved at that memory number, and the cost paid to achieve it. A scenario item that only gives you a memory target, with no mention of a training budget, labeled data, or an accuracy bar, is usually testing whether you notice that the memory number alone cannot tell you which technique is correct — you need the constraint, not just the target.

8. Why this trio is on the NCP-GENL exam

Domain 4, Model Optimization, is 17% of the NCP-GENL blueprint — the single largest domain on the exam, ahead of every other topic area including retrieval, evaluation, and deployment. Within that domain, quantization is the first technique the official study guide's structure addresses, and PTQ-vs-QAT is explicitly the comparison the exam's own objective language foregrounds: objective 4.2 asks you to reason about measured accuracy tradeoffs, which is exactly the axis that separates PTQ from QAT from GPTQ. You will not be asked to derive Hessian mathematics or implement a QDQ node. You will be asked to recognize which of the three a described situation calls for, and to reject the two named misconceptions the source material calls out explicitly as traps.

The question tends to arrive in a small number of recognizable shapes:

• A "which technique" scenario naming a constraint. "A team has no training budget and needs a 70B model to fit on one GPU with minimal accuracy loss at INT4" points at GPTQ specifically, because it names the absence of a training budget alongside a demand for low-precision accuracy — the exact GPTQ niche between plain PTQ and QAT.
• A "which technique" scenario naming a training budget. "A team has labeled data and GPU time and needs the best possible accuracy at INT4 for a production model" points at QAT, because retraining is available and the precision target is in QAT's strongest regime.
• A definition-discrimination item. Four descriptions of quantization approaches, one correctly attributing calibration-without-retraining to PTQ, one correctly attributing QDQ-nodes-and-retraining to QAT, one correctly attributing weight-only-Hessian-one-shot to GPTQ, and one scrambled distractor.
• A misconception-correction item. A claim like "quantization improves accuracy" or "GPTQ requires labeled training data," offered as a statement to evaluate true or false, or as one of four answer choices where three are wrong precisely because they violate a named trap.

What the distractors typically look like

Expect GPTQ described as "a form of QAT" or "requiring retraining" — both false, since GPTQ is strictly post-training and gradient-free on the task loss. Expect PTQ described as needing labeled data — false, calibration sets are unlabeled. Expect quantization broadly described as a way to improve accuracy rather than to protect it while cutting cost — false in all three techniques; the best case is always "as close to the unquantized baseline as possible," never "better than it." And expect the three techniques presented as interchangeable synonyms in a distractor answer, when the entire tested skill is knowing they are not.

9. Common mistakes about quantization

Mistake	Symptom you would actually observe	Fix
Believing quantization improves accuracy	A report claims a quantized model "performs better" than its FP16 baseline and nobody checks why	Quantization protects accuracy while cutting memory/latency; it never improves it. Any apparent gain is noise, a different eval set, or a bug
Treating PTQ, QAT, and GPTQ as interchangeable	A team picks whichever tool is easiest to run, then is surprised by the accuracy or cost result	Each occupies a different point on the retraining-cost/accuracy tradeoff; the choice should follow from whether a training budget exists and how low the target precision is
Calling GPTQ a QAT variant	A written explanation states GPTQ "trains the model to tolerate quantization"	GPTQ is post-training and weight-only; it never runs a training loop or touches task-loss gradients
Assuming PTQ needs labeled data	A team blocks a PTQ rollout waiting on labels that were never required	PTQ (and GPTQ) use small unlabeled calibration sets to observe activation behavior, not labeled training sets
Assuming QAT is always the right choice because it is "more accurate"	A team burns a full retraining budget on a target precision where PTQ would have been indistinguishable	The QAT-vs-PTQ accuracy gap widens specifically at very low precision (INT4/FP4); at INT8/FP8 the two are often close enough that PTQ's speed wins
Confusing PTQ's per-tensor scale factors with GPTQ's per-column optimization	A description of GPTQ says it "calibrates scale factors" and stops there	GPTQ additionally uses Hessian-based error compensation to adjust remaining weights after each column is quantized — a mechanism plain calibration-based PTQ does not have
Believing quantization is free	Throughput improves and a quality regression ships unnoticed	Every precision drop costs accuracy that must be measured on an evaluation set, not assumed away
Forgetting that GPTQ is weight-only	A claim states GPTQ quantizes activations the same way it quantizes weights	GPTQ leaves activations at higher precision and dequantizes weights on the fly at inference; full weight-and-activation quantization is what PTQ schemes like SmoothQuant do instead

10. Glossary recap: the terms this lesson introduced

Term	One-line definition
Quantization	Representing values in a lower-precision numeric format to cut memory and compute cost
PTQ (Post-Training Quantization)	Quantizing an already-trained model using calibration data; no retraining
QAT (Quantization-Aware Training)	Retraining a model with quantize/dequantize nodes so weights adapt to reduced precision
GPTQ	A one-shot, post-training, weight-only quantization method using approximate Hessian information
Calibration	Running representative unlabeled data through a model to observe activation ranges and derive scale factors
Scale factor	The per-tensor (or per-channel) multiplier mapping a high-precision range onto a lower-precision representable range
QDQ node	A quantize/dequantize node inserted into a graph to simulate quantization effects during training ("fake quantization")
Straight-through estimator	The gradient approximation that lets backpropagation flow through the technically non-differentiable rounding step in a QDQ node
SmoothQuant	An INT8 PTQ recipe that redistributes quantization difficulty between weights and activations before calibrating
AWQ (Activation-aware Weight Quantization)	An INT4 PTQ recipe that weighs which weights to preserve based on the activations they interact with
Weight-only quantization	Quantizing weights while leaving activations at higher precision; well suited to memory-bound serving
Hessian (approximate, second-order information)	The matrix of second derivatives GPTQ uses to estimate how sensitive a layer's output is to weight changes, guiding error-compensating updates
TensorRT	NVIDIA's inference optimizer, supporting both PTQ calibration and compilation of QAT-trained, QDQ-annotated networks

11. Key takeaways on PTQ, QAT, and GPTQ

• Quantization never improves accuracy. It cuts memory and latency; accuracy is the cost you measure, not a benefit you gain.
• PTQ calibrates after training, with unlabeled data, and never retrains. It is the fastest path to a quantized model and is generally sufficient at INT8/FP8.
• QAT retrains with QDQ nodes so weights learn to tolerate reduced precision. It costs a training run but wins the accuracy comparison specifically at very low precision (INT4/FP4).
• GPTQ is a distinct third approach — one-shot, post-training, weight-only, and driven by approximate Hessian information — not a variant of QAT and not identical to generic PTQ.
• The 175B-to-3-4-bit result is the signature GPTQ fact: a few GPU-hours, negligible accuracy loss, and roughly 3-4.5x end-to-end speedup over FP16 on high-end GPUs.
• TensorRT supports both the PTQ and QAT paths, and NVIDIA ModelOpt/NeMo PTQ tooling names FP8, INT8 SmoothQuant, and INT4 AWQ as specific recipes within the PTQ family.
• Sort every quantization claim by two questions: does it retrain, and does it touch activations? Those two axes resolve almost every PTQ-vs-QAT-vs-GPTQ scenario item.
• Domain 4 is 17% of the NCP-GENL blueprint, the single largest domain, and this PTQ/QAT/GPTQ trio is its most heavily tested confusable set.

12. Next: KV caching as the primary latency lever

Quantization answers "how do I make the weights themselves smaller and cheaper to compute with." It does not touch the other major cost center in autoregressive serving: the attention computation repeated at every single decoding step, which grows with every token already generated. That cost is governed by a different mechanism entirely, and understanding it is what separates a memory-sized deployment plan from a latency-sized one.

Next: KV caching as the primary latency lever — how storing per-token key and value tensors avoids recomputing attention over an entire growing prefix at every decoding step, why the exam calls this out as the main lever for autoregressive latency rather than for memory, and how it composes with a quantized model rather than substituting for one.