Mixed Precision and Tensor Cores: FP16 vs BF16 for LLM Training

Identity statement: a Tensor Core is a specialized GPU arithmetic unit, distinct from an ordinary CUDA core, purpose-built to perform a small fused matrix multiply-accumulate operation at much higher throughput on reduced-precision inputs than the same operation achieves running through general-purpose FP32 units.

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): "FP16/BF16 compute on Tensor Cores dramatically increases throughput; it's the default fast path on A100/H100." Read that claim in two separate pieces, because they answer two different questions. The memory claim — FP16 and BF16 are each 2 bytes per value, half of FP32's 4 — is true regardless of what hardware is running the arithmetic; a CPU with no Tensor Cores at all still gets the memory halving from switching formats. The throughput claim is a property of the hardware, not the format alone: a Tensor Core takes 16-bit inputs specifically and executes a fused multiply-accumulate on them at a rate an ordinary FP32-only unit cannot match, which is why "mixed precision" and "Tensor Cores" are taught together rather than as two unrelated topics. Without Tensor Cores present on the chip, running the same FP16 numbers through ordinary arithmetic units still gets you the memory win but not the multiplied throughput — the format alone does not manufacture speed; the hardware path does.

[GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md) additionally notes that "self-attention is dominated by GEMMs (general matrix multiplications); distributing and fusing these efficiently is central to throughput," which is exactly what makes transformer training and inference such a good match for Tensor Cores in the first place: nearly every expensive operation inside a transformer layer — the query/key/value projections, the attention score computation, the feed-forward matrices — is a large matrix multiplication, which is precisely the operation Tensor Cores are built to accelerate.

This match is not automatic just because a layer happens to contain a matrix multiplication. Tensor Cores execute their fused multiply-accumulate on small, fixed-shape tiles of a matrix at a time, so a GEMM's dimensions engaging that hardware path efficiently benefit from being reasonably large and, in practice, aligned to shapes the hardware tiles well — an attention head width or a hidden dimension that is a small, oddly-sized number relative to the tile size can leave part of a Tensor Core's fused operation doing wasted work on padding rather than genuinely useful arithmetic. This is a secondary consideration relative to the format-and-safeguard decision this lesson is centrally about, but it explains why "just cast everything to FP16 or BF16" is necessary but not automatically sufficient for the full throughput a Tensor Core can offer — the matrix shapes flowing through a model's layers matter too, and a framework's automatic-mixed-precision path typically handles this shape-awareness so a model author does not have to reason about tile alignment by hand.

Identity statement: FP16 and BF16 are both 16-bit floating-point formats, but they split their 16 bits differently between exponent (range) and mantissa (precision), which produces two genuinely different risk profiles for the same nominal byte savings.

L1 — Intuition: two ways to spend the same 16 bits

A floating-point number's bits are split among a sign bit, an exponent (which determines how large or small a magnitude the format can represent) and a mantissa (which determines how finely it can distinguish values within that range). FP32 spends its 32 bits generously on both dimensions — a wide exponent and a long mantissa — which is why it is the reference format everything else is compared against. FP16 keeps a comparatively narrow exponent field but a wider mantissa relative to its total size, buying it fine resolution but at the cost of a range that is much closer to overflowing (too large to represent) or underflowing (too small to represent, rounding to exactly zero) than FP32's. BF16 makes the opposite tradeoff at the same 16-bit budget: it keeps an exponent field the same width as FP32's — meaning it can represent numbers across essentially the same enormous range FP32 can — at the cost of a much shorter mantissa than either FP32 or FP16, so it has less fine-grained resolution within whatever range a value falls in.

L2 — Mechanism: why this makes FP16 more prone to underflow

Gradients late in backpropagation, particularly for parameters far from the loss computation, are frequently small in magnitude — well within FP32's representable range but capable of falling below FP16's smallest representable nonzero magnitude. When that happens under FP16, the gradient rounds to exactly zero: not a small approximation error, but a complete loss of that gradient's information, and the parameter it would have updated simply stops moving. [GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): "FP16 without loss scaling risks gradient underflow." BF16, because its exponent range matches FP32's, does not compress the space of representable magnitudes the way FP16 does — a gradient small enough to matter numerically in FP32 is, in the overwhelming majority of cases, still representable (if less precisely) in BF16, because the range problem that causes FP16's underflow simply does not exist to the same degree for a format whose exponent field was never narrowed in the first place. [GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md): "BF16 has a wider exponent range and often needs no loss scaling."

L3 — What "often needs no loss scaling" does not claim

The word "often" in that source statement is doing real work, and treating it as "never" is an overreach the exam's own material does not make. BF16's wider exponent range removes the underflow risk that specifically motivated FP16's loss scaling, but it does so at the cost of a shorter mantissa — meaning BF16 has less fine-grained numerical resolution than FP16 does for values that both formats can represent without trouble. That is not a claim that BF16 dominates FP16 in every respect; it is a claim that the specific failure mode loss scaling was invented to patch is much less likely to occur under BF16, which is a different statement from "BF16 is unconditionally more accurate." A model whose training is sensitive to fine numerical resolution rather than to underflow risk is not automatically better served by BF16 just because BF16 tolerates the absence of loss scaling; the two formats are a genuine tradeoff, not a strictly-better-or-worse pair.

Identity statement: loss scaling multiplies the loss value by a large constant before the backward pass, so that every gradient computed from it is proportionally scaled up into FP16's representable range, and then divides the resulting parameter update back down by the same constant before it is applied — a mechanism specific to protecting FP16 from the underflow risk section 2 describes.

Because backpropagation is a chain of multiplications, scaling the loss by a constant factor scales every gradient derived from that loss by the same factor, uniformly, before any of those gradients are ever stored in FP16. A gradient that would have underflowed to zero at its true, unscaled magnitude survives the trip through FP16 storage once it has been shifted up into a representable range by the scaling factor. After the backward pass computes these artificially inflated gradients, the training loop divides the resulting update back down by the same scaling constant before applying it to the weights, so the final parameter change is mathematically identical to what an unscaled FP32 computation would have produced — the scaling factor only ever protects values while they are in transit through FP16's narrow range, and it cancels out of the final result entirely.

The constant is commonly chosen as a power of two, so the multiplication and its later reversal are numerically clean operations rather than themselves introducing rounding noise. Too small a scaling factor leaves some gradients still underflowing; too large a factor risks pushing some gradients toward FP16's overflow ceiling instead, producing infinities or NaNs where none existed before. Dynamic loss scaling — adjusting the constant automatically based on whether overflow was observed in a given step — exists specifically to find a workable value without requiring it to be hand-tuned in advance for every model and every training configuration.

Take an illustrative decoder model and quantify what switching from FP32 to a 16-bit format buys, separating the memory effect from the throughput effect the way section 1 insisted on. Constructed scenario — every number below is derived from stated assumptions, not measured from a real training run.

Model: 8 billion parameters.

Weight memory at FP32 (4 bytes/parameter):
  8e9 x 4 bytes = 32e9 bytes = 32 GB

Weight memory at FP16 or BF16 (2 bytes/parameter -- identical for
both formats, since this is purely a byte-count question):
  8e9 x 2 bytes = 16e9 bytes = 16 GB

Memory savings from switching to either 16-bit format: 16 GB, or 50%
of the FP32 weight footprint -- identical whichever 16-bit format is
chosen, because this savings is a pure byte-count effect independent
of exponent/mantissa split.

That first calculation is complete and correct, but it answers only the memory question, and it is the same answer for FP16 and BF16 alike — the memory lever does not discriminate between the two formats at all. The throughput lever is a separate claim about the hardware path the arithmetic runs through, not about the bytes stored:

Illustrative relative throughput for a large matrix multiplication,
same GPU generation, same matrix dimensions (a constructed
illustration, not a spec-sheet figure for any named product):

  FP32 arithmetic units:    baseline throughput
  Tensor Cores, FP16 or BF16 inputs, FP32 accumulation:
                            substantially higher than baseline

The exact multiplier is generation- and workload-dependent and is
not something this lesson quotes as a fixed number -- what matters
for the exam is the *direction and cause* of the effect: Tensor
Cores accelerate 16-bit matrix multiplication specifically, and
self-attention's GEMMs are exactly the operation this benefits.

The lesson to generalize: memory savings from 16-bit formats are a format property, identical for FP16 and BF16; throughput gains from Tensor Cores are a hardware property that applies to both 16-bit formats roughly equally, since Tensor Cores accept both as native inputs on modern generations. The decision that actually differs between FP16 and BF16 is not memory or raw throughput — it is the underflow-and-loss-scaling question sections 2 and 3 worked through.

Take two scenarios that call for opposite choices, and work each one to its answer using only the identities established above.

Scenario A: "We are training on A100 GPUs. Our model has shown
gradient-underflow symptoms in early experiments -- a loss that drops
partway, then plateaus with no NaN or inf anywhere in the log -- and
we want to minimize the engineering overhead of tuning a loss-scaling
schedule."

  Reasoning: the underflow symptom described is precisely FP16's named
  failure mode. BF16's wider exponent range is specifically the
  property that avoids this failure mode without a hand-tuned or even
  dynamic loss-scaling schedule. A100 GPUs support BF16 Tensor Core
  operations.
  Choice: BF16, specifically because it sidesteps the underflow
  problem this team has already observed, without the operational
  overhead of getting loss scaling right.

Scenario B: "We are training on an older Tensor Core generation whose
support for BF16 is limited or absent, and no underflow symptoms have
appeared with a working loss-scaling configuration already in place."

  Reasoning: BF16's advantage in section 2 is entirely about avoiding
  the underflow risk that motivates loss scaling in the first place.
  If BF16 support is limited on the target hardware and loss scaling
  is already solving the underflow problem satisfactorily under FP16,
  there is no symptom BF16 would fix that is not already handled.
  Choice: FP16 with the existing loss-scaling configuration, since
  switching formats would add risk (unverified hardware support) to
  fix a problem that is not actually present.

Constructed scenario, with both cases authored to isolate the decision cleanly. The generalizable rule: BF16 is the natural choice when underflow is an observed or anticipated risk and the hardware supports it well; FP16 with working loss scaling remains a perfectly valid choice when underflow is not the actual problem, and switching formats for its own sake is not automatically an improvement.

Read the table's middle rows together and the whole decision compresses to one comparison: FP16 spends its 16 bits on resolution and pays for that with a narrower range, which is why it needs a safeguard against underflow; BF16 spends its 16 bits on range and pays for that with coarser resolution, which is why the same safeguard is usually unnecessary. Neither spending choice is objectively superior — they are answers to different questions about what a given training run's numbers actually need — and the practical answer for a specific job depends on the failure mode actually observed or anticipated, not on a blanket rule that one format always wins.

The identity established in sections 1 through 3 — 16-bit formats for memory and Tensor Core throughput, with FP16's underflow risk and BF16's mitigation of it — applies most directly to training, where a backward pass is actually producing gradients that can underflow in the first place. Inference has no backward pass and therefore no gradient to underflow, which changes which half of this lesson's mechanism is even relevant. A model served at FP16 or BF16 still gets the memory halving and the Tensor Core throughput multiplier on its forward-pass matrix multiplications, but loss scaling has nothing to protect during inference, because there is no loss and no gradient being computed at serving time at all. This is worth stating precisely because a scenario question can describe an inference-serving context and still ask about loss scaling as a distractor: the correct read is that loss scaling is a training-time safeguard, full stop, and offering it as relevant to a pure-inference scenario is offering a real technique attached to the wrong stage of the model's lifecycle.

What inference does still have to decide is which 16-bit format to serve at, and the exponent-versus-mantissa tradeoff from section 2 still applies in a different guise: a served model's numerical behavior under FP16 versus BF16 can differ slightly even with identical trained weights, because the two formats round intermediate activation values differently during the forward pass. In practice this difference is usually small relative to other sources of serving variance, but it is not literally zero, and a team that trained in one 16-bit format and serves in the other should not assume the switch is numerically invisible without checking.

Mixed precision changes how much each individual number costs and how fast the arithmetic on it runs; it does not change how many GPUs are involved or how the model or batch is divided among them, which is exactly the territory M7-01, M7-02, and M7-03 cover. A training job can run tensor parallelism across GPUs, shard its optimizer state via FSDP within its data-parallel group, and choose FP16 or BF16 for the arithmetic each individual GPU performs — all three decisions are independent axes that compose rather than substitute for one another. A team whose actual bottleneck is a layer too wide to fit on one GPU will not solve that by switching from FP16 to BF16, because neither 16-bit format changes how many GPUs a layer's tensors are spread across; that is tensor parallelism's job, not mixed precision's. Equally, a team whose actual bottleneck is FP16 gradients silently underflowing will not solve that by adding more tensor-parallel or pipeline-parallel GPUs; switching to BF16 or adding loss scaling is the lever that addresses that specific failure mode. Matching the lever to the actual symptom, the discipline this whole module keeps returning to, applies here exactly as it does to the parallelism and memory-sharding choices in the preceding lessons.

There is one place these axes genuinely interact rather than staying fully independent: tensor parallelism's all-reduce, described in M7-02, communicates a tensor whose size depends directly on the precision it is stored at. An all-reduce over BF16 or FP16 gradients moves half the bytes an equivalent FP32 all-reduce would move, so choosing a 16-bit format also reduces the communication volume tensor parallelism (and data parallelism's own gradient synchronization) has to pay for on every step, on top of the compute-throughput benefit this lesson centers on. That is a genuine compounding benefit across the two decisions, not a coincidence — but it does not change which decision addresses which symptom: precision reduces the bytes moved per collective operation, while the parallelism strategy itself decides how many such operations happen and where the communication has to travel across the interconnect.

Domain 7, GPU Acceleration and Optimization, is 14% of the NCP-GENL blueprint, the second-largest domain behind Model Optimization's 17%. Objective 7.3 is specifically about throughput on hardware, and [GROUND TRUTH] (Sources/ncp-genl/domain-7-gpu-acceleration.md) frames self-attention's GEMM-dominated computation and Tensor Core throughput as central to that objective. The domain's own named trap is exact and narrow: "FP16 without loss scaling risks gradient underflow; BF16 has a wider exponent range and often needs no loss scaling" — a scenario question is far more likely to test whether you can predict which format needs which safeguard than to ask you to recite either format's bit layout in isolation.

How the question tends to be phrased

Expect a scenario describing a training run's symptom — a loss that plateaus early with no error, or a question about which format is the default fast path on A100/H100 — and asking which format or safeguard addresses it. "A team trains in FP16 and observes a partial loss drop followed by a flat plateau with no NaN; what is the most likely missing safeguard?" keys to loss scaling. "Which 16-bit format is less likely to require loss scaling, and why?" keys to BF16, with the reason (wider exponent range) as the part that actually earns the point, since naming the format alone without the mechanism is a shallower answer than the scenario is testing for.

What the distractors typically look like

Expect FP32 offered as a "safer" choice that avoids the whole issue — true in the narrow sense that FP32 has no underflow risk of this kind, but wrong as an answer to a throughput or memory question, since FP32 forfeits both the memory halving and the Tensor Core throughput multiplier this lesson describes. Expect BF16 offered as though it required loss scaling exactly like FP16 does — a real technique's safeguard, misapplied to a format whose defining property is reducing the need for it. And expect Tensor Cores described as software rather than hardware, or as something usable on any 16-bit computation regardless of GPU generation — a real acceleration path, detached from the hardware generation it actually depends on.

No, even though both involve using fewer bits per value, and the resemblance is exactly why the two get conflated. Mixed precision, as this lesson defines it, is a training- and inference-time compute format decision: it runs the forward pass, loss computation, and (during training) the backward pass in FP16 or BF16 instead of FP32, and it is concerned with representable range and rounding behavior at 16 bits specifically, on hardware that has a dedicated fast path (Tensor Cores) for exactly those two 16-bit formats. Quantization, covered as its own topic in M4-01, typically pushes weights down to 8-bit or 4-bit representations, well below the 16-bit formats this lesson discusses, and it is usually applied after training is complete (post-training quantization) or with a training-aware simulation of the lower precision (quantization-aware training) specifically to shrink a model for deployment, often accepting a measured accuracy cost that mixed precision, done correctly with its safeguards in place, does not have to accept at all. A model can use both: BF16 compute during training, and then INT8 or INT4 quantization of the trained weights for serving — the two decisions sit at different points in a model's lifecycle and are evaluated by different criteria, so treating a mixed-precision choice and a quantization choice as the same lever is a category error even though both are, in the broadest sense, "using fewer bits than FP32."

• FP16/BF16 on Tensor Cores is the default fast path on A100/H100, and the throughput gain is a hardware property specific to Tensor Cores, separate from the memory halving both 16-bit formats provide identically.
• FP16 risks small gradients underflowing to zero without loss scaling, because its exponent range is narrower than FP32's; loss scaling multiplies the loss before the backward pass and divides the update back down afterward, protecting the gradient in transit without changing the final result.
• BF16's wider exponent range often removes the need for loss scaling, but "often" is not "never" — BF16 trades resolution (a shorter mantissa) for that wider range, and is a genuine tradeoff against FP16, not a strict improvement.
• Memory savings are identical for both 16-bit formats; the decision that actually differs between FP16 and BF16 is the underflow-and-loss-scaling question, not byte count.
• Mixed precision changes per-value cost and arithmetic speed; it does not change how a model or batch is divided across GPUs — that is the separate territory M7-01 through M7-03 cover, and the two kinds of decisions compose rather than substitute for each other.
• Domain 7 is 14% of the NCP-GENL blueprint, second only to Model Optimization's 17%, and objective 7.3's throughput focus is exactly where this lesson's format-versus-safeguard distinctions are tested.

Precision decides how cheap each individual number is to store and compute with; it says nothing about how large a batch a training run can effectively process under a fixed memory ceiling. M7-05 picks that question up next: gradient accumulation, the technique that lets a training run reach a large effective batch size — per-device batch size times accumulation steps — without requiring more GPUs or more memory, and the companion trap that accumulation does not reduce total compute even though it does relieve a memory constraint.