GPU Memory Requirements for Training an LLM

Training memory is the sum of four contributions, and any capacity plan that omits one of them is wrong:

Inference needs only the first term plus a KV cache (12-05). Training needs all four. The ratio between them is the whole story of why fine-tuning is hardware-constrained in a way that inference is not.

Two supporting facts before the arithmetic. First, bytes per value depends on numeric precision: fp32 is 4 bytes, and fp16/bf16 are 2 bytes each. The differences between those formats — bf16's wider exponent range versus fp16's finer precision — are 12-01, and here we only need the byte counts. Second, GB and GiB differ by about 7%, and GPU capacity is usually advertised in the decimal sense while allocators report the binary sense. That 7% is the difference between a run that fits and a run that OOMs at 98% utilisation, so both are shown throughout. M0.4 is the unit discipline this lesson leans on.

L1 — The intuition: four copies, not one

The mental model that makes this stick: for every number in the model, a full-parameter training run is holding several numbers.

• One for the weight itself.
• One for its gradient, so the optimizer knows which way to move.
• One or two more for the optimizer's memory of past gradients, because Adam is adaptive and tracks a running mean and a running variance per parameter.
• Possibly one more, a high-precision master copy, because mixed-precision training keeps an fp32 reference to avoid accumulating rounding error in tiny updates.

So the question is never "does the model fit?" It is "does the model fit five or six times over, plus workspace?" And that reframing is what makes the LoRA answer obvious in advance: if you can make most of those copies unnecessary, the problem dissolves.

L2 — The per-parameter byte budget

Work in bytes per parameter, then multiply by parameter count at the end. This is the most transferable form of the calculation because it is model-size-independent.

Case A — full fine-tune, fp32 everything, Adam:

weights            4 bytes  (fp32)
gradients          4 bytes  (fp32)
Adam moment 1      4 bytes  (fp32, running mean)
Adam moment 2      4 bytes  (fp32, running variance)
──────────────────────────────
total             16 bytes per parameter

Case B — full fine-tune, mixed precision (bf16 compute), Adam, no fp32 master copy:

weights            2 bytes  (bf16)
gradients          2 bytes  (bf16)
Adam moment 1      4 bytes  (fp32 — moments are kept in fp32 for stability)
Adam moment 2      4 bytes  (fp32)
──────────────────────────────
total             12 bytes per parameter

Case C — full fine-tune, mixed precision with an fp32 master weight copy, Adam:

bf16 weights       2 bytes
bf16 gradients     2 bytes
fp32 master copy   4 bytes
Adam moment 1      4 bytes
Adam moment 2      4 bytes
──────────────────────────────
total             16 bytes per parameter

Case D — SGD with momentum instead of Adam, mixed precision:

bf16 weights       2 bytes
bf16 gradients     2 bytes
momentum buffer    4 bytes  (one state tensor, not two)
──────────────────────────────
total              8 bytes per parameter

Case E — inference only, bf16:

bf16 weights       2 bytes
──────────────────────────────
total              2 bytes per parameter   (+ KV cache, separately)

Two conclusions worth memorising from this table. Full-parameter training under Adam costs roughly six to eight times inference memory for the same model, depending on which mixed-precision recipe you use. And the optimizer, not the weights, is the largest single contributor in every training case: 8 of the 12 bytes in Case B, 12 of the 16 in Case C. Anyone who says "the model is 14 GB so a 24 GB card is fine" has costed Case E and is about to run Case C.

L3 — Activations, and why they are the term that surprises people

The four terms above are static: once allocated, they do not change size during the run. Activations are dynamic, and they are the reason two runs of the same model on the same GPU can have wildly different peak memory.

During the forward pass, backpropagation requires the intermediate outputs of each layer in order to compute that layer's gradients (01-06). So those intermediates are retained rather than discarded. Their size scales, to a first approximation, as:

activation memory ≈ batch_size × sequence_length × hidden_size × num_layers × bytes_per_value × k

where k is an implementation-dependent multiplier accounting for how many tensors per layer the framework saves — attention scores, feed-forward intermediates, layer-norm inputs, and so on. k is not a clean constant and depends heavily on the framework, whether fused kernels are used, and whether attention is computed with a memory-efficient implementation. Treat any specific k as an estimate to be calibrated empirically, not as a published figure.

What matters for prediction is the proportionalities, and they are exact:

Gradient checkpointing (also called activation recomputation) is the most important lever here. Instead of retaining every layer's activations, the framework retains a subset of checkpoints and recomputes the rest during the backward pass. The memory saving is large; the cost is roughly an extra forward pass worth of compute, commonly cited as something like a 20–30% slowdown — a figure that varies by model and implementation and should be measured rather than assumed.

Gradient accumulation is the companion lever and works on a different axis. It lets you get the statistical effect of a large batch while only ever holding a small one in memory: run several small forward/backward passes, sum the gradients, and step the optimizer once. Effective batch size goes up, activation memory does not.

effective_batch = micro_batch × grad_accum_steps × num_devices

This is the standard reconciliation between "the recipe says batch size 64" and "my card holds 2."

Two more L3 items that appear in real capacity planning and in exam distractors:

• Optimizer state sharding (the ZeRO family of techniques) splits the optimizer state, gradients, and optionally the weights across data-parallel workers, so each device holds only a fraction. This reduces per-device memory without changing the total. It is a distributed-training technique and belongs with 12-04.
• 8-bit optimizers store Adam's moments in 8-bit rather than 32-bit, cutting the 8-byte moment cost to roughly 2 bytes per parameter. This is a real and widely used memory lever, and it attacks the term that dominates.
• Framework overhead and fragmentation are real and not negligible. CUDA context, kernel workspaces, cached blocks the allocator has not returned, and temporary buffers all consume memory that does not appear in your arithmetic. Plan a headroom margin rather than budgeting to 100%.

The column that matters most is "scales with." Weights scale with total parameters and cannot be reduced except by quantisation or a smaller model. Gradients and optimizer state scale with trainable parameters, and trainable parameters are a design choice. That is the entire mechanism by which PEFT works, and it is a memory argument before it is anything else.

An important distinction the exam can test: the KV cache is an inference concern, not a training concern. During training you process a whole sequence in parallel with a known target, so there is no incremental decode loop to cache for. During generation you produce one token at a time and cache the keys and values to avoid recomputing the whole prefix. Confusing the two is a common error, and the two terms grow along different axes — training activations grow with batch and sequence, KV cache grows with generated length.

This is the arithmetic the whole module hinges on. All assumptions are stated; the numbers are derived, not quoted.

Assumptions:

Parameter count P        = 7,000,000,000  (a "7B" model, rounded)
Precision                = bf16 for weights and gradients (2 bytes)
Optimizer                = Adam, moments in fp32 (4 bytes each)
Master weight copy       = none (Case B above)
1 GB                     = 1e9 bytes;  1 GiB = 1,073,741,824 bytes
Available device memory  = 16 GB advertised → ~14.9 GiB, minus ~1 GiB
                           for CUDA context and framework overhead

Step 1 — the weight term.

7e9 params × 2 bytes = 14e9 bytes = 14.0 GB = 13.04 GiB

Already this alone consumes nearly the whole 16 GB card, before a single gradient exists. Inference would just about fit. Training has not started.

Step 2 — the gradient term.

7e9 trainable params × 2 bytes = 14e9 bytes = 14.0 GB = 13.04 GiB
Running total: 28.0 GB

Step 3 — the optimizer-state term. Adam holds two moments per parameter, both fp32:

7e9 × 4 bytes (moment 1) = 28e9 bytes = 28.0 GB
7e9 × 4 bytes (moment 2) = 28e9 bytes = 28.0 GB
Optimizer subtotal        = 56.0 GB = 52.15 GiB
Running total: 84.0 GB

Step 4 — the static total.

weights        14.0 GB   (16.7%)
gradients      14.0 GB   (16.7%)
optimizer      56.0 GB   (66.7%)
────────────────────────
static total   84.0 GB = 78.23 GiB

Two-thirds of the requirement is optimizer state. That is the term to attack.

Step 5 — add activations. Even a modest configuration adds several GB. Take a deliberately small setting: micro-batch 1, sequence length 1,024, and suppose the model has 32 layers and hidden size 4,096. The dominant per-layer term is on the order of batch × seq × hidden × bytes multiplied by however many tensors the framework saves:

per layer, per saved tensor:
  1 × 1,024 × 4,096 × 2 bytes = 8.39e6 bytes ≈ 8.4 MB
over 32 layers, per saved tensor:
  32 × 8.4 MB ≈ 268 MB

If the framework saves on the order of ten such tensors per layer — an illustrative multiplier, not a measured one — activations land in the low single-digit GB for this tiny configuration, and they scale linearly with both batch size and sequence length. Raise the micro-batch to 8 and the sequence to 4,096 and the same estimate grows by a factor of 32.

Step 6 — compare against the hardware.

Required (static, batch 1)        ≈ 84.0 GB   (78.2 GiB)
Required with modest activations  ≈ 87 GB+
Available on a 16 GB card         ≈ 14 GiB usable
Shortfall                         ≈ 5.6× over capacity
Available on an 80 GB card        ≈ 74.5 GiB usable
Shortfall on 80 GB                still short, before activations

The 7B full fine-tune does not fit on a 16 GB GPU. It does not comfortably fit on an 80 GB GPU either. That is the impossible number, and it is derived from four multiplications.

Step 7 — apply the memory levers one at a time and re-total.

Baseline (Case B, Adam, bf16)                          84.0 GB
→ SGD with momentum instead of Adam (8 B/param)        56.0 GB   (-33%)
→ 8-bit Adam moments (~2 B each instead of 4)          42.0 GB   (-50%)
→ Gradient checkpointing                     activations only, ~unchanged static
→ Optimizer sharding across 8 devices        ~24.5 GB per device
→ LoRA: 0.1% trainable parameters                    see step 8

Step 8 — the LoRA arithmetic, which is the resolution. Suppose an adapter configuration whose trainable parameters are 0.1% of the model — a constructed illustration chosen for clean arithmetic, not a recommended or measured ratio.

Trainable params = 0.001 × 7e9 = 7,000,000  (7 M)

frozen weights   7e9   × 2 bytes = 14.00  GB   ← unchanged; still must be resident
adapter weights  7e6   × 2 bytes =  0.014 GB
gradients        7e6   × 2 bytes =  0.014 GB   ← was 14.0 GB
Adam moment 1    7e6   × 4 bytes =  0.028 GB   ← was 28.0 GB
Adam moment 2    7e6   × 4 bytes =  0.028 GB   ← was 28.0 GB
──────────────────────────────────────────────
static total                     ≈ 14.08 GB = 13.12 GiB

From 84.0 GB to about 14.1 GB — a factor of roughly six — by changing one thing: which parameters are allowed to move. The three terms that scale with trainable parameters collapsed by a factor of a thousand. The weight term did not budge, because the frozen weights still have to be in memory to run the forward pass.

Step 9 — close the last gap. 14.1 GB static plus activations still crowds a 16 GB card. Two further levers finish the job:

→ Gradient checkpointing: activations drop substantially
→ Quantise the frozen base weights to 4-bit:
     7e9 × 0.5 bytes ≈ 3.5 GB   (from 14.0 GB)
     new static total ≈ 3.6 GB

Loading the frozen base in 4-bit while training a bf16 adapter on top of it is the recipe usually called QLoRA, and the arithmetic above is why it exists: once gradients and optimizer state are gone, the frozen weights become the dominant term, so the next thing to attack is the bytes per frozen weight. 12-02 is the quantisation mechanics and its accuracy trade-offs.

Step 10 — state the conclusion in the form the exam wants. A 7B full fine-tune under Adam needs on the order of 84 GB and cannot run on a 16 GB device. The same model can be adapted on that device because gradients and optimizer state scale with trainable parameters, and a parameter-efficient method makes that count tiny.

You launched a run and it died with an out-of-memory error. Which term blew up, and what do you change?

The ordering in the second row is deliberate. When gradients and optimizer state are what killed the run, the highest-leverage change by far is reducing the trainable parameter count — a factor-of-hundreds reduction — rather than shaving bytes off each state slot for a factor of two. Reach for PEFT first and micro-optimisations second.

The NCA-GENL blueprint asks the associate to identify system data, hardware, or software components required to meet user needs, and to assist in deployment and evaluation of model scalability, performance, and reliability under the supervision of a senior team member. Both objectives are directly about capacity: can this run on the hardware we have, and if not, what changes? Memory arithmetic is the most concrete, most examinable form of that question.

It also supplies the quantitative justification for the customisation ladder. Any candidate who can compute that gradients and optimizer state dominate full fine-tuning memory can derive why PEFT exists, rather than memorising that it is efficient. That derivation is the single highest-value insight in this module.

A calibration note that matters for how you study this: candidate reports for this exam consistently describe deep GPU-hardware and spec-sheet detail as overkill and absent. So learn the structure of the calculation and the proportionalities, not memory bandwidth figures or per-SKU capacities. Know that training needs far more than inference and why; know which term dominates; know that gradients and optimizer state track trainable parameters. Do not memorise HBM capacities.

Question phrasings to expect:

• "Which of the following contributes to GPU memory usage during training but not during inference?" — gradients, optimizer state, stored activations.
• "A 7B-parameter model occupies 14 GB in half precision. Why does full fine-tuning require substantially more?" — gradients, two Adam moment tensors, and activations.
• "Which component typically consumes the most memory during full fine-tuning with Adam?" — optimizer state.
• "Which technique reduces activation memory at the cost of extra computation?" — gradient checkpointing / activation recomputation.
• "How can a team obtain the effect of a larger batch size without more memory?" — gradient accumulation.
• "Why does LoRA reduce training memory so dramatically?" — gradients and optimizer state scale with trainable parameters, which LoRA makes tiny; frozen weights need no gradient or optimizer slot.
• "What is the KV cache and when does it consume memory?" — inference-time generation, not training.

Distractor families:

Under a full-parameter fine-tune with Adam and bf16 compute, on the order of 84 GB of static memory before activations — 14 GB of weights, 14 GB of gradients, and 56 GB of optimizer state — which is why this configuration is a multi-GPU job rather than a single-card one. Add an fp32 master weight copy and it rises to about 112 GB. Swap Adam for SGD with momentum and it falls to about 56 GB. Use 8-bit optimizer states and it falls to roughly 42 GB.

With a parameter-efficient method it is a completely different question. If the adapter's trainable parameters are a small fraction of a percent of the model, the gradient and optimizer terms become rounding errors and the total is dominated by the frozen weights: roughly 14 GB at bf16, or roughly 3.5 GB if the frozen base is loaded in 4-bit. That is a single-GPU job on modest hardware.

The general formula to carry into the exam, with every symbol defined:

static_memory ≈ P_total × B_weight
              + P_trainable × B_grad
              + P_trainable × B_state × N_state

P_total     = total parameters
P_trainable = parameters receiving gradient updates
B_weight    = bytes per weight (2 for bf16, 4 for fp32, ~0.5 for 4-bit)
B_grad      = bytes per gradient (usually matches compute precision)
B_state     = bytes per optimizer state slot (4 for fp32, ~1–2 for 8-bit)
N_state     = state tensors per parameter (Adam 2, SGD+momentum 1, SGD 0)

then add activations, which scale with
  batch × sequence × hidden × layers × bytes
and are reduced by gradient checkpointing.

Every number in this lesson is that formula with different substitutions. Memorise the formula, not the numbers.

Because inference is a forward pass and training is a forward pass plus a backward pass plus a parameter update, and each of the latter two requires state that inference simply does not create.

The forward pass alone needs, at any instant, only the current layer's weights and the tensor flowing through it — intermediate results can be discarded as soon as the next layer consumes them. The backward pass cannot discard them, because computing a layer's gradient requires that layer's inputs. So training retains activations across the entire depth of the network. Then the gradient itself is one number per trainable parameter — a full-size copy of the model. Then the optimizer, if it is adaptive, keeps a running summary of each parameter's gradient history — one or two more full-size copies.

That is three to four extra model-sized tensors plus a depth's worth of activations, against inference's one model plus a KV cache. Six-to-eight-fold is the honest ratio for Adam full fine-tuning, and it is why the hardware conversation for training and the hardware conversation for serving are different conversations with different answers.

No — it explicitly trades compute for memory, and that trade is the point. Without checkpointing, every activation needed by the backward pass is stored. With checkpointing, only a subset of layer boundaries are stored and the intermediate activations are recomputed on demand during the backward pass. Memory falls substantially; wall-clock time rises by roughly the cost of an additional forward pass.

Whether that trade is good depends on which resource is binding. If you are memory-bound — the run will not start otherwise — a slower run is infinitely better than no run, and checkpointing is obviously correct. If you have memory headroom and are paying by the GPU-hour, checkpointing is a tax. The correct posture is to enable it when you need it and turn it off when you do not, and to know which situation you are in by having computed the arithmetic first.

Note also which term it addresses. Checkpointing touches activations only. It does nothing for weights, gradients, or optimizer state. So if the arithmetic in section 4 says you are 70 GB short on static memory, checkpointing will not save you — the lever you need is PEFT or sharding. Matching the lever to the term is the whole diagnostic skill.

Plain stochastic gradient descent, because it keeps no per-parameter state at all — just weights and gradients. Adding momentum costs one state tensor per parameter. Adam and its variants cost two, because they track both a running mean and a running variance of the gradient per parameter. In bytes per parameter, with fp32 state:

plain SGD             weights + gradients + 0 state
SGD with momentum     weights + gradients + 4 bytes
Adam / AdamW          weights + gradients + 8 bytes
8-bit Adam            weights + gradients + ~2 bytes

Memory is not the only consideration, which is why Adam remains the default despite being the most expensive: its per-parameter adaptive step sizes make it far more forgiving on the learning rate, and learning-rate sensitivity is exactly the kind of problem that costs you a week of failed runs. 01-06 covers the optimiser mechanics.

The pragmatic hierarchy when memory is tight: reduce trainable parameters first (PEFT), then quantise optimizer state (8-bit Adam), then consider a cheaper optimizer, and only then compromise the batch configuration. The first lever is worth a factor of hundreds; the others are worth factors of two.

• Four terms, always: weights, gradients, optimizer state, activations. Any estimate that counts only weights is costing inference and calling it training.
• Full-parameter training with Adam costs roughly 12 to 16 bytes per parameter of static memory — six to eight times inference — before a single activation is stored.
• Optimizer state is the dominant term in a full fine-tune: 8 of 12 bytes per parameter in a bf16+Adam recipe, two-thirds of the total.
• A 7B full fine-tune needs on the order of 84 GB and does not fit on a 16 GB device, or comfortably on an 80 GB one. That is derived arithmetic, not a quoted figure.
• Gradients and optimizer state scale with trainable parameters, not total parameters. This is the sentence to carry into the exam: it is the quantitative argument for LoRA and PEFT in its entirety.
• Freezing most of the model collapses three of the four terms. In the constructed 0.1%-adapter example the static total falls from 84 GB to about 14 GB — a factor of six — leaving the frozen weights as the new dominant term.
• Activations scale with batch size, sequence length, layers, and precision, and gradient checkpointing reduces them by recomputing rather than storing — a deliberate compute-for-memory trade, not a free win.
• Gradient accumulation raises effective batch size without raising peak memory; it does not reduce activations.
• Match the lever to the term. Checkpointing fixes activations. PEFT and sharding fix gradients and optimizer state. Quantisation fixes weights. Learning rate fixes nothing memory-related.
• Mind units and headroom. GB versus GiB is a 7.4% error, and framework overhead plus fragmentation is real. Do not budget to 100%.

The arithmetic above has one loose end and it is the biggest one in the module: three of the four memory terms scale with trainable parameters, so what exactly do you train if you are not training the weights? The answer is a pair of small low-rank matrices placed alongside each frozen weight matrix, and it turns out to recover most of a full fine-tune's behavioural effect at a fraction of the memory, the storage, and the risk. Next: 11-05 builds LoRA from the same first principles, defines rank and alpha and target modules, and has you predict a run's peak memory before launching it.