KV Caching: The Primary Latency Lever in Autoregressive LLM Inference

KV caching is a trade, not a discount: it spends memory to buy speed. Every decoding step in an autoregressive model needs the key and value tensors of every token that came before it, and rather than recompute those tensors from scratch on every step, the model stores them the first time they are produced and reads them back on every subsequent step. That storage is the KV cache, and keeping it resident in GPU memory is what makes generating a long response take roughly linear time instead of quadratic time. The exam's favorite inversion of this fact is the claim that "KV caching reduces memory" — it does the opposite. It reduces recomputation, and recomputation is compute, not memory. The cache itself is an additional memory cost that did not exist before the optimization, and on long-context, high-concurrency serving it is often the dominant memory cost on the GPU, larger than the model's own weights.

This lesson treats KV caching as what the source material calls it: the primary latency lever for decoder inference, one of the three runtime optimizations named alongside streaming attention and TensorRT compilation. It also sits deliberately close to quantization, distillation, and pruning, because all four are levers in the same domain and a serving team routinely stacks them. Getting KV caching right — what it stores, why it is safe to reuse, how its size grows, and what it costs — is worth disproportionate exam points, because Model Optimization is the single largest domain on the NCP-GENL blueprint at 17%, ahead of every other domain including fine-tuning and evaluation.

1. What decoding would cost without a KV cache

Autoregressive generation produces one token at a time, and each new token depends on every token that came before it through self-attention. To generate token number t, the model's attention mechanism must compare that token's query against the keys of tokens 1 through t, and combine values from all of them. That dependency is unavoidable — it is what makes the model coherent over a growing context. The question is not whether attention looks at the whole prefix; it always does. The question is whether the model is forced to recompute the keys and values of that entire prefix every single time it wants to produce one more token.

Without a cache, that is exactly what happens. To emit token 1, the model runs a forward pass over 1 token. To emit token 2, it does not just process the new token — a naive implementation reprocesses the whole sequence so far, meaning it recomputes keys and values for token 1 all over again just to get to token 2. To emit token 500, it reprocesses all 499 prior tokens. Summed over an entire generated response, the total number of token-positions processed grows like the sum 1 + 2 + 3 + ... + n, which is proportional to n². A response of 1,000 tokens costs on the order of 500,000 token-position evaluations rather than 1,000. That quadratic blowup is the cost this lesson exists to eliminate, and it is why an un-cached decoder becomes unusable well before responses reach any length a real product would want to produce.

It is worth being precise about what actually gets wasted. The wasted work is not the attention computation for the new token against the old ones — that computation is required no matter what. The waste is recomputing the keys and values of the old tokens themselves, over and over, when those tensors have already been computed once and have not changed. That distinction — attention against history is unavoidable, recomputing history's keys and values is avoidable — is the entire idea behind the optimization, and it is worth holding onto before the next section explains why it works.

Why this is specific to decoder inference, not training

During training, the model sees a full sequence at once and computes every position's keys, values, and attention in a single parallel pass — there is no step-by-step generation, so there is nothing to cache in the same sense. The quadratic-recomputation problem above is purely an inference-time, autoregressive-generation problem: it only exists because generation is inherently sequential, one token revealed at a time, with no way to know token t+1 before token t has been produced and sampled. Encoder-only models that read a full input and immediately return an output (classification, embedding) never hit this issue either, for the same reason — they make one pass, not one pass per output token. KV caching is therefore a decoder-inference optimization specifically, which is exactly how the source material frames it: a lever for autoregressive decoding.

2. What the KV cache actually stores, and why recomputation is avoidable

Self-attention produces three vectors per token per layer: a query, a key, and a value. The query is used once — to score the current token against everything preceding it — and then it is discarded. The key and value are different: because of the causal structure of autoregressive attention, the key and value computed for token n depend only on token n and the tokens before it. They never depend on anything that comes later. That means once token n's key and value are computed, they are correct forever; no future token can retroactively change them.

That single fact is what makes caching legal. If a token's key and value could change depending on what gets generated afterward, caching them would produce wrong answers. Because they cannot change, storing them once and reading them back on every later step produces the exact same result as recomputing them every time — the cache is not an approximation, it is an exact elimination of redundant, deterministic work.

So the mechanism is: on each decoding step, the model computes the query, key, and value for exactly the one new token, appends that new key and value into the cache, then runs attention using the new query against the entire cache of keys and values accumulated so far. The forward pass processes one token's worth of new arithmetic; the cache supplies everything else it needs to know about history. This is why decoding cost per generated token becomes roughly constant instead of growing with position — the model still "looks at" the whole prefix through attention, but it never has to reconstruct that prefix's representations again.

One clarification that closes a common gap: KV caching does not change what the model attends to, and it does not shorten the sequence the model reasons over. A model with a 1,000-token KV cache is still attending over all 1,000 positions on every step — the attention computation against the cache still scales with how much is cached. What the cache removes is the redundant recomputation of keys and values for tokens already processed, not the ongoing cost of attention over history. Confusing "the cache removes recomputation" with "the cache removes the cost of long context" is a real source of exam-style traps, addressed directly in the common-mistakes table below.

3. How KV cache size scales: layers, batch, sequence, and heads

The KV cache is not one tensor; it is one pair of tensors — keys and values — per attention head, per layer, per token, per sequence in the batch. That multiplication is the whole scaling story, and every term in it maps to something a serving engineer actually controls or must plan around.

• Layers. Every transformer layer has its own attention mechanism and therefore its own keys and values. A 40-layer model caches 40 independent copies of "history," not one shared copy. Doubling the layer count doubles the cache, all else equal.
• Attention heads. Within a layer, attention is typically split into multiple heads, each with its own smaller key and value vectors. The per-token cache cost for one layer is proportional to the number of heads whose keys and values are stored times the width of each head's vector — and those two multiply back out to the model's hidden size in the standard case where every head gets its own key and value.
• Sequence length. Every token generated (and every prompt token processed) adds one more entry to the cache, for every layer, for every head. This is the term that grows during a single request, unlike the other terms, which are fixed once the model and the batch are chosen.
• Batch size. Each concurrent request carries its own independent cache — nothing about one user's conversation can be shared with another's, because their token sequences differ. Serving 32 conversations at once means 32 separate caches sitting in memory simultaneously, not one cache 32 times more efficient.

Put those four together and the growth-rate intuition is: KV cache size is proportional to layers × heads-worth-of-key/value-width × sequence length × batch size. None of these four terms is optional and none of them can be reasoned about in isolation — a change in any one of them scales the cache linearly, and because sequence length and batch size are both runtime, request-driven quantities rather than fixed architecture choices, the cache is the one major memory consumer in LLM serving that is not known in advance from the model file alone. Weight memory is a constant you can read off a config. Cache memory is a moving target that depends on how the model is actually used in production, which is precisely why long-context serving is described as a capacity-planning problem rather than a one-time sizing exercise.

4. Worked example: sizing a KV cache at production context lengths

Numbers make the scaling claim concrete. Take an illustrative 13-billion-parameter-class decoder: 40 transformer layers, hidden size 5,120, 40 attention heads (so each head is 5,120 ÷ 40 = 128 wide), served at BF16 precision (2 bytes per stored number) for both weights and cache.

Step 1 — the per-token cache cost, across the whole model.

per layer, per token: 2 (key + value) x 40 heads x 128 head-width x 2 bytes
                     = 2 x 5,120 x 2 bytes  =  20,480 bytes
across 40 layers:      20,480 x 40          = 819,200 bytes  ~  0.8 MB per token

The shortcut worth memorizing: per-token cache cost = 2 × layers × hidden size × bytes-per-element. Everything else is multiplication by however many tokens and however many concurrent requests you are serving.

Step 2 — one request at a realistic context length.

4,096-token context:   4,096 x 0.8 MB  ~   3.3 GB   for this one conversation's cache

Step 3 — the same request pushed to a long-context limit.

32,768-token context (8x longer):  32,768 x 0.8 MB  ~  26.2 GB   for this one conversation's cache

Eight times the context produced eight times the cache — linear, as the scaling rule predicts — but the absolute number went from a few gigabytes to a number comparable to the entire weight file of the model.

Step 4 — add concurrency.

weights at BF16 (13B params x 2 bytes)        ~  26 GB   (fixed, one copy, shared by everyone)

8 concurrent requests at 4,096-token context:   8 x 3.3 GB   ~  26.4 GB of cache
8 concurrent requests at 32,768-token context:  8 x 26.2 GB  ~ 209.6 GB of cache

Read that last line as an engineer, not an arithmetician. At the shorter context, the cache alone, summed across just eight users, already matches the weight file's own footprint — a single 80 GB accelerator would be holding roughly as much cache as model. At the longer context, the cache for the same eight users exceeds what a single high-end GPU can hold at all, before the weights are even loaded. This is the exact mechanism behind the claim that long-context serving is memory-bound rather than compute-bound: the GPU does not run out of arithmetic throughput first, it runs out of bytes to hold the keys and values that make correct generation possible. Every input to that arithmetic — layer count, hidden size, head count, precision — is published in a model's configuration file, so this is a calculation a serving team can and should do before committing to a context-length limit, not after an outage.

5. The memory-for-speed trade: why KV caching never reduces memory

This is the point the exam tests most directly, because it is the point most people get backwards on first exposure. KV caching's entire benefit is a compute benefit: it eliminates the quadratic recomputation described in section 1 and replaces it with linear, one-token-at-a-time work. That benefit is real and large — the difference between roughly n²/2 and n token-position evaluations for a response of length n is enormous once n reaches the hundreds or thousands.

But that benefit is not free, and it is not paid for in memory saved — it is paid for in memory spent. Before caching existed as a design choice, there was no persistent per-request GPU memory allocation holding keys and values; recomputing everything every step is wasteful of compute but costs nothing extra to store, because nothing from history sticks around between steps. The moment you introduce a cache, you introduce a growing, per-request memory allocation that did not exist in the naive approach. Section 4's arithmetic shows exactly how large that allocation gets: comparable to, and at long contexts and high concurrency, far exceeding the model's own weight memory.

So the correct one-sentence framing, and the one worth having ready for a scenario question, is: KV caching spends memory to buy speed. It does not, and cannot, reduce memory — the exam's stated common trap is exactly this inversion. Anyone who describes KV caching as a memory-saving technique has the mechanism backwards; they have likely confused "it avoids wasted compute" with "it avoids memory use," which are opposite claims. If a scenario item offers "KV caching reduces memory footprint" as an option, it is describing a real technique's benefit in the wrong currency, and that is the signature of this domain's distractor style: a real technique, misapplied to the wrong tradeoff axis.

The corollary worth stating explicitly: because the cache is a memory cost, not a memory saving, it competes directly with everything else that wants GPU memory — the weights, the activation workspace, and any headroom needed for batching more requests. A team that adds KV caching and then acts surprised that memory pressure went up has misunderstood what the optimization does. The surprise should run the other way: without the cache, you would be paying an even larger cost, just in GPU-seconds instead of GPU-bytes.

6. KV caching alongside quantization, distillation, and pruning

Model Optimization is not one technique tested in isolation; it is a toolbox, and KV caching is the runtime-facing member of a set that also includes quantization (post-training quantization, quantization-aware training, and GPTQ), knowledge distillation, and structured sparsity. These interact, and the exam's scenario questions often reward knowing which lever addresses which constraint.

Quantizing the cache itself. The same bytes-per-element term that appears in weight-memory arithmetic also appears in the KV cache formula from section 4. Storing cached keys and values at 8-bit precision instead of 16-bit halves the cache's memory footprint for the same context length and batch size, at some measurable accuracy cost that must be evaluated rather than assumed — exactly the discipline objective 4.2 demands for every optimization in this domain. This is a distinct decision from quantizing the weights: a team can run full-precision weights with a quantized cache, quantized weights with a full-precision cache, or quantize both, and each combination changes a different part of the memory budget.

Quantizing the weights helps a different bottleneck. Lowering weight precision — the PTQ, QAT, and GPTQ techniques from earlier in this domain — shrinks the model's fixed memory footprint and reduces the bytes that must be streamed from memory on every decoding step, which speeds up the memory-bandwidth-bound decode phase. But it does nothing to the KV cache's own growth with sequence length and batch size. A team whose memory problem is actually a long-context, high-concurrency cache problem will not solve it by quantizing weights alone; the cache term keeps scaling regardless of what precision the weights are stored in. Reading a scenario correctly means identifying whether the stated constraint is the fixed weight footprint or the scaling cache footprint, because the two levers do not substitute for each other.

Distillation and pruning shrink what needs a cache to begin with. A smaller distilled model, or a pruned model exploiting structured 2:4 sparsity for Tensor Core acceleration, has fewer layers or a narrower hidden size — and because both terms sit directly inside the per-token cache formula, a smaller model produces a smaller cache automatically, on top of its other benefits. This is a case where two optimizations compound: shrink the model, and every cached token gets cheaper as a side effect, without touching the KV-caching mechanism itself.

The unifying discipline across all of these, and the one the source material states as a scope note for the whole domain, is that every optimization here is a bargain — memory or latency traded for some accuracy risk (or, for KV caching specifically, memory traded for latency with no accuracy risk at all, since caching is exact, not approximate). Anchor every answer on which constraint is actually binding: memory, latency, accuracy, or hardware.

7. KV caching's runtime neighbors: streaming attention and TensorRT

KV caching does not act alone at inference time. The domain groups it with two other runtime optimizations, and knowing the boundary between the three prevents a common mix-up.

Streaming, or sliding-window, attention addresses a different problem: even with a KV cache eliminating recomputation, the cache itself keeps growing without bound as a conversation gets longer, and attention over an ever-growing cache eventually becomes the bottleneck in its own right. Sliding-window attention bounds the span of history the model actually attends to — and therefore caches — so memory for very long sequences stays controlled rather than growing indefinitely. Where KV caching answers "how do we avoid recomputing history," streaming attention answers "how do we avoid needing to cache all of history in the first place." They are complementary, not competing: a serving stack can cache efficiently and still bound the window it caches.

TensorRT operates at a different layer of the stack entirely. It is NVIDIA's deployment optimizer: it compiles a trained model, fuses kernels, applies precision calibration, and auto-tunes execution for a specific target GPU. It is not itself a caching mechanism and it is not an inference server — a standing exam trap states the distinction plainly: TensorRT optimizes; Triton serves. TensorRT can be part of the pipeline that runs a KV-cache-enabled model efficiently, but "KV caching" and "using TensorRT" answer different questions, and an exam distractor that substitutes one for the other is testing exactly this separation of concerns.

The practical takeaway: if a scenario names a latency problem in autoregressive decoding specifically, the keyed answer is KV caching. If it names an unbounded-memory problem in very long conversations, the keyed answer is more likely streaming or sliding-window attention. If it names compiling or tuning a model for a specific GPU target, the keyed answer is TensorRT — and if it names serving, load balancing, or concurrent model execution, the correct territory is Domain 8's deployment stack, not this one.

8. Why KV caching is the single biggest latency lever for autoregressive inference

Section 1 established the mechanism: without a cache, generating a response of length n costs on the order of n² token-position evaluations; with a cache, it costs on the order of n. No other single optimization in this domain touches that particular exponent. Quantization and pruning make each unit of work cheaper or skip some of it, but they do not change the shape of the cost curve the way caching does — a faster quadratic curve is still a quadratic curve, and it still becomes catastrophic as response length grows. KV caching is the one lever that converts the curve itself from quadratic to linear.

That is the precise sense in which the source material calls it the primary, or single biggest, latency lever for decoder inference: it is not merely the largest improvement among comparable techniques, it is a change in asymptotic behavior that every other latency optimization in this domain is applied on top of, not instead of. A production decoder without a KV cache is not "somewhat slower" at generating a 2,000-token response — it is attempting roughly two million redundant token-position evaluations to do it, which is a difference in kind, not degree. Every serving stack you will encounter professionally assumes KV caching is present as a baseline; the interesting engineering questions — quantize the cache, bound the window, batch more requests, page the memory — all sit on top of that assumption rather than replacing it.

9. Why KV caching is on the NCP-GENL exam

Model Optimization is objectives 4.1 through 4.7, and it is, by the study material's own framing, the biggest domain on the exam at 17% — larger than Fine-Tuning, larger than Evaluation, larger than any other single domain in the ten-domain blueprint. Objective 4.2 specifically requires measuring an accuracy tradeoff for optimization techniques, and while KV caching is unusual among this domain's techniques in costing no accuracy at all (it is an exact optimization, not an approximation), it is squarely covered by objectives concerned with runtime optimization choices for a given hardware and task.

Expect KV caching to appear in a few recurring shapes:

1. Direct identification. "Which technique avoids recomputing attention over the whole prefix at every decoding step?" The keyed answer is KV caching, and the study material's own self-check phrases this almost exactly.
2. The memory-vs-speed inversion. A scenario or statement asserts that KV caching reduces memory. This is the domain's named common trap, and the correct read is the reverse: it uses memory to save recomputation, and the win is speed.
3. Scaling and capacity questions. "What does KV cache size grow with?" tests whether you know the multiplication from section 3 — layers, heads, sequence length, batch size — rather than confusing it with parameter count, which governs weight memory instead.
4. Runtime-optimization discrimination. A question naming KV caching, streaming attention, and TensorRT in the same list and asking which does what is testing exactly the boundary drawn in section 7.
5. Cross-domain tie-ins. A question about serving compute tradeoffs for decoder-only models (a Domain 8 topic) may lean on KV caching as the named mitigation for sequential, one-token-at-a-time generation latency — recognizing the mechanism pays off even outside its home domain.

How the question tends to be phrased

Professional-level items in this domain tend to state a constraint and ask which lever addresses it: "a decoder-only model's response generation is too slow; which single change addresses this most directly?" with KV caching as the keyed answer against distractors like increasing beam width or adding attention heads — both of which either add cost rather than removing it, or require retraining rather than a runtime change. Another recurring shape states a wrong mechanism as fact and asks you to evaluate it — "caching is used to reduce a model's memory footprint during generation" — where the correct response identifies the inversion.

What the distractors typically look like

The standard traps in this domain's style are: offering KV caching as a memory-reduction technique (the named common misconception); offering "next-sentence prediction" or an unrelated pretraining objective as the answer to a decoding-latency question (a real technique, wrong problem entirely); offering "increasing beam width" as a fix for latency, when beam search is itself a compute-for-quality trade that makes generation slower, not faster; and conflating TensorRT's compilation role with Triton's serving role in the same item. Each distractor, true to this domain's house style, is a real, nameable technique attached to the wrong problem.

10. Common mistakes about KV caching and memory

Mistake	What is actually true	Fix
"KV caching reduces memory"	It uses additional GPU memory to avoid recomputation; the benefit it delivers is latency, not memory savings	Say "trades memory for speed," never "saves memory"
Assuming cache size is independent of batch size	Every concurrent request holds its own separate cache; doubling batch size roughly doubles total cache memory at a fixed context length	Budget cache memory as layers × heads × sequence length × batch size, not per-model alone
Believing the cache shortens what the model attends to	Attention still scores against the entire cached history on every step; the cache removes recomputing keys/values, not the cost of attending over them	Distinguish "avoids recomputation" from "avoids long-context attention cost"
Treating KV caching as compute-free	Reading the cache from memory on every step has a real bandwidth cost that grows with cache size, which is why very large caches can themselves slow decoding	Recognize that eliminating recomputation does not make the cache free to use
Confusing KV cache growth with parameter count	Weight memory is fixed by parameter count and precision; KV cache memory is a separate, runtime-scaling quantity tied to sequence length and batch size	Keep the two memory terms — weights and cache — in separate columns of any capacity plan
Assuming weight quantization also shrinks the cache	Quantizing weights changes the fixed model footprint; the cache shrinks only if the cache itself is quantized or the context/batch is reduced	Treat cache quantization as its own, separate decision from weight quantization
Treating TensorRT as the caching mechanism	TensorRT compiles and tunes kernels for a GPU target; it does not itself define KV caching, and it is not a serving engine (that is Triton)	Keep "optimizes" (TensorRT) and "caches" (KV caching) and "serves" (Triton) as three distinct jobs

11. Glossary recap: KV caching terms this lesson introduced

Term	One-line definition
KV cache	Stored key and value tensors for tokens already processed, kept in GPU memory so decoding never recomputes them
Key (K) / Value (V) / Query (Q)	The three attention projections; K and V are cacheable because they never change once computed, Q is used once and discarded
Autoregressive decoding	Generating output one token at a time, each depending on every token produced before it
Recomputation	Re-deriving a value that was already computed and has not changed — the specific waste KV caching eliminates
Memory-for-speed trade	Spending additional memory to reduce computation; the exact framing of what KV caching does, and never the reverse
Streaming / sliding-window attention	Bounding the span of history a model attends to (and therefore caches) to control memory on very long sequences
TensorRT	NVIDIA's inference compiler and optimizer: kernel fusion, precision calibration, and GPU-specific auto-tuning — not a cache, not a server
KV-cache quantization	Storing cached keys and values at reduced precision (for example 8-bit instead of 16-bit) to shrink the cache's own footprint
Per-token cache cost	2 × layers × hidden size × bytes-per-element, the base multiplier that sequence length and batch size then scale
Memory-bound serving	A regime where GPU memory capacity, not arithmetic throughput, is what limits how much can be served — the typical state of long-context, high-concurrency decoding

12. Key takeaways on KV caching

• KV caching is the primary latency lever for autoregressive inference. It converts the cost of generating a response from roughly quadratic in length to roughly linear, by storing keys and values instead of recomputing them at every step.
• It spends memory to buy speed. It never reduces memory — that inversion is this topic's single most tested trap.
• Only keys and values are cached; queries are not, because a query is used once for the current token and discarded, while a key or value never changes once computed under the causal mask.
• Cache size scales with layers × heads × sequence length × batch size. Sequence length and batch size are runtime, request-driven quantities, which is why cache memory cannot be planned from the model file alone the way weight memory can.
• The per-token cache cost is 2 × layers × hidden size × bytes-per-element — a number you can compute from any published model config before deploying anything.
• Quantizing the cache and quantizing the weights are separate decisions that shrink different parts of the memory budget; neither substitutes for the other.
• Streaming attention bounds unbounded cache growth; TensorRT compiles and tunes; Triton serves. Three distinct jobs, frequently tested against each other.
• Model Optimization is 17% of the NCP-GENL blueprint, the largest domain, and KV caching is one of its most directly and repeatedly tested single facts.

13. Next: tensor parallelism vs pipeline parallelism, the GPU acceleration domain's signature pair

KV caching answers how a single GPU serves a single model efficiently once it is loaded. It says nothing about what happens when the model itself is too large to fit on one GPU in the first place, or when a team wants to split serving and training work across many accelerators at once. That is the subject the GPU Acceleration domain picks up next, and its own signature distractor pair mirrors the memory-versus-speed precision this lesson has been building: tensor parallelism splits the work inside a single layer across GPUs, while pipeline parallelism splits across consecutive layers, handing each GPU a different stage of the same forward pass. The two are frequently swapped for each other on the exam in exactly the way KV caching is swapped with memory-reduction claims here — a real technique, attached to the wrong axis of the problem.

Next: Tensor parallelism vs pipeline parallelism — the GPU Acceleration domain's most commonly confused pair, and the natural next step once a single accelerator's memory, cache included, is no longer enough.