Latency and Throughput: TTFT, Tokens per Second, and p95

Latency is a per-request clock: how long does this one call take, from the moment the request lands to the moment the response is complete. Throughput is a system-wide rate: how many tokens does the server emit per second, summed over everyone it is serving at once. They are related but not the same measurement, and a deployment can improve one while making the other worse.

For an LLM specifically, latency is not one number — it decomposes into two phases with different bottlenecks:

• Time-to-first-token (TTFT). The delay from request arrival to the first generated token reaching the client. This covers queueing (waiting for a batch slot or a free worker), network transit, and the prefill pass — the single forward pass over the entire input prompt that builds the initial KV cache. TTFT is dominated by prompt length: a 4,000-token prompt takes meaningfully longer to prefill than a 200-token one, because prefill is compute-bound and scales with input size.
• Inter-token latency (ITL). The time between each subsequent generated token during the decode phase. Decode generates one token at a time, each pass reading the full KV cache built so far. 12-05 covers why this phase is memory-bandwidth-bound rather than compute-bound — the GPU spends most of its time moving the KV cache and weights through memory, not doing matrix math. ITL is often reported as its inverse, tokens per second per user, which is more intuitive: "18 tokens/sec" tells a reader roughly how fast text will stream onto their screen.

Throughput, by contrast, is a fleet-wide figure: total tokens generated per second across every request a batch of GPUs is currently processing. A server can hold dozens of requests in flight at once (see 12-06 on batching), and its throughput number is the sum of all their token streams, not any one user's stream. This is why a vendor's "10,000 tokens/sec" headline is a throughput claim about the whole deployment, not a promise about how fast any individual response will feel.

The relationship between the two is the crux of this lesson: throughput goes up by admitting more concurrent requests into a batch, and TTFT for any one of those requests goes up because it has to wait its turn — either queueing before prefill starts, or sharing decode compute with everyone else already in the batch. Optimizing blindly for one metric degrades the other, and a serving system has to pick an explicit operating point rather than pretend it can maximize both without constraint.

L1 — The intuition: two clocks, one shared resource

Picture a coffee shop. TTFT is how long you wait in line before the barista starts your order. ITL is the pace of the espresso machine once it's working on your cup — a roughly constant drip. Throughput is how many drinks the whole shop serves per hour. If the shop lets more people order at once (batching), it serves more drinks per hour — throughput goes up — but each individual line gets longer, so TTFT goes up too. If the shop insists on serving one person start-to-finish before starting the next, TTFT for the first customer is great and throughput for the shop is terrible. There is no configuration that maximizes both simultaneously; every serving setup is a point somewhere on that trade-off curve, and the job is choosing the point deliberately rather than discovering it by accident in production.

L2 — The mechanism: prefill, decode, and queueing

Three stages determine end-to-end latency for a single request:

1. Queueing delay. The request arrives and waits for a scheduling slot — either an open spot in a static batch or admission into a continuously-batched engine (see 12-06 and 12-07). Under light load this is near zero; under heavy load it can dominate TTFT entirely, because a request simply sits in line behind others.
2. Prefill. One forward pass processes the entire prompt at once, computing attention over every input token and populating the KV cache. Prefill is compute-bound: it scales roughly with prompt length, and a long system prompt or a large retrieved-context block (as in a RAG pipeline — see 07-09 for the full pipeline stage by stage) directly inflates TTFT because there is simply more to process before the first output token can be produced.
3. Decode. The model then generates output tokens one at a time, each step reading the growing KV cache and appending one new token. Decode is memory-bandwidth-bound, as 12-05 covers: each step moves a large volume of cached keys and values through GPU memory relative to the small amount of new compute it does. This is why ITL stays roughly flat per token regardless of how long the response has grown — each step pays a similar memory-movement cost — while total generation time simply scales with the number of tokens requested.

Throughput sits on top of all three stages. A server that batches many requests' decode steps together (continuous batching, 12-06/12-07) amortizes the fixed per-step overhead across more tokens, raising tokens-per-second at the fleet level. But every request sharing that batch experiences slightly higher ITL than it would running alone, because the GPU's attention is split. The p95 latency a user actually experiences is queueing delay, plus prefill time, plus (output length × per-token decode time under current load) — and every one of those four terms grows as the operator pushes for higher throughput.

L3 — Why the tail (p95/p99) matters more than the mean

Average latency is a poor description of user experience because LLM request populations are heterogeneous by nature: prompt lengths vary, output lengths vary, and load on the server varies minute to minute. A handful of very long prompts or a momentary traffic spike can push a small fraction of requests to multiples of the typical latency, and that fraction is invisible in a mean but highly visible to the users who hit it. This is why production dashboards report p50 (median — a stable read on typical experience), p95 (a stricter bound most requests should clear), and sometimes p99 (how bad the worst realistic case gets). A service with a great p50 and an ugly p95 is telling you that its architecture works fine at typical load and falls over intermittently — usually because of batching contention, a burst of long prompts, or GPU memory pressure evicting KV cache entries. Service-level objectives (SLOs) for LLM APIs are almost always written against p95 or p99, not the mean, precisely because the mean hides exactly the failure mode operators most need to catch.

These three numbers are frequently confused because all three come from the same serving stack and all three move when you change the batch size — but they answer different questions and are optimized by different levers.

12-09 already covers the cost-per-token calculation in full — this lesson does not re-derive it, but the connection matters: throughput is the denominator of the cost equation. Higher tokens/sec per GPU-hour is exactly what lowers cost per token, which is why an operator chasing lower cost is, mechanically, chasing higher throughput — and therefore accepting worse TTFT and worse p95 latency as the trade-off's other side. There is no configuration that improves cost, latency, and throughput simultaneously without limit; every serving decision spends one to buy another.

Take a small, explicitly constructed example — twenty logged requests from one hour of a chat deployment, each with an end-to-end latency in milliseconds and an output token count. This data is illustrative, built to demonstrate the arithmetic; it is not a measured benchmark of any real system.

Request latencies (ms), sorted ascending:
210, 220, 225, 230, 235, 240, 245, 250, 255, 260,
265, 270, 280, 290, 310, 340, 380, 420, 610, 1450

Step 1 — median (p50). With 20 sorted values, the median sits between the 10th and 11th: (260 + 265) / 2 = 262.5 ms. Half of users see a response in under roughly a quarter of a second.

Step 2 — p95. The 95th percentile position in a sorted list of 20 is index 0.95 × 20 = 19 (using the common "nearest-rank" method, rounding up to the 19th value). The 19th sorted value is 610 ms. So 95% of requests finish at or under 610 ms — more than double the median.

Step 3 — p99 (if this were a larger sample). With only 20 points p99 is not statistically meaningful — you need hundreds of samples before a p99 estimate stabilizes. This is a real operational trap: teams report p99 off a handful of production requests and the number swings wildly hour to hour because it is estimated from too few tail events. The 20th (worst) value here, 1,450 ms, is nearly 6× the median — that single outlier is the kind of data point p99 exists to characterize, and why one slow request can move it disproportionately.

Step 4 — throughput. Suppose those 20 requests produced a combined 20 × 85 = 1,700 output tokens (an average of 85 tokens per response — a constructed assumption for this example) over the 60-minute window, and at any given moment the server was serving roughly 4 requests concurrently. Aggregate throughput ≈ 1,700 tokens ÷ 3,600 seconds ≈ 0.47 tokens/sec averaged across the whole hour — a deliberately low number because this constructed log is sparse; a busy production server pushing hundreds of requests per minute would show throughput two to three orders of magnitude higher. The point of the arithmetic is the method — sort, index, divide — not this specific illustrative figure.

Step 5 — reading the gap. The gap between p50 (262.5 ms) and p95 (610 ms) is the number an SLO should be written against. A team that only tracks the mean here — roughly 315 ms, pulled up by the two outliers — would believe the system is healthier than 5% of its users actually experience it to be.

The decision rule underneath this table: identify whether a human is watching the response arrive in real time. If yes, TTFT and ITL dominate and you protect them even at some cost to aggregate throughput — cap batch size, reserve capacity, add priority lanes. If no — a batch job, an overnight pipeline, an async worker — throughput and cost per token dominate, and you should push batch size and concurrency as high as the hardware allows, because nobody is measuring an individual request's wait.

The exam's Software Development domain includes objective 4.1 ("assist in the deployment and evaluations of model scalability, performance, and reliability") and 4.4 ("identify system data, hardware, or software components required to meet user needs") — both of which are directly about reasoning through exactly this trade-off. COURSE-INDEX's calibration notes that exam questions are general-level rather than deep-technical: expect a scenario ("users report the chat feels slow to start responding, but the support dashboard shows healthy average latency") that tests whether you can correctly diagnose a TTFT problem hiding behind a mean that ignores the tail, or a scenario asking which lever (batch size, concurrency, hardware) moves throughput versus which one moves TTFT. Distractor families to expect: options that swap "throughput" and "latency" as if they were interchangeable measures of "speed"; options that report a mean where the scenario actually calls for a percentile; and options that propose increasing batch size as an unconditional fix, ignoring that it improves throughput while degrading the exact tail latency the scenario describes. The correct answer usually requires identifying which of the two axes the question is actually asking about before picking a lever.

Prompt length drives TTFT almost directly, because the entire prompt has to pass through the prefill stage before the first output token can be produced, and prefill's cost scales with the number of input tokens being processed. A short chat turn of a few dozen tokens prefills almost instantly; a RAG request that stuffs several retrieved documents into the context window (see 07-04 on vector databases and the retrieval step that produces that context) can push the prompt into the thousands of tokens, and that added prefill work shows up as added TTFT before any generation has even started. This is one reason retrieval systems trim and rerank context rather than dumping everything a search returns into the prompt: every extra token in the prompt is a small, direct tax on how long the user waits before seeing anything at all. It is also why two requests to the same model, on the same hardware, at the same moment can show very different TTFT — the difference is often nothing more than how long each caller's prompt happened to be.

TTFT measures the delay before the very first token of a response appears — dominated by queueing and the prefill pass over the input prompt. Tokens per second measures the rate at which subsequent tokens stream out during decode (from one user's point of view) or the aggregate generation rate across all concurrent requests (from the server's point of view). A deployment can have a fast TTFT and a slow tokens-per-second rate, or the reverse — they are governed by different stages of the request lifecycle and different bottlenecks (compute-bound prefill vs memory-bandwidth-bound decode, as 12-05 explains).

Batching lets a GPU process the decode step for several requests in a single pass, which spreads its fixed per-step overhead across more tokens and raises aggregate tokens-per-second. But every request in that batch waits for the others' work to be scheduled alongside it — both in the initial queueing delay before prefill starts and in the reduced share of compute each request gets during decode — so each individual request's TTFT and inter-token latency get worse even as the fleet's total output rises. This is precisely the trade-off 12-06 and 12-07 describe: static and dynamic batching trade user-facing latency for GPU utilization, and continuous batching (used by vLLM and similar engines) narrows that trade-off without eliminating it.

Average latency is dominated by the typical, fast majority of requests and can look healthy even while a meaningful minority of users have a bad experience — long prompts, momentary load spikes, or GPU memory pressure evicting cache entries all create a tail that a mean simply averages away. p95 (or p99, given enough samples) directly reports how bad the experience gets for the unlucky fraction of requests, which is the number that actually correlates with user complaints and is the number production SLOs are almost always written against.

• TTFT (time-to-first-token): the delay from request arrival to the first output token reaching the client, dominated by queueing and prefill.
• Inter-token latency (ITL): the time between successive output tokens during decode; its inverse is tokens/sec per user.
• Prefill: the single forward pass over the full input prompt that populates the initial KV cache; compute-bound and scales with prompt length.
• Decode: the one-token-at-a-time generation phase after prefill; memory-bandwidth-bound, as covered in 12-05.
• Throughput: aggregate tokens generated per second across all concurrently-served requests.
• p50 / p95 / p99: percentile latency figures — the median and two tail measures — read off a sorted distribution of request latencies.
• SLO (service-level objective): an explicit target, usually written against a percentile rather than a mean, that a deployment is tuned to meet.

TTFT and inter-token latency describe one user's wait; throughput describes the whole fleet's output rate; and pushing throughput up by batching more aggressively reliably pushes both of those latency figures — and the p95/p99 tail especially — in the wrong direction. The fix is never a single number chased in isolation: pick the metric that matches who's waiting (a live user vs an overnight batch job), write an explicit SLO against a percentile rather than a mean, and treat every batching or concurrency change as a trade against that SLO rather than a free win. 12-09 already covers the dollar side of this same lever — throughput is the mechanism that turns GPU-hours into cost per token — so a deployment decision is really one trade-off viewed from two ledgers at once: milliseconds on one side, dollars on the other.

Next: 12-11 turns from serving mechanics to the request itself — specifically, what happens across a multi-turn chat when a follow-up question ("what about the second one?") is meaningless to a retriever without the conversation history that came before it, and how query rewriting turns that fragment into something a vector index can actually search.