LLM Cost: Per Million Tokens, Per Request, and Per Month

The atomic unit of LLM cost is the token, and 02-03 established the crucial preliminary: tokens are not words. A rough working ratio for English is around 1.3 tokens per word, or about 4 characters per token, but that ratio varies by tokenizer, by language, and by content — code, JSON, and non-Latin scripts tokenize far less efficiently than plain English prose. Any cost estimate built on word counts rather than measured token counts is wrong by an unknown factor, and the fix is trivial: run your actual prompts through the actual tokenizer and count.

The API cost model. Providers bill separately for input (prompt) and output (completion) tokens:

cost_per_request = (input_tokens  × input_price_per_token)
                 + (output_tokens × output_price_per_token)

Prices are conventionally quoted per million tokens, so input_price_per_token = price_per_million / 1e6. The structural fact to remember is that output tokens cost more than input tokens — commonly by a factor of two to five, depending on provider and model. The mechanism is the one 12-05 explained: input tokens are processed in a single parallel prefill pass with high arithmetic intensity, while every output token requires its own memory-bandwidth-bound decode step. Output is genuinely more expensive to produce, and pricing reflects that.

I am deliberately not quoting specific vendor prices anywhere in this lesson. They change frequently, they differ by model tier and region, and a memorized price is a liability rather than an asset. Every worked example below uses explicitly labelled illustrative prices so you can follow the arithmetic and then substitute today's real numbers.

The self-hosted cost model. Here you rent or own GPUs by the hour and produce tokens with them:

cost_per_million_tokens = (gpu_hourly_cost / tokens_per_hour) × 1e6

where tokens_per_hour = output_tokens_per_second × 3600 × utilization

Three things follow immediately, and they are why self-hosting economics confuse people:

1. Throughput is the denominator. Doubling tokens per second halves cost per token. Every throughput technique in this module — batching, quantization, continuous batching, paged attention — is therefore a cost technique.
2. Utilization is a multiplier on the denominator. A GPU running at 20% utilization produces a cost per token five times higher than the same GPU at 100%. Idle capacity is pure loss, and it is the most common reason self-hosting turns out more expensive than expected.
3. Input tokens are not free but they are cheaper. Prefill consumes GPU time too. In a self-hosted budget you account for prefill time and decode time separately, because a workload with 4,000-token prompts and 100-token answers spends its GPU time very differently from one with 100-token prompts and 4,000-token answers.

The full cost surface. Token cost is the largest term for most LLM applications but never the only one. A complete monthly figure includes:

That last row deserves attention. Teams routinely produce a cost model containing only the LLM API line and are then surprised by the total. Human review, evaluation runs, and vector-database hosting are real and frequently comparable in magnitude.

L1 — Intuition: three numbers, multiplied

Every LLM cost estimate is the same shape:

monthly cost = requests per month × cost per request

and

cost per request = tokens per request × price per token

So the whole exercise is: measure the tokens, count the requests, look up the price. The reason estimates go wrong is almost never the arithmetic. It is that one of the three numbers was guessed, and the two most-guessed are tokens per request (because people count words, or forget that retrieved context and chat history are tokens too) and requests per month (because people use the average and ignore peak).

L2 — Mechanism: the token budget, line by line

Build the token budget as an itemized list, not a single number. For a RAG chat request:

Two observations that change architecture. First, the user's own words are 1.2% of the input. Everything else is something you chose to include, which means nearly all of your input cost is a design decision. Second, retrieved context is 62% of the input. Reducing k from 5 to 3 removes 1,000 tokens — a 25% cut in input cost — and whether that hurts answer quality is an empirical question your eval set can answer. That is the shape of a real cost optimization: a measurable token reduction, gated on a measurable quality check.

Where multi-turn chat cost hides. Chat history grows with every turn. If you resend the entire conversation each time, cost per turn rises linearly and cumulative cost across a conversation rises quadratically. A ten-turn conversation with naive full history resends the first turn ten times. This is the cost argument for truncation, summarization, or a sliding window, and 12-11 treats the mechanics.

Where RAG cost hides. Beyond retrieved chunks in the prompt, RAG adds an embedding call per query, possibly a reranking call over tens of candidates, and vector-database query cost. It also adds a one-off but sometimes large ingest cost: embedding the entire corpus. And as 12-12 explains, changing the embedding model means re-embedding everything, which makes that a recurring cost risk rather than a one-time expense.

L3 — The self-hosting crossover, amortization, and the traps

Computing the crossover. Self-hosting has a fixed hourly cost and API usage has a variable per-token cost, so there is a monthly volume above which self-hosting is cheaper. Setting them equal:

crossover_tokens_per_month = (gpu_hourly_cost × hours_per_month × num_gpus)
                             / api_price_per_token

Below that volume the API is cheaper; above it, self-hosting is — provided you can actually achieve the throughput you assumed and keep utilization high. Both provisos are where real projects fail. A self-hosting plan that assumes 100% utilization and peak-benchmark throughput will overstate its advantage by a large factor.

Why utilization dominates. Traffic is not flat. A business-hours application might see 10× more traffic at 2 p.m. than at 2 a.m. If you provision for peak and run continuously, your average utilization might be 25%, quadrupling your effective cost per token. Three mitigations, each with a cost of its own:

Multi-Instance GPU (MIG) is worth a named mention here because it is the NVIDIA feature that addresses exactly this: hard partitioning of a single GPU into isolated instances, each with its own memory and compute slice. Where you have several models too small to justify a whole GPU, MIG converts one underutilized device into several well-utilized ones. Whether a given GPU generation supports it and at what granularity is generation-specific and should be checked against current documentation rather than memorized.

Cost per token is not cost per useful answer. This is the framing error that produces genuinely bad decisions. A cheaper model that needs two attempts, or that produces answers requiring human correction 20% of the time, may cost more per successful outcome than an expensive model that gets it right first time. The right denominator is the business unit — a resolved support ticket, an accepted draft, a correct extraction — not a token. Compute both, and be explicit about which one you are quoting.

The prompt-caching discount. Where a provider offers reduced pricing for cached prompt prefixes, or where your self-hosted stack implements prefix caching (12-07), the fixed portion of your prompt becomes substantially cheaper. This rewards a specific prompt design: put invariant content first and variable content last, so the shared prefix is as long as possible and byte-identical across requests. A timestamp at the top of a system prompt destroys the entire benefit. That is a concrete, cheap, and frequently missed optimization.

Reserved capacity and commitments. Cloud GPU pricing typically offers on-demand, reserved/committed, and interruptible (spot) tiers, with progressively lower rates and progressively worse flexibility. Interruptible capacity is excellent for batch generation and unacceptable for interactive serving. Committed capacity lowers the hourly rate at the cost of paying for it whether traffic materializes or not — which is a bet on your own volume forecast.

The decision framing that actually works. Ask three questions in order: Is my volume high and steady? (if no, use the API); Can I keep utilization above roughly 50–60%? (if no, the crossover math will not deliver); Do I have the engineering capacity to run a serving stack? (if no, the total cost includes salaries you are not counting). Only when all three answer yes does self-hosting reliably win on cost, and it may still win on privacy or latency control when it loses on cost.

A fourth option that is often the right one. A smaller model, or a routed architecture where a cheap model handles easy requests and an expensive one handles hard ones, frequently beats both. Model routing is a cost architecture, and its correctness is an eval question: you need a classifier or heuristic that identifies which requests the cheap model handles acceptably, and you need per-slice eval data to prove it does.

A constructed scenario with explicitly illustrative prices. None of these figures are vendor quotes; substitute current real prices to use this as a template.

The workload.

requests per month              = 200,000
input tokens per request        = 4,050   (from the itemized budget in §2)
output tokens per request       =   300
embedding calls per request     = 1 query embedding, ~50 tokens

Path A — API pricing. Illustrative prices: input 0.50 per million tokens, output 1.50 per million tokens (a 3× output premium, which is a typical shape).

monthly input tokens  = 200,000 × 4,050 =   810,000,000 tokens = 810 M
monthly output tokens = 200,000 ×   300 =    60,000,000 tokens =  60 M

input cost  = 810 M × $0.50/M  = $405.00
output cost =  60 M × $1.50/M  =  $90.00
                                 --------
LLM subtotal                     $495.00 / month

cost per request = $495 / 200,000 = $0.002475  ≈ 0.25 cents

Now the observation that reframes the whole exercise: input is 93% of the token volume and 82% of the cost, despite output tokens being priced 3× higher. In a RAG system the prompt dwarfs the response. Every instinct to optimize output length is aimed at the smaller term.

Path A optimization, quantified. Reduce retrieved chunks from 5 to 3, cutting 1,000 input tokens:

new input tokens per request = 3,050
monthly input tokens         = 200,000 × 3,050 = 610 M
input cost                   = 610 M × $0.50/M = $305.00
new LLM subtotal             = $305 + $90       = $395.00
saving                       = $100/month = 20.2%

A 20% cost reduction from one configuration change. And it is a change with a quality risk, so it is gated on the eval set: re-run with k=3 and check context recall and faithfulness (09-07). If the metrics hold, you have banked 20%. If they drop, you have learned that those two chunks were earning their cost — which is also valuable.

Path A's second optimization — prompt caching. The system prompt plus few-shot examples are 600 fixed tokens, 15% of the input. If the provider discounts cached prefixes, or your self-hosted stack caches them, that 15% becomes much cheaper. It requires that the 600 tokens be byte-identical and leading, which is a prompt-engineering constraint worth designing for from day one.

Path B — self-hosted. Illustrative: one GPU instance at $2.00 per hour, 730 hours per month, achieving an illustrative 1,200 output tokens per second aggregate throughput under continuous batching at this workload's prompt/response shape.

monthly GPU cost = $2.00 × 730 = $1,460.00 (one instance, always on)

capacity check — can one instance handle the load?
  monthly output tokens needed = 60 M
  monthly output token capacity at 100% utilization
    = 1,200 tok/s × 3,600 s × 730 h = 3,153,600,000 = 3,154 M
  utilization required = 60 M / 3,154 M = 1.9%

That is the entire self-hosting problem in one number. The instance is capable of 3,154 M output tokens per month and the workload needs 60 M, so it would run at 1.9% utilization. Cost per million output tokens:

$1,460 / 60 M output tokens = $24.33 per million output tokens

against the API's $1.50 per million output tokens. Self-hosting is roughly 16× more expensive at this volume, entirely because of idle capacity. The GPU is not slow; it is empty.

Path B at the volume where it makes sense. Find the crossover. Compare total monthly costs, holding the 4,050/300 token shape:

API monthly cost   = requests × [(4,050 × $0.50/1e6) + (300 × $1.50/1e6)]
                   = requests × [$0.002025 + $0.00045]
                   = requests × $0.002475

Self-hosted cost   = $1,460 flat, up to the instance's capacity
  instance capacity in requests (output-bound)
                   = 3,154 M / 300 tokens = 10,513,333 requests/month
  (prefill capacity must also be checked; ignored here for clarity and flagged
   as a simplification)

crossover: requests × $0.002475 = $1,460
           requests = 589,899 ≈ 590,000 requests/month

So at roughly 590,000 requests per month the two are equal, and above that self-hosting wins — assuming you achieve the assumed throughput and that the instance really can absorb the load. At 200,000 requests the API wins decisively; at 5,000,000 requests, self-hosting on one instance costs 1,460 against the API's 12,375, a 88% saving. The same architecture is a 16× mistake at one volume and an 8× win at another, which is why the answer to "should we self-host" is always "at what volume."

Path C — the full monthly bill, which nobody remembers to compute. Returning to the 200,000-request API scenario and adding everything else, with illustrative prices:

LLM tokens (optimized, k=3)                          $  395.00
Query embeddings: 200,000 × 50 tokens = 10 M
  at $0.02/M                                        $    0.20
Reranking: 200,000 queries × 30 candidates × 300 tok
  = 1,800 M tokens at $0.10/M                       $  180.00
Vector database hosting (illustrative flat)          $  250.00
Evaluation runs: 100-item eval set × 20 runs/month
  × ~5,000 tokens/item = 10 M at judge-model $1.00/M $   10.00
Logging and storage (illustrative)                   $   40.00
Human review: 2% of 200,000 = 4,000 reviews
  × 3 min × $30/h                                   $6,000.00
                                                    ----------
TOTAL                                                $6,875.20
LLM share of total                                        5.7%

Human review is 87% of the bill and the LLM is 5.7%. This is the constructed example's most important result. A team that spends a week optimizing prompt length to save 100 while a 2% human-review rate costs 6,000 has optimized the wrong term. The correct optimization target is the review rate, which means improving answer quality — and improving quality may well mean spending more on tokens, using a better model or more retrieved context. Cost optimization that ignores the non-LLM terms can point in exactly the wrong direction. Reranking is also worth noting at 180 against the LLM's 395: a component people rarely include in cost models at all.

The ordering principle: measure first, then attack the largest term. In the constructed example above the largest term was human review, the second was the vector database, the third was reranking, and prompt tokens were fourth. A cost programme that starts with prompt tokens because they are the most visible is a cost programme aimed at the fourth-largest line item.

Cost falls under objective 4.4 (identify system data, hardware, or software components required to meet user needs) and 4.1 (assist in deployment and evaluation of model scalability, performance, and reliability), and the course source material names cost per token explicitly as part of the latency, throughput, and capacity-planning content. The module brief notes that the underlying question bank includes a Cost Optimization category, which lands here.

The exam asks this at reasoning depth rather than arithmetic depth, consistent with the finding that questions are general-level. You are far more likely to be asked which change reduces cost or why output tokens cost more than to be asked to compute a monthly bill. But the arithmetic is what makes the reasoning reliable, which is why it is here.

Question phrasings:

• "Which typically costs more, input tokens or output tokens?" — output, because each output token requires its own memory-bound decode step while input is processed in one parallel prefill pass.
• "A RAG application's costs are rising. Which component most likely dominates the prompt?" — retrieved context.
• "Which change would most reduce the cost of a multi-turn chatbot?" — limit or summarize conversation history instead of resending it in full.
• "A team's self-hosted GPU costs more per token than the equivalent API. What is the most likely cause?" — low utilization; they are paying for idle capacity.
• "At what point does self-hosting become cheaper than an API?" — above the volume where fixed GPU cost divided by achieved throughput falls below the per-token price.
• "Which optimization reduces cost per token on self-hosted infrastructure?" — anything that raises throughput: continuous batching, quantization, larger effective batch.
• "Why is cost per token an incomplete metric?" — because a cheaper model that requires retries or human correction can cost more per successful outcome.
• "What must be measured to estimate LLM cost accurately?" — actual token counts from the actual tokenizer, not word counts.

Distractor families:

Output tokens almost always cost more — commonly two to five times more, depending on provider and model tier. The reason is mechanical rather than commercial. Input tokens are processed during prefill, a single forward pass over the whole prompt in parallel, which has high arithmetic intensity and uses the GPU efficiently, so the per-token cost of prefill is low. Output tokens are produced during decode, where each token requires its own step that streams the entire model's weights out of HBM to perform very little arithmetic — the memory-bandwidth-bound regime 12-05 describes. One output token therefore consumes far more GPU time than one input token, and pricing reflects that. The practical consequence is asymmetric: cutting output length saves more per token, but in a RAG system input volume is usually so much larger that total input cost still dominates — in the constructed example above, input was 82% of the bill despite output being priced 3× higher.

Multiply each token category by its price and add: (input_tokens × input_price) + (output_tokens × output_price). Prices are conventionally quoted per million tokens, so divide by 1e6 to get a per-token price. The only difficult part is getting input_tokens right, because the input is not just the user's question — it is the system prompt, plus few-shot examples, plus every retrieved chunk, plus the conversation history you resent, plus the user's query. In the constructed itemization above, the user's own words were 1.2% of the input and retrieved context was 62%. So the honest procedure is: take a representative sample of real requests, run them through the actual tokenizer, and use the measured distribution rather than an estimate. Then multiply by requests per month, and add the non-LLM lines — embeddings, reranking, vector database, evaluation, and human review — because the LLM is often a minority of the total.

Above a computable volume, and only if two assumptions hold. The crossover is (gpu_hourly_cost × hours × instances) / api_price_per_token tokens per month: below that, the API's pay-per-use structure wins; above it, the fixed GPU cost is spread thinly enough to be cheaper. In the constructed example, one GPU at an illustrative $2.00/hour against illustrative API pricing crossed over around 590,000 requests per month, and at 200,000 requests self-hosting was roughly 16× more expensive — not because the GPU was slow but because it ran at 1.9% utilization. The two assumptions that must hold are achieved utilization (a GPU at 25% utilization has a cost per token four times its nameplate figure) and achieved throughput on your actual prompt-and-response shape rather than a benchmark's. Add engineering and on-call cost to the self-hosted side and the crossover moves further out. Self-hosting can still be the right choice below the crossover for reasons that are not cost: data residency, latency control, and freedom from rate limits.

Almost always because tokens per request grew while request volume stayed flat. The usual culprits, in order of frequency: someone increased the number of retrieved chunks or the chunk size, so every prompt got longer; the system prompt accumulated instructions over months of iterative fixes; conversation history is being resent in full and average conversation length increased; a reranker or an extra retrieval stage was added; max_tokens was raised or removed; or a model tier was changed. The diagnostic is straightforward if you instrumented for it — log input and output token counts per request, tagged by endpoint and version, and plot the average over time. Without that instrumentation you are guessing. This is also the argument for treating prompt templates as versioned artifacts (05-04): a prompt change is a cost change, and an unversioned prompt change is an untraceable cost change.

It is the right engineering metric and the wrong business metric. Cost per million tokens tells you how efficiently your serving stack converts money into generated text, and it responds correctly to the levers this module teaches: batching, quantization, utilization, throughput. What it cannot see is whether the text was any good. A cheaper model that needs two attempts, or whose answers require human correction 20% of the time instead of 2%, can cost far more per successful outcome while looking better per token. In the constructed full-bill example, human review at a 2% rate was 87% of the total cost while the LLM was 5.7% — meaning a change that raised token cost by 50% and halved the review rate would cut total cost by roughly 43%. So compute both numbers, quote the business one to stakeholders, and be alert to the possibility that the correct cost optimization is to spend more on inference.

Five that are routinely omitted from cost models. Reranking — a cross-encoder scoring thirty candidates per query processes far more tokens than the final generation does; in the constructed example it was 180 against the LLM's 395. Vector database hosting, which scales with index size, query rate, and replication, and is a flat monthly line that does not appear in any token count. Corpus embedding at ingest, a one-off that becomes recurring the moment you change the embedding model, since that forces re-embedding the entire corpus (12-12). Evaluation runs, especially with an LLM-as-judge, where each eval item may consume several thousand tokens and the suite runs on every release. And human review, which is usually the single largest line item and is almost never in the spreadsheet. A cost model containing only the generation API call will understate a real RAG system by a large multiple.

• Three numbers: tokens per request × requests per month × price per token. Measure all three; guessing any one invalidates the estimate.
• Tokens are not words. Count with the real tokenizer on real traffic.
• Output tokens cost more than input tokens — typically 2–5× — because decode is memory-bandwidth-bound while prefill is parallel and efficient.
• In RAG, input volume usually dominates the bill anyway. Retrieved context is typically the largest single line in the token budget.
• The user's own words are a tiny fraction of the input. Nearly all input cost is a design decision you control.
• Self-hosted cost per token = gpu_hourly_cost / tokens_per_hour. Throughput is the denominator and utilization is a multiplier on it.
• The API-versus-self-hosting crossover is computable. In the constructed example it was ~590,000 requests/month; below it self-hosting was 16× worse, purely from idle capacity.
• Resending full chat history makes cumulative conversation cost quadratic. Truncate, window, or summarize.
• Prompt caching requires invariant content first. A timestamp at the top of a system prompt destroys the discount.
• Cost per token is not cost per outcome. In the constructed full-bill example, human review was 87% of total cost and the LLM was 5.7% — so the right optimization may be to spend more on inference.
• Measure the full cost surface before optimizing anything: reranking, vector DB, embeddings, eval runs, and human review are all real.

Next: 12-10 supplies the other half of what a serving decision needs. Cost tells you what a request is worth in dollars; latency tells you what it is worth in milliseconds — and because batching improves throughput by making individual requests wait, the two are in direct tension. Time to first token, inter-token latency, and the p95 tail are three different numbers, and reporting an average of them hides exactly what users complain about.