42 of 52 chapters published · about 301 minutes of reading so far · last revised 2026-08-11

Part 1 — The application layer — APIs and frontier models4/14

You buy tokens from someone else's endpoint. Your levers are what you send, what you ask for back, and how often. Most teams live here and never leave, and that is often correct.

1.1What you are actually paying for4 min

Output costs 5× input. Cache reads cost a tenth of input. Cache writes cost more than input. Three published ratios, and almost every app-layer decision follows from them.

1.2The output premium5 min

The 5× output premium is constant across a 10× price ladder. Changing model tier scales your bill; it never changes your exposure. Only fewer output tokens does that — and reasoning models bill their thinking as output.

1.3Reading your first AI bill5 min

Anthropic exposes usage at minute granularity and cost only at daily. Each endpoint has a blind spot the other covers, and two attribution holes return null. Here is how to read what you are actually spending.

1.4The cost levers, ranked5 min

Nine levers, ordered by the multiple each one has actually been measured at in this book — not by how much attention each gets. The largest is 24×. Most teams pull the 1.4× one first.

1.5Tokenizer economicsplanned

Why non-English text and code cost more per unit of meaning — and why JSON schema field names are billed on every single call.

1.6Prompt caching, preciselyplanned

Prefix matching, TTLs, published discounts, and the ordering bug that silently voids it.

1.7Context windows and what long context really costsplanned

Attention scaling, the price of stuffing, and when retrieval beats a bigger window.

1.8Reasoning models — when thinking earns its priceplanned

Reasoning tokens bill as output. Deciding per call site, not per app.

1.9Embeddings and vector searchplanned

Index build cost, dimensionality, re-embedding, and the storage tail.

1.10Fine-tuning versus promptingplanned

The volume at which a shorter prompt pays for a model you now maintain.

1.11Images, audio, and videoplanned

How non-text modalities are metered, and why they break token intuitions.

1.12Agents — why loops multiply billsplanned

Turn count, context regrowth, tool-call overhead, and runaway guards. Evals are loops too, and are budgeted the same way.

1.13Routing and cascadesplanned

Cheap-first-pass architectures, and building the eval before the router.

1.14Gateways and routersplanned

OpenRouter, LiteLLM, Portkey. The layer that makes every other choice reversible, and what it costs.

Part 2 — The hosting layer — hyperscaler, neocloud, or your owncomplete

You rent accelerators by the hour instead of buying tokens. Cost per token becomes a function of how busy you keep the hardware. Relevant once volume is steady and large enough to fill a GPU.

2.1What an H100-hour actually costs5 min

The same 8 GPUs cost $55, $88, or $98 an hour depending only on which logo is on the invoice. The full spread is 4.7×, and the break-even maths that decides commitments.

2.2Hyperscaler versus neocloud6 min

Run one H100 continuously with 10 TB of egress and the spread is 4.5×. The neocloud discount is real and it is not free — this is what it costs you, itemised.

2.3Spot, preemptible, and the economics of interruption7 min

AWS gives you two minutes. Azure gives you thirty seconds. Google's default is zero — and Google says outright that if you want warning on a GPU, don't use Spot. None of those windows is long enough to save a loaded model.

2.4Reservations, savings plans, and commitment maths10 min

AWS advertises "up to 72% off." On a p5 the best real number is 62.4%, and the flexible plan gives you 33.4%. The premium you pay for flexibility scales with how scarce the silicon is — 1.20× on a c5, 1.77× on a p5.

2.5Serverless GPU versus dedicated9 min

The crossover is not set by how much traffic you have. It is set by how long your jobs are — because a 2-second job on RunPod's defaults bills 17 seconds. And Modal's headline rate is the preemptible one; guaranteed execution is 3× that.

2.6Managed inference endpoints8 min

One open-weights model, four vendors, a 6.7× spread. AWS and Google price it identically to the cent — and the independent that undercuts them both is not the one you would guess.

2.7Egress, storage, and the bill around the GPU6 min

GCP charges 33% more than AWS to move a byte out, and gives you a free tier 100× smaller. At 10 TB a month that gap alone is worth two weeks of a dedicated H100.

2.8Regions, residency, and sovereignty9 min

AWS runs 106 regions. The H100 p5 instance exists in 15 of them, and exactly one is in the European Union. Before you price residency, check whether the hardware is there at all — and know that an "EU (…)" region label is not a statement about the European Union.

2.9The total cost of self-hosting9 min

Amazon shortened its server depreciation from six years to five and told the SEC why — AI. A DGX H100's nameplate says 19.8 kW and its own heat spec says 11.30. Both numbers change the buy-versus-rent answer, and neither appears in the spreadsheet people actually build.

2.10The neocloud landscape10 min

Two of these companies file with the SEC, and one discloses that a single customer is 67% of its revenue. The other five disclose nothing, because they don't have to. That is the tier boundary — not price.

2.11Lock-in is a cost, and it is largest for the smallest buyer11 min

AWS will refund a Savings Plan in full — if the hourly commitment is $100 or less, you ask within seven days, and the calendar month hasn't ended. Every exit in this market is an absolute dollar threshold, which means they are generous to small commitments and meaningless to large ones. The small buyer still carries the larger cost.

2.12The small-business path10 min

Fifty thousand a year is $5.71 an hour. That single conversion decides everything — it buys between 0.48 and 2.21 continuously-running H100s depending only on where you buy, and it puts you at 5.71% of the one refund threshold AWS offers. The small buyer's advantages are real and almost nobody uses them.

Part 3 — The silicon layer — what the hardware actually costscomplete

Below the hourly rate is a physical machine with a fixed memory bandwidth and a fixed amount of VRAM. Those two numbers set the ceiling on everything above. Relevant when you are choosing accelerators or explaining why the cheap one was not cheaper.

3.1Memory bandwidth is the product5 min

H100 and H200 have identical compute. The H200 serves 1.9× the inference throughput. Once you understand why, most accelerator decisions answer themselves.

3.2The accelerator ladder8 min

Fifteen rungs spanning 13.9× in price. Rank them by dollars per unit of memory bandwidth instead and the order scrambles — the cheapest card on the ladder is also the cheapest bandwidth, and the mid-range card everyone buys is the most expensive thing on it.

3.3VRAM sizing8 min

Qwen2.5-72B's KV cache costs exactly 320 KiB per token — so one request at its full 32K context needs 10 GiB, on top of the weights. Work that through and an H200 serves 6.4× the concurrency of an H100 for 1.47× the price, which inverts the per-gigabyte ranking entirely.

3.4FLOPs, MFU, and the utilisation trap7 min

A single-stream decode on an H100 runs at 0.34% of the chip's arithmetic capacity while nvidia-smi reports 100% utilisation. The ceiling has a closed form — batch size × bandwidth ÷ peak FLOPS — and it collides head-on with the VRAM limit from the last chapter.

3.5Quantization9 min

Halving the precision does not halve the checkpoint. Measured on published INT4 weights, a 72B model comes out 14.4% larger than the arithmetic says and a 7B model 46.3% larger — because one block never quantizes. The same asymmetry runs through the quality loss, and it points the technique at the opposite end of the model range from where most people apply it.

3.6TPU, Trainium, and AMD8 min

A Trainium instance costs 60.9% less per hour than an H100 instance from the same vendor on the same day — and 6.8% more per TB/s of memory bandwidth, which is what decode actually consumes. The inference-specific chip is the real bargain, and it is not the one being marketed at you.

3.7Interconnect and topology10 min

Inside a p5 the GPUs talk at 900 GB/s each. Outside it they talk at 50 GB/s each — an 18× cliff at the edge of the box, on the fastest network AWS sells. The fabric itself is cheap: doubling it on a Trainium instance costs 15.26%, and the marginal gigabit costs a sixth of the average one.

3.8Owning the hardware10 min

Buying converts a variable cost into a fixed one, and the bill arrives whether the machine is busy or idle. CoreWeave's filings put numbers on it — $25.83 million of capex per megawatt of live capacity, against $6.04 million of revenue per megawatt-year, which means depreciation alone consumes 71.3% of revenue at the six-year life the company itself uses.

Part 4 — Scaling — serving economics under loadcomplete

The same GPU serves five requests a second or fifty depending entirely on how you schedule work onto it. This is where cost per token is actually decided, and where the largest wins live for anyone running their own inference.

4.1Continuous batching5 min

Batch size is where you sit on the roofline, and KV cache fragmentation is what caps it. The measured gap between naive and good serving is 24×.

4.2KV cache and paged attention5 min

The 24× and the 2–4× are different baselines, and conflating them will make you buy the wrong thing. Paging fixes waste; sharing fixes duplication. Only one of them helps a chatbot.

4.3Prefill/decode disaggregation8 min

Your inference server runs two workloads with opposite hardware appetites and bills them on one machine type. Splitting them delivered 1.4× throughput at 20% lower cost on Azure production traces — and made the older GPU the better buy for half the job.

4.4Speculative decoding10 min

The one lever in this book that can make your product faster and your bill bigger at the same time. It buys latency with arithmetic, and the exchange rate is set by a single number most teams never measure.

4.5Tensor, pipeline, and expert parallelism10 min

Compute per chip falls as you add chips. Communication does not. That asymmetry is the entire economics of splitting a model — it is why a 62B model can beat a 540B one on efficiency, and why a mixture-of-experts model computes 13B parameters but must store 47B.

4.6Autoscaling accelerators10 min

The cold start is not a latency problem you tolerate. It is a billed line item you pay for on every scale-up — and AWS's serverless inference does not offer GPUs at all.

4.7Tail latency versus cost6 min

Over-provisioning is what you buy to hide your tail. Queueing theory prices it — the last nine points of utilisation cost eleven times the wait — and Google's own conclusion is that tail-tolerance is how you stop paying for it.

4.8Multi-tenancy and co-location6 min

A model that needs 20 GB on an 80 GB card wastes three quarters of what you rented. Hardware partitioning fixes that with a guarantee; time-slicing fixes it with a hope. NVIDIA describes the failure mode precisely, and it is the exact resource this book says you are short of.

4.9Benchmarking tokens per second per dollar6 min

A throughput number is not a number until you know what it was measured under. MLPerf's rulebook is effectively a catalogue of how benchmarks lie — including a rule that bans quoting TDP as power. Use it as a checklist, including on this book.

Part 5 — Operations — keeping it down in productioncomplete

Savings decay. This part is the practice that holds them: attribution, unit economics, guardrails, and the operating cadence. Applies at every layer above.

5.1Making it stick5 min

Savings that depend on someone remembering decay. Anthropic's Spend Limits API makes the cap a data structure with inheritance and an approval loop — and the gotchas in it are instructive about what durable cost control requires.

5.2Unit economics for AI features6 min

Three layers of the same industry publish three different denominators for the same cost. NVIDIA sells cost per token, Microsoft warns shareholders about margin, and Doximity buries inference inside a cost-of-revenue line whose growth outran revenue by almost exactly two to one.

5.3Attribution at scale6 min

Attribution fails at the join, not the tag. The three clouds disagree about case, length and count, so the only portable key is the intersection — 63 characters, lowercase — and on the trace side the GenAI token convention is still marked Development.

5.4Showback, chargeback, and ownership6 min

The choice is not showback versus chargeback, it is whether the number lands on someone who can change it. Azure states the constraint that decides your start date — tags are not applied to historical data, so allocation can only ever begin the day you tagged.

5.5Cost in CI5 min

Budget alerts cannot be the control, because the vendors document how slow they are — AWS Budgets updates up to three times a day, 8–12 hours apart, and Google states that an alerts-only budget is not a hard cap. The gate has to move to the pull request.

5.6Capacity planning and forecasting6 min

Forecasting dollars for AI is forecasting a moving target. Forecast capacity instead — and note that a commitment discount is really a utilisation threshold. At AWS's published 62.4% three-year discount, the committed rate is 37.6% of on-demand, so the machine has to be busy 37.6% of the time to break even.

5.7Incident economics6 min

A cost incident is not detected by the billing alert — AWS documents updates up to three times a day, 8–12 hours apart. On one p5.48xlarge at $55.04/hour, a 12-hour blind window is $660.48 spent before anyone is told. The control has to be a ceiling, not an alarm.

5.8Vendor management and negotiation6 min

No vendor publishes what it concedes, so this chapter carries no negotiated figures and never will. What it does instead is turn the public record into leverage — the published discount ladder is your floor, the switching cost from lock-in is your credibility, and a listed vendor's own filings tell you how badly it needs you.

5.9The operating cadence5 min

The right review rhythm is not a management preference — it is set by how fast the data arrives. AWS updates budgets up to three times a day, Azure needs up to 48 hours on a new subscription and never backdates a tag. Reviewing faster than the pipeline reports is reviewing noise.

Why the unwritten chapters are listed.

Because the map is worth more than the territory covered so far. Publishing the whole outline says what this intends to be, lets you tell me which chapter you actually need next, and fixes each chapter’s address from the day it is planned — so a link written today still resolves when the chapter lands. Every infrastructure figure is priced on AWS, GCP, and Azure; a chapter citing one cloud is a draft, not a chapter. Tell me what is missing and it moves up the list.