Most of the bill is re-reading

A language model that answers a question about your codebase has to be told about your codebase first. Today that telling happens on every single turn. The material is retrieved, or pasted, or replayed from a cache, and it arrives in the prompt again, and again, and again. In the best measured trace of real agentic coding that exists in public, 4,265 developer sessions and 357,161 model steps, the total came to 54.90 billion input tokens against 186.9 million output tokens. That is 293.7 tokens in for every token out, and 52.56 billion of the input tokens, 95.7 percent of them, were context being re-read rather than anything new.

This is not a bug. It is how the interface works: a model has no memory between calls, so the only way to give it your material is to send your material. But it is the largest single line in the bill, and it is the line a model that can actually learn the material should be able to remove. We build such a system, so we have an interest in the answer being large. This piece works out how large it honestly is, using published figures where they exist and saying so where they do not.

The question is: how much of what the industry spends on inference is spent re-reading things the model has already been told, and how much of that does learning the material instead actually save?

Conclusions

How big the pot is, when you only count what is disclosed

Before asking what share of the money is re-reading, it is worth establishing how much money there is, because the figure most often repeated is not a revenue figure. Figure 1 puts the two quantities side by side.

What the data says

The largest laboratories state their size as annualised run-rates: above 65 billion dollars for Anthropic at the end of July 2026 and above 40 billion for OpenAI in August. Both figures reached the press through investors rather than through filings, and the two companies may compute the metric differently, so the ordering between them is softer than it looks. The only fully audited annual figure in the sector belongs to OpenAI: 13.07 billion dollars of 2025 revenue against a 20.92 billion dollar operating loss. The revenue figure is about a third below its own end-of-year run-rate, which is the gap Figure 1 draws. A quarterly comparison of booked revenue also circulates, but we could not trace it to a source either company stands behind, so we do not use it. Adding every laboratory's run-rate reaches roughly 120 billion, a sum that mixes revenue booked gross with revenue booked net, recurring revenue with audited revenue, and February vintages with August ones, and should not be used. The best-documented large number in the sector sits on the other side of the ledger: Microsoft's fiscal-2026 annual report discloses 24.1 billion dollars of revenue from OpenAI, including revenue-sharing. What a laboratory pays its cloud provider is now audited. What any laboratory earns is not.

Two figures set the scale. Private valuations: Anthropic 965 billion dollars post-money, OpenAI 852 billion, xAI 250 billion, DeepSeek 52 billion, Moonshot 35 billion, Mistral at a last confirmed 11.7 billion euros. That is about 2.2 trillion dollars of paper value across the independent labs, against 5.16 trillion for NVIDIA alone. And 2026 capital expenditure guided by the four largest cloud providers comes to 720 to 745 billion dollars, which is about seven times the two largest labs' combined run-rate.

Why this is hard to pin down

Only NVIDIA has genuinely separable AI revenue. Gemini has no revenue line anywhere. Meta discloses two segments, neither of them AI, and is spending 130 to 145 billion dollars against zero disclosed AI revenue. xAI's revenue sits inside a segment its own filing defines as "our AI compute, Grok, and X", which in the prior full year was 57.6 percent advertising. Alibaba's model appears in filings only as a cost. Any table with a clean AI revenue column across all the majors is manufacturing several of its cells.

What it means for building

Size a market from the two companies that disclose, and treat the rest as unbounded. Be careful which total you quote: 73, 105 and 120 billion are all defensible and they describe different quantities.

Also true

The picture is circular. Analysts put Anthropic and OpenAI at roughly 73 percent of Amazon's AI revenue and above 70 percent of Microsoft's, while Amazon invested 50 billion dollars in OpenAI, Microsoft counts OpenAI's Azure spend inside its own AI run-rate, and NVIDIA invested 30 billion in OpenAI. Adding NVIDIA's data-centre revenue to cloud AI revenue to laboratory run-rates counts the same dollars two and three times.

How much of the compute is inference, and why five answers disagree

Every one of the figures below is real and none of them can be averaged with another, because each has a different denominator. We print the denominator in its own column, which is the only honest way to show them together.

What the data says

2026 is the first year inference spending is projected to exceed training spending on the narrowest denominator, and the direction is the same on all of them. The most quotable of the four is the weakest: NVIDIA's 40 percent dates from May 2024 and has never been restated, and when the chief executive was asked to update it on the November 2025 earnings call he declined, saying it is hard to know what the percentage will be at any given moment.

Why this is hard to pin down

No published split of cloud capital expenditure into training and inference exists. One research firm models exactly this and keeps the output behind a paywall. The International Energy Agency states that none of the companies disclosing per-query metrics has offered a comparable breakdown, and its own report carries no training-versus-inference electricity split either.

Also true

Inference is real cost of goods. OpenAI's gross margin is estimated at 33 percent with inference costs of 8.4 billion dollars in 2025 rising to a projected 14.1 billion in 2026, and Anthropic cut a projected gross margin by about ten points after inference costs ran roughly 23 percent above plan. Both are third-party estimates. Neither company publishes the split.

How much of that is re-reading

This is the load-bearing section. The best measurement available is a trace of real agentic coding published in June 2026: 4,265 sessions from 43 developers, 357,161 model steps and 432,510 tool calls, recorded across two commercial coding agents and worth about 40,400 dollars of API-equivalent spend. It splits input tokens into two kinds. Append tokens are genuinely new material. Prefix tokens are the conversation so far, sent again. Figure 2 shows where the money goes once caching is applied.

What the data says

Of 54.90 billion input tokens, 52.56 billion were prefix and 2.34 billion append. That is 95.7 percent of input tokens being re-read, 293.7 input tokens for every output token, and only 12.5 to 1 counting genuinely new input alone. The reported medians per step are about 119,000 prefix tokens, 875 append tokens and 214 output tokens: roughly two orders of magnitude more prefix than append.

Two independent measurements land in the same place. A published account of a large porting project, 6,778 commits over eleven days with about 64 agents running concurrently, reports 5.9 billion uncached input tokens, 72 billion cached input reads and 690 million output tokens for about 165,000 dollars: 113 to 1, with 92.4 percent of input being re-read. And a token-marketplace study reports more than 85 percent of agentic token burn originating in cached prompts. Figure 3 puts those beside our own.

Why this is hard

Caching is supposed to solve exactly this, and it does not, because caching makes the re-read cheaper rather than unnecessary. Cache reads cost about a tenth of the input rate at OpenAI, Anthropic and Google, and the trace above already achieves a 95.7 percent hit rate. The re-read line is still 59.5 percent of the bill, because at 281 cached input tokens per output token a tenfold discount still leaves the largest line.

What it means for building

If your agent's bill is uncomfortable, the first thing to look at is not which model you pick but the fraction of your prompt that is unchanged from the previous call. One published case study moved a single dynamic identifier out of a cacheable prefix, took the hit rate from 7 percent to 74 and later 84, and cut cost by 59 percent without changing the model at all.

Also true

Agentic coding is the extreme end of this distribution, not the average. General traffic through a large model marketplace ran at roughly 15 prompt tokens per completion token in late 2025, a twentieth of the agentic ratio. And agentic coding is not all of AI revenue: token-metered API is estimated at 15 to 20 percent of OpenAI's revenue and 70 to 75 percent of Anthropic's. So the 59.5 percent cannot be lifted to an industry-wide share of revenue, and since no provider publishes an input-to-output split, that industry-wide number is not derivable from anything public. We do not derive it.

What we measured: the prompt for a question whose answer has been learned

Learner 1.0 takes a frozen base model, Qwen3.6-27B in bf16, adds about 1.14 billion trainable parameters, runs a short warm-up phase, and then trains only those parameters on whatever a user teaches it. The mechanism is proprietary and this piece describes none of it. It describes a measured consequence: once the material is learned, the prompt for a question about it is the question. Figure 4 is 68 previously graded questions from three demonstrations, each asked three ways.

What the data says

Across all 68 questions the retrieval control pays a median of 51.9 times more prompt tokens, with an interquartile range of 32.4 to 61.7 and a distribution-free interval on the median of 42.0 to 56.0. Per lane the medians are 38.5 for the handbook, 25.5 for the reference page and 59.2 for the invented-language corpora. The whole-document control runs from a median of 29.1 on the smallest source to 2,132.9 on the largest, and we report it per lane rather than pooled, because the pooled interval runs from 89.8 to 1,987.2 and a range that wide is not a summary of anything.

Those four numbers are the whole of the ratio. The learned prompt is 11 to 37 tokens across every lane and does not grow: 14 after a 706-word handbook, 26 after 44,000 words of invented language. Every context-based route pays for the material on every question. Learning pays for it once.

The statistical statement, said precisely

This is a census of 68 previously graded questions over three corpora that we chose, not a random sample from a population of workloads. All 68 ratios exceed one, so a sign test returns a p-value near ten to the minus twenty-one, which is true and tells you nothing: the direction was never in question, because one arm's prompt is the question and the other's is the question plus the material. The quantity in doubt is the magnitude, and for strongly right-skewed ratios the right summary is the median with a distribution-free interval, the 42.0 to 56.0 above. It licenses a statement about these three corpora. It does not license one about workloads in general, or about dollars.

Replicate it

Teach the same handbook, ask the same question, compare the prompt you sent: teach_my_doc(document=...) then ask_learner(question=...). The document, the 68 questions, the three token counts each and the counter itself are in the data repository under token-efficiency/.

Also true

These are prompt tokens only. Completion tokens were not counted, on the assumption that they are similar across arms and would dilute every ratio equally. That assumption is reasonable and it is not a measurement, and the conversion from a token ratio to a bill ratio depends on it, which makes it the largest gap in this piece. The measurement that closes it is specified below.

At equal accuracy?

A token ratio means nothing if the cheap arm answers worse. This is the question we can answer least completely, so it is split into three buckets that must not be mixed: one live paired measurement of ours, our own pass rates, and the published literature on the alternatives. Figure 5 is the third bucket only.

Bucket one: our one live paired measurement

On one question, both arms were served and graded live on the same day. The retrieval control answered correctly using 858 prompt tokens. The learned arm answered correctly using 18, a ratio of 47.667. That is the only point in the whole distribution where control-arm correctness was measured rather than assumed, and n is one. It is reported precisely because it is the shape of measurement the rest of the distribution lacks.

Bucket two: our own pass rates, each on its own metric

One number governs how all of those are read. Repeating a training run with nothing changed moves about one row of 8 to 12, in either direction. That is a measured floor, established in a run where nothing was unlearned, and every strict per-row count above sits inside it. We publish the strict count anyway.

Bucket three: what the literature says about the alternatives

The cleanest controlled comparison toggles browsing on and off with the same models, prompts and grader. On biographies, which are long-tail entity facts, error rates fall by factors of 4.25 and 7.39 when browsing is enabled. On broad conceptual questions retrieval barely helps, and for one model it made things slightly worse. That is the most useful shape in this literature: retrieval's benefit is concentrated almost entirely on the long tail. There is also a theoretical floor. A 2025 analysis proves a base model's error rate on facts is lower-bounded by the share of those facts appearing exactly once in its training data.

Also true

Bucket three bounds the alternatives and says nothing about us: we have not run our system on any of those benchmarks. The phrase "hallucination rate" also means at least five different things across those sources, so no two of them appear in one sentence here. Abstention silently drives all of them, since a model halves its rate by refusing more, having learned nothing.

From a token ratio to a bill ratio

Here the honest number is much smaller than the exciting one, for an old reason. If a technique improves one part of a system, the total improvement is capped by how large that part was. Output tokens are 11.2 percent of an agentic coding bill and nothing on the input side touches them. Figure 6 draws the consequence.

What the data says

Work it through at a checkable pace. The bill splits 29.2 percent new input, 59.5 percent re-read, 11.2 percent output. Shrink all input by a factor R and the remaining bill is 0.887 over R, plus 0.112. At R of 10 that is 0.201, or 5.0 times cheaper. At our measured 52 it is 0.129, or 7.8 times. At infinity it is 0.112, or 8.9 times and no more. So our 52 already captures 7.75 of the 8.93 available, which is 87 percent of everything an input-side technique could ever win. The remaining headroom is small, and that is the useful thing to know.

The conservative reading is lower and belongs beside it. If learning displaces only the re-read portion and leaves genuinely new input alone, which is the honest assumption since new code and new user text are new, the bill goes to 0.292 plus 0.112, or 2.5 times cheaper. The defensible band on an agentic coding bill is therefore roughly 2.5 to 7.8 times, and where you land depends on how much of your prompt is material that could have been learned in advance.

So, a hundred times cheaper?

No, and the reason is arithmetic. To reach a hundred times on a whole bill the total has to fall to 1 percent of what it was. Output alone is 11.2 percent. So even with input driven to zero you would still need output tokens to fall by a further factor of 11.2. We have not measured output tokens at all, so we claim none of that. The honest bounded statement is this: on prompt tokens, for a question whose answer has been learned, 52 times, median, over 68 questions on three corpora. On a whole agentic bill, projected from a measured cost split, between 2.5 and 7.8 times, with a hard ceiling of 8.9 that no input-side technique can pass.

Why this is hard

The temptation is to quote the largest true number, and the largest here is 2,649 times, a real ratio for a real question against a real control. It is also the ratio for putting 22,000 words in a context window on every turn, which nobody sensible does, and it converts to under 8.9 times on a bill just like every other ratio above about a hundred. Quoting it would be true and would mislead.

Also true

This is a projection, not a measurement. It composes two measured quantities, a cost split from somebody else's agentic coding trace and a prompt ratio from our own single-document question answering, and the composition is untested. Whether a 52 times prompt ratio survives on a repository-scale coding workload is open, and the honest experiment is a large one.

What it costs to learn instead

A ratio per question hides a cost per document. Teaching is a training run, it takes minutes and it costs money, and it happens once. So the buyer's question is not the ratio, it is how many questions it takes to get the money back. Figure 7 answers it at published list prices.

What the data says

Measured teach costs, in our own compute: nine structured facts took 1,120.6 GPU-seconds for about 62 cents. One 706-word document took about 50 cents on its own and 1.50 to 2.00 dollars through the full demonstration path. Two 22,000-word corpora together took 6,071 GPU-seconds for about 4.60 dollars, so about 2.30 dollars each. The per-question saving is the token difference multiplied by the price: 576 tokens for the handbook against retrieval, 1,243 against the whole document, and 57,884 tokens for the 22,000-word corpus against the whole document. Divide and you get the figure.

What it means for building

Learning is worth it when a document will be asked about many times, or when it is large enough that shipping it is expensive. A reference manual a whole team queries daily clears break-even in a week. A document uploaded once, asked about twice and never seen again does not, and retrieval is the right tool for that. Any honest system does both, and chooses.

Also true

Those are our measured compute costs on current hardware and settings, not a customer price, and they will move. The break-even calculation counts input tokens only, for the same reason the ratio does.

The optimum, and where it actually lands

The best case is simple to state and we have measured it. If the material is learned, the prompt is the question: 11 to 37 tokens across every lane, flat in the size of what was learned. There is nothing beyond that, since you cannot ask a question in fewer tokens than the question.

Practice sits below that optimum for four reasons, all worth naming.

The long tail still wants retrieval. The controlled browsing comparison above puts retrieval's benefit almost entirely on rare entity facts. Material that is genuinely new to the world sits in that regime until somebody teaches it, and something has to fetch it in the meantime.

Learning is a decision taken before the question. Retrieval is one taken during it. A system that only learns cannot answer about a document uploaded a second ago, so the honest architecture keeps both routes.

The teach cost has to be amortised, which is the whole of the previous section.

Our serving path has an 8,192-token context window. A real constraint, and it cuts both ways: it is why the whole-document control was computed rather than served for the large corpora, and it means that for us the alternative to learning a 22,000-word corpus is not a long prompt but retrieval, or nothing.

One more reason not to assume a bigger window solves this: a 2025 study measured accuracy as input length grows, with retrieval held perfect and irrelevant tokens stripped out, and found drops of 13.9 to 85 percent across five models. Length itself costs accuracy even when retrieval is flawless. Cost moves the same way. One forecaster projects token prices falling about 95 percent by 2030 while inference cost per agentic workflow rises more than fivefold through 2028, because tokens per task grow faster than price per token falls.

What a token costs

Published list prices per million tokens, observed on 2026-08-25. Promotional rates change and at least one row below is explicitly temporary.

Three things move real cost more than the headline row. Anthropic's newer tokenizer produces about 30 percent more tokens for the same text, so a price cut across a model generation is not the whole story. OpenAI, Google and xAI roughly double the input rate above about 200,000 tokens of context, while Anthropic bills a 900,000-token request at the same rate as a 9,000-token one. And Google alone bills cache storage by the hour, from 50 cents to 4.50 dollars per million tokens per hour, on top of the cache-read rate.

What matters more than any row above is what output actually costs once you pay for the input that produced it. Under the measured agentic mix of 12.5 fresh plus 281.2 cached input tokens per output token, the effective cost per million output tokens is 456 dollars for Claude Fable 5 against its 50-dollar list price, 228 for Opus 5, 182 for gpt-5.6-sol, 172 for grok-4.6 and 93 for Gemini 3.1 Pro. The list price of output is not what output costs, and the difference is the re-reading.

The honest gaps, and the measurement that closes the biggest one

Four things here are weaker than they look, and one is fixable this week.

We did not count completion tokens. The 68-question distribution counts prompt tokens only, assuming completions are similar across arms. That is a reasonable prior and not a measurement, and the bill ratio depends on it, so it is the gap that matters most. Three outcomes are possible: the learned arm answers more tersely, because it is not paraphrasing a retrieved passage, and the bill ratio beats 7.8. Or the two match, and 7.8 stands. Or the retrieved text constrains the control into shorter answers, and the ratio is worse.

The measurement that closes it is small and specific. Serve all three arms live on the same checkpoint in one window, on the same 68 questions so every new number pairs to an existing one. Count prompt and completion tokens from the served usage fields rather than re-counting. Grade every arm with the same strict grader, which also turns control-arm correctness from one point into 68. Then report the paired total-token ratio, the subset where both arms are correct, a Wilcoxon signed-rank test on the paired difference and an exact McNemar test on the correctness disagreements. The bar is written down in advance: if the median paired ratio on the both-correct subset falls outside a factor of 1.5 of the prompt-only ratio, the arithmetic here is re-derived before publication rather than after.

The bill projection composes two measurements never made together: somebody else's agentic coding cost split and our own single-document ratio. Whether the ratio survives on a repository-scale workload is unmeasured, and the honest experiment is a large one.

Control-arm correctness has one live point. One question, both arms correct, 858 tokens against 18. The re-measurement above turns that into 68.

There is no energy number here, because there is no energy number to have. No laboratory publishes energy per token. Every official figure is per query, and they are not comparable to one another. The published spread from a simple text query to an agentic reasoning query is roughly a thousandfold, from about 0.05 to about 50 watt-hours on one workload-level estimate, which is both the most decision-relevant figure in that literature and the reason a token-to-joule conversion would be fiction. The International Energy Agency notes that even at a generous blended average of one watt-hour per query, ten billion queries a day comes to roughly 3.6 terawatt-hours a year, under one percent of what data centres already consume, so published per-prompt energy cannot explain the build-out.

Method, at the level we can state it

The base model is Qwen3.6-27B in bf16, frozen. About 1.14 billion parameters are trainable. Teaching runs a short warm-up phase and then a single continuous optimizer over the material, with no task identifiers and no boundary signal of any kind. Serving uses an 8,192-token context window. All three prompt variants were counted with one counter identical across arms, chosen over any per-model tokenizer precisely so the arms stay comparable. The learned arm's prompt is the question with an empty context, exactly as the two live single-question measurements ran it. The retrieval control is BM25 over the top four chunks of the same source. The whole-document control is the question plus the entire source, computed rather than served.

Open questions

1. Does a single-document prompt ratio survive on a repository-scale agentic workload? The projection in this piece assumes it does and does not test it.
2. Do completion tokens differ across arms, and in which direction? Unmeasured. The design is above.
3. What share of all AI revenue is re-read context? Not derivable from anything public. It needs an input-to-output split published by a provider and no provider publishes one.
4. What is the energy ratio? Not derivable. No laboratory publishes energy per token.
5. How does the ratio behave as one learner accumulates fifty documents rather than one? Every measurement here is one document or one pair of corpora.
6. What is the retrieval control's accuracy across the whole distribution, rather than at one point?
7. Does the argument survive a million-token context window? The published finding that accuracy falls with input length even under perfect retrieval suggests a larger window changes less than its size implies, but we have not tested it.
8. Four of eight propositions in the teaching demonstration failed, all of them ties between a name and a specific organisation. The extraction fix shipped and extracted all four on a replay. Whether they then answer correctly at serving time, in several wordings, is the next measurement in that line.

Data and protocol

Everything below is published so the numbers can be checked, and the training documents are synthetic and invented precisely so that they can be.

Citation

Sources

Every industry figure above is numbered here with its source and the date it was observed. Figures we measured ourselves are in the data repository instead.

1. Anthropic run-rate: CNBC, 2026-08-17; TechCrunch, 2026-08-17. Anthropic has published no audited annual revenue figure for any year.
2. Microsoft revenue from OpenAI, 24.1 billion dollars inclusive of revenue-sharing: Microsoft FY2026 Form 10-K, SEC EDGAR, filed 2026-07-29.
3. OpenAI run-rate and enterprise crossover: CNBC, 2026-08-14; Reuters via Investing.com, 2026-08-13.
4. OpenAI audited 2025 revenue and operating loss: Where's Your Ed At, 2026-06-16, verified by the Financial Times.
5. The two companies count revenue differently: Forbes, 2026-03-25.
6. Valuations: Anthropic Series H; OpenAI financing announcement; CNBC on the xAI merger; Caixin on DeepSeek. Market capitalisations at close 2026-08-25: stockanalysis.com.
7. Segment disclosure limits: Alphabet Q2 2026 8-K; Meta Q2 2026; SpaceX S-1; Alibaba 6-K.
8. Inference share of compute: Gartner, 2026-08-10; Deloitte TMT 2026; Epoch AI; NVIDIA's 40 percent and the declined restatement, NVIDIA Q3 FY2026 call transcript.
9. The agentic coding trace: Zhu, Jacob, Ma, Pan, Wang, Krishnamurthy and Kasikci, TraceLab: Characterizing Coding Agent Workloads for LLM Serving, arXiv:2606.30560, 2026-06-30.
10. The porting case study: Bun, 2026-07-08. General traffic ratios: OpenRouter State of AI. Cached-prompt share of agentic burn: ppc.land on OpenRouter data, 2026-08-25. Caching case study: ProjectDiscovery, 2026-04-10.
11. Prices observed 2026-08-25: OpenAI, Anthropic, Google, xAI, DeepSeek. Caching terms: Anthropic, OpenAI.
12. Recall and hallucination: GPT-5 System Card; SimpleQA; AA-Omniscience; Why Language Models Hallucinate; Large Language Models Struggle to Learn Long-Tail Knowledge.
13. Retrieval helps and leaves a residue: FreshLLMs; Vectara hallucination leaderboard, updated 2026-05-11; Stanford RegLab on legal research tools; ReDeEP.
14. Long context is not free: Context Length Alone Hurts LLM Performance Despite Perfect Retrieval; Lost in the Middle; Context Rot.
15. Cost per agentic workflow rising while token prices fall: Gartner, 2026-08-17.
16. Energy: Google on inference energy and its technical paper; Microsoft Research in Joule; Epoch AI; IEA, Key Questions on Energy and AI, 2026-04-16.
17. Token volumes: Google I/O keynote, 2026-05-19; Menlo Ventures on OpenRouter, 2026-05-26.

Figures marked as measured come from a trace or a benchmark. Figures marked as estimated come from an analyst or a third party. Figures marked as projected are forward-looking. Where a quantity does not exist publicly, this piece says so rather than approximating it: no provider publishes an input-to-output token split, no laboratory publishes energy per token, no published split of cloud capital expenditure into training and inference exists, and no current bottom-up study attributes enterprise value to AI at the product level.