A cached token costs a tenth of a fresh one, and most teams never turn it on

Ask most teams where their inference budget goes and they will talk about output tokens. Output is the visible part — it is what the model wrote, it is what the user read, and it is what the demo showed. It is rarely where the money is.

The money is in the prompt, and specifically in the part of the prompt you send again on every single call.

The ratio

Providers charge substantially less for input tokens they can serve from cache than for input tokens they have to process fresh — on the order of a tenth. That is not a rounding difference. For a workload with a large fixed preamble and a small variable tail, which describes nearly every agent, it is most of the bill.

And here is the part that catches people: on the Anthropic API, prompt caching is not on by default. It is something you opt into, per request, by marking which part of the prompt is stable. A framework that does not do it for you, and an author who did not know to, produces a system that pays full price on every call for a system prompt that has not changed in six weeks.

What the fixed part usually is

• The system prompt — often long, often carefully tuned, and identical every time.
• Tool definitions. These are sent in full on every call, and they are priced in full on every call, whether or not any tool is selected. A dozen tools with thorough JSON schemas is a substantial fixed cost per request.
• The conversation so far, in any multi-turn workload.

All three are stable across calls. All three are cacheable. Most of the time, none of them are cached.

Why it survives review

Because nothing breaks. There is no error, no warning, no degraded output — just an invoice that is roughly an order of magnitude larger on the input line than it needed to be. The failure is silent, and silent failures survive code review indefinitely.

It also does not show up in a per-call price comparison, which is how most model decisions get made. Two models compared on their published input rate will rank the same way regardless of cache behaviour. The comparison that matters is what your actual traffic costs against your actual cache-hit rate, and that is not on anyone's pricing page.

The second-order problem

Once you are caching, the cache changes what "cheaper model" means. Switching a call to a smaller model discards the cache entry for that prompt, and the fresh-input cost of the next call can exceed what the cheaper model saved. Routing that ignores this makes bills worse while appearing to make them better. That is the subject of the next post.

The Ledger reports cache-hit rate per workload, because it is the first number worth looking at. If you want us to look at yours, book an audit.