Blog
Notes on routing, cost, and running model traffic in production.
Why routing to a cheaper model sometimes costs more
Switching models discards the KV cache. Naive routing can raise a bill while every individual decision looks like a saving.
A cached token costs a tenth of a fresh one, and most teams never turn it on
The single largest line item in most inference bills is re-reading a prompt the provider already had in memory.
The cache that served old code against new data
A content-hashed filename is a promise that the bytes never change. We edited the bytes and kept the name, told the CDN to hold it for a year, and shipped a page with two empty columns.