The cache that served old code against new data

This site is a static export. Every file under _next/static has a hash of its own contents in its name, which is what makes the deploy configuration below safe rather than reckless:

gcloud storage objects update "gs://$BUCKET/_next/static/**" \
  --cache-control="public, max-age=31536000, immutable"

A year at the edge, and the browser is told never to revalidate. That is fine as long as the name is honest. If the contents change, the hash changes, the name changes, and the new name has never been cached by anyone. The old file can sit in a cache until the heat death of the universe because nothing will ever ask for it again.

Where the promise broke

We do not build this page from source. It arrived as an export and we rebrand it with a set of Python passes that rewrite copy, swap artwork, and replace a wordmark — operating directly on the emitted chunks. One of those passes edited a chunk in place. Same filename, different bytes.

Nothing failed at deploy. The sync ran green, the objects updated, the summary printed a commit SHA. The page loaded. What did not load was a chunk that a returning visitor already had pinned for a year under a name that now meant something else. Their browser held the old bytes and paired them with the new page, and the comparison table rendered with two columns empty.

The clue was in the shape of the report. It was not broken for us. It was broken for people who had been here before — which is the set of people whose opinion had already been formed.

Two ways to fix it, one of them wrong

The tempting fix is to relax the header. Drop immutable, cut the max-age, let the edge revalidate. It makes the symptom go away in an afternoon and it is the wrong move: it pays for a build mistake with a permanent tax on every visitor, forever, and it leaves the actual defect — a filename that lies — sitting in the tree to be tripped over again.

The right fix restores the invariant instead of working around it. A pass walks every chunk it edited, computes the real hash of what is now on disk, and renames the file to match. The names stop lying, and immutable becomes true again rather than merely asserted.

The rename has a tail

Renaming chunks introduces its own problem, and it is worth knowing about if you ever edit a webpack output by hand. The runtime does not reference chunks by filename. It carries a table mapping numeric chunk id to a bare hex hash and assembles the URL at request time. Rename a file and that table is immediately wrong.

Our first attempt updated the table as part of the rename, which works exactly once. Re-run the pipeline from a pristine page and the chunks are renamed again, while a hash written by an earlier generation is still sitting in the map. The failure is loud but late: the page paints, hydration starts, webpack requests a chunk under a hash nothing was ever written to, and the section dies with a ChunkLoadError. We watched 6655.131a790a754df01e.js 404 while the file on disk was 6655.b189fb9f505c5626.js.

Chasing renames across generations is the wrong shape of fix, because it requires every pass to be correct, in order, permanently. The version that holds reads the truth off the filesystem: for every id in the map, find the file whose name starts with that id and write its real hash. It does not care how many renames came before it. It is idempotent and it converges from any state.

What we actually learned

• A content-hashed name is a claim about the bytes. Any tool that edits build output has taken on responsibility for keeping that claim true.
• Caching bugs sort your users by loyalty and break it for the ones who came back. You will not see it in your own browser.
• When a pipeline step can run twice, derive its state from the filesystem rather than from what the previous step believed. Convergent beats correct-in-order.
• Weakening a cache header to hide a build defect is a trade you keep paying for long after you have forgotten what you bought.