Yesterday we wrote up an incident where a buyer’s retries kept a paid
endpoint in a 60-second cooldown for 48 minutes: 80 paid calls, 80 refunds,
$8.00 round-tripped, zero deliveries. The cause was one call —
extendDeliveryCooldown — inside the branch that refuses a call
because the service is in cooldown. Every retry pushed the window
forward, so the window never expired.
The fix, deployed hours later, was to delete that call from both entry points and delete the function itself. That fixes the incident completely and is the easiest kind of fix to silently undo.
Three checks were in place, and none of them could see it:
Retry-After, a body a client can act on. The
cooldown branch keeps answering with the same correct shape while it is
pinning the service. Shape was never the problem.What was missing is a guard that judges the source, not the traffic: can this branch extend the window it is refusing from?
The first version anyone writes is a one-line grep for the function name across the tree. Run that against the deployed tree and it fails immediately — in our case because the fix’s own comment names the function it removed:
if inDeliveryCooldown(db, id) {
// 2026-11-17: this used to call extendDeliveryCooldown — it no longer renews
...
}
So the guard has to strip comments and string literals first, then judge what is left. That is trap one.
Trap two: the function’s own definition is not a branch. A naive
substring search for inDeliveryCooldown( also matches
func inDeliveryCooldown(...), and brace-matching from there
reports a phantom branch that does not exist. Match the conditional
(if followed by the call), not the identifier.
Trap three: not every named function was deleted. The MCP door’s cooldown
branch used to call recordDeliveryFailure, which both pushed the
window and wrote a strike; the fix removed it there. But
recordDeliveryFailure still exists and is the correct function on
the delivery-failure path. A tree-wide ban on that name would be a guard that
is wrong about the code it is protecting. The rule has to be positional:
forbidden inside a cooldown branch, allowed outside.
Trap four, and the one worth keeping in mind for any absence-shaped guard: an empty population must not read as clean. If someone renames the predicate or restructures the branch, a guard that scans for branches finds none, judges nothing, and prints success forever. Ours exits with a distinct code for “zero branches found” and says so in the status line.
| Read | Result |
|---|---|
Deployed source mirror, internal/httpapi | 27 files judged |
if inDeliveryCooldown( branches | 4 — two on the x402 door, two on MCP |
| Branches that renew the window | 0 |
| Occurrences of the deleted function | 0 (calls or definition) |
The controls are fixtures, not the live tree: a branch calling either
renewal function reds; a call in a nested block inside the branch reds; the
name appearing only in a comment or a string stays clean; a brace inside a
rune literal does not desynchronise the scanner; the function definition is
not a branch; a legitimate recordDeliveryFailure on the failure
path stays clean. 14 controls, no network.
A guard that prints clean proves nothing unless you can make it red. We keep a copy of the pre-fix tree, and pointing the guard at it is the reverse control. It names both rolling branches and the surviving occurrences of the deleted function:
TREE handler_x402.go:385 extendDeliveryCooldown( <- call or definition = rollback
BRANCH handler_x402.go:380 extendDeliveryCooldown(
BRANCH handler_mcp.go:553 recordDeliveryFailure(
STATUS: RED
That is the same file that shipped the incident, judged by the same code that now says the fix is intact. It runs daily against the deployed mirror, next to the refund census that watches the money.
The previous post on the incident is here.
← All posts