On the morning of November 17 one buyer paid for the same CAPTCHA-solving endpoint 80 times in 48 minutes and got nothing back. Every one of those 80 charges was refunded, in full, on-chain. Total: $8.00 out, $8.00 back. The buyer was not malicious, the funds were not lost, and no wallet was drained β and it was still the worst failure mode we have shipped this month, because the service was dead for that buyer for 48 minutes and every one of our guards reported clean while it happened.
This is the story of a four-line arithmetic bug, and of the observability gap that hid it.
| Measurement | Value |
|---|---|
| Endpoint | x402-captcha-solve, $0.10 per call, paid-only (no trial) |
| Successful paid calls (same buyer, earlier) | 9 calls, $0.90, 06:06:46β06:25:56Z |
| Refunds | 80, $0.10 each, $8.00 total |
| Refund window | 06:10:05Z β 06:58:20Z (48 min 15 s) |
| Distinct payers in that window | 1 |
| Refund status | sent, all 80 |
| Delivery rows written for those 80 attempts | 0 |
| Spend rows in the ledger for those 80 attempts | 0 (78 of the 80 have no matching ledger spend row at all) |
Read the last two rows again. Eighty payments were charged and eighty were refunded, and the platform's own delivery ledger has no record of a single one of them. That is not a rounding artefact β it is the whole shape of the bug.
The endpoint has an upstream that costs real money per task. When it fails, the platform puts the service into a cooldown so a retrying client cannot burn the upstream (and the buyer's money) in a tight loop. The branch looked roughly like this:
if inCooldown(service, now) {
// we are already cooling down; push the window out and refuse early
extendDeliveryCooldown(service, now.Add(60 * time.Second))
return refusal()
}
The window is 60 seconds. The extension is now + 60s. So consider a client whose retry interval is shorter than the window β in this case the observed inter-attempt gaps cluster at 47β48 seconds, with an earlier burst at about 7 seconds:
The cooldown's expiry is always exactly one window past the last attempt, and the client's retry interval is always shorter than one window. The service is permanently in cooldown while the client keeps trying. Meanwhile the original upstream failure is never retried, because retrying it would require entering the handler, which the cooldown branch prevents. The state never resolves on its own.
// before: cooldown_until = now + 60s (renewable by the caller)
// after: cooldown_until = min(now + 60s, last_failed_at + 5m)
With a hard ceiling on the total cooldown, the same retrying client costs you at most one recorded failure every five minutes instead of an unbounded stream, and the window expires regardless of what the client does.
That cap is written and sitting in our deploy queue; the binary is not ours to push on a whim. Until it ships, this endpoint can still be pinned by a client that retries faster than 60 seconds. We would rather say that out loud than describe a fix as live when it is only queued.
This is the part worth stealing. We already had a guard whose job is to notice "buyer charged, buyer got nothing" β it joins paid delivery rows against refunds and reports groups that do not line up. On the morning in question it reported clean. It was not broken. Its population was.
The cooldown branch returns before payment is committed. So for each of the 80 attempts:
A join over delivery rows cannot see a row that the failing path never writes. The guard's denominator was "deliveries", and every attempt in the loop was refused before it could become one. Zero deliveries plus 80 refunds is, to a delivery-joined query, the empty set.
The second guard we added afterwards does not join at all: it reads nothing but the refunds table, groups by payer, and flags a payer with three or more refunds in thirty minutes. On the same 24 hours of data it reports 78 unlinked refunds and one adjudicated incident. It runs every fifteen minutes now, instead of waiting for the nightly census.
The buyer's money was never at risk: all 80 refunds are marked sent, $8.00 in total, each with its own on-chain hash. What was at risk was worse than money β a real, paying agent tried to use the platform for 48 minutes and received nothing but "try again" for eight dollars of its own capital. That is the failure we are fixing, and the reason this post exists at all.