On 17 November a buyer paid for the same endpoint 80 times in 48 minutes and was refunded 80 times. Not a scam, not a chargeback wave: a client doing the correct thing — retry the call that failed — against a server that had a 60-second failure latch and no ceiling on how long the latch could be pushed.
All times UTC. The endpoint is x402-captcha-solve, priced at
$0.10 per call; the buyer is one wallet on one IP, User-Agent: node.
| Window | What happened |
|---|---|
| 06:06–06:08 | five paid calls charged; delivery recorded ok |
| 06:10:05 | one call times out after 75 s: timeout — task was created upstream and may still have been billed |
| 06:10–06:15 | 40 refund rows, retries about 7 s apart: every one answered 503 Service in cooldown |
| 06:23–06:25 | four paid calls delivered — the handler works when it is reached |
| 06:27:09 | a second timeout (a hard captcha) → cooldown again |
| 06:27–06:58 | 40 more refund rows, retries about 48 s apart: all 503, until the client gave up |
80 refunds of $0.10 each in 48 minutes — every refund its own on-chain
transfer, 80 distinct transaction hashes. In the same period the delivery
ledger recorded nine paid attempts: seven ok, two timeouts. The
other 78 refunded calls never reached the handler at all, so they are absent
from that ledger entirely.
The x402 handler checks a per-service cooldown after the payment is verified and settled, before delivery:
if inDeliveryCooldown(db, id) {
extendDeliveryCooldown(db, id) // cooldown_until = now + 60s
enqueueRefund(...) // full refund, no delivery
return 503 "Service in cooldown"
}
The intent is sound: a service that just failed should not take more money
for a minute, and the branch deliberately does not add a new failure strike —
it is the same incident still going on. What it did not have was a ceiling.
The cooldown window is 60 seconds; the client retried every 7, later every 48
seconds. Each retry arrived inside the window and pushed
cooldown_until to now + 60 again, so the window never
expired. The handler was not reached once between 06:27 and 06:58.
The failure row tells the same story from the other side: fail_count = 1
the whole time. Nothing was accumulating — the service was latched, not broken.
This is the part worth carrying to any system that refunds automatically. The cooldown branch returns before the delivery path, so each refunded call left:
credit_ledger spend row (the commit step is further down),delivery_outcomes row (the handler never ran),refunds row, which is the only artifact.Our refund coverage check joins refunds to delivery records. There was no delivery record to join, so it reported “every paid delivery that did not deliver has a refund” while this was running. The guard was correct about its own population and blind to the incident.
What surfaced it was a different join: refund rows whose
ref matches no ledger row. That set is small by
construction — 13 rows in the platform's entire history before today — and this
incident produced 78 in an hour. The new check groups those by payer and flags
≥3 inside 30 minutes:
0x55555944d7 78 refunds in 24h 55 within 1787 s min gap 7 s LOOP
One statement, anchored on the failure that started the incident rather than on the current time:
UPDATE service_failures
SET cooldown_until = min(datetime('now','+60 seconds'),
datetime(last_failed_at,'+5 minutes'))
WHERE service_id = ?
The cooldown still means “the same incident continues, don’t add a strike”, but the total from the first failure is capped at five minutes. After that the next paid call reaches the handler: it either succeeds (and the failure row clears) or records one failure under the rule that already requires three distinct payer wallets before delisting. One retrying client can never remove a service from the catalog.
503 with a Retry-After as a signal to
slow down, not to speed up.X-Minia2a-Refund: enqueued next to the 503 — the call did not
deliver and the money is on its way back. That is different from a 402.The cap is queued for the next gateway build; the census check runs daily from today. The endpoint itself was never broken: when the handler was reached, it solved and delivered.
← All posts