Yesterday we published the count: every strike our gateway recorded came from our own scanners, and no service had been marked bad by anything a service did. That was the useful half. This is the other half — we went looking for inputs to that counter which are not our scanners, and found one that our own request handling feeds.
The counter in question is short and severe. Three failed deliveries take a service off the shelf. There is no automatic path back on. So the question "what may increment this" is not a monitoring question — it is the input side of an irreversible action, and it deserves the same review as the action itself.
Every delivery lands in a ledger with one of three outcomes:
| Outcome | What it means | Counts toward delisting |
|---|---|---|
| served | The service answered and the caller got the result | no — and it clears the counter |
| refused | The caller's input was wrong; the service was never asked | no |
| failed | We tried and did not deliver | yes, +1 |
The middle row is the one that has to be earned, and the reason is symmetrical. Recording a caller's mistake as a service failure takes a healthy service off the shelf. Recording a real service failure as the caller's mistake does the opposite damage: a broken service never accumulates a strike, keeps its place in the catalog, and keeps taking money. Both mistakes are expensive, in opposite directions, which is why the classification has to be made where the evidence still exists — at the point of failure, not later from a string.
That was yesterday's post. Today's is a case the three-row model does not cover on its own, because there is a fourth situation hiding inside it: we refused the call, but not because of anything the caller or the service did wrong.
After a failed delivery, a service goes into a short cooldown — sixty seconds — during which we will not send it more work. The intent is narrow and good: if something upstream is falling over, do not pile requests onto it. During that window a paid call is answered with a refusal, and the caller is refunded in full, because they paid for something we chose not to attempt.
Then the refusal does something it should not. It records a delivery failure.
The reasoning behind that is not absurd, which is why it survived: "the service is in cooldown because it failed, and it is still failing, so this is another failure." But it is a closed loop wearing the costume of an observation. The cooldown was our decision, made sixty seconds ago from one data point. Re-observing our own decision is not new evidence, and there is nothing in it about the service that was not already counted once. The service is not asked anything at all — it does not receive the call.
Two things follow, and the second is the one that bites:
We had assumed this was mostly theoretical, on the reasoning that a service in cooldown is a service that is, by definition, broken — so the calls that hit the branch would be calls that were going to fail anyway. Reading the order of operations in the paid path disproved that:
1. classify the request body (empty input is the caller's error)
2. liveness probe against the service
└─ fails → refuse, do not settle, do not count
3. settle the payment
4. derive the refund amount and network
5. cooldown check
└─ in cooldown → refund, refuse, and count a failure ← here
The cooldown check is after the liveness probe. So every request that reaches it has just been probed and answered — the service is responsive at the moment we decide it is too broken to call. And it is after settlement, so the caller has paid. The branch is not reached by obviously-doomed traffic. It is reached by the comparatively ordinary case: a service that stumbled once sixty seconds ago and is fine now, being retried by a client that is behaving correctly.
None of this is exotic. It is the shape that every one of these systems converges on, because a refusal and a failure look identical from inside a request handler. Both are "we did not deliver the result." They are distinguished by a question the handler does not naturally ask: what evidence are we acting on, and is it evidence about the thing being judged?
A counter that only measures can be sloppy. A counter that measures and enforces needs two predicates, and they are not the same predicate:
Collapsing those two into one number produces a counter that is partly a measurement and partly a record of our own decisions, and no reader can tell which half a given row came from. We have now found two separate ways that went wrong — first a stripped error class, then a refusal counted as a failure — and they were not variants of one bug. They were two different violations of the same rule.
The failure mode of a monitoring counter is a wrong reading. The failure mode of an enforcement counter is a wrong action. They earn the same attention only if you agree that a number can be both.
One thing from yesterday is live and unchanged: the reset. It used to live in two hand-written places, and only the paid path reached either of them — so a service whose traffic is almost entirely trial calls could accumulate strikes and never clear one, because the success that would have cleared it was on a path that never wrote the counter. It now lives in a single place, in the delivery recorder, driven by the same outcome value that increments it. Success clears, failure increments, and both are consequences of one classification made once. That is the shape the rest of this post is arguing for, and it made the cooldown gap easier to see rather than less — because with one writer, "who else touches this" became a question with a short answer.
The cooldown path is not patched. Both candidate fixes — have the reset also clear the cooldown timestamp, or stop counting refusals as failures at all — land inside the paid refusal path, which is also the path that issues refunds. Changing when a payment is declined and refunded is not a patch to be made the same afternoon it was found, and we would rather write down the gap accurately than ship a rushed change into the one code path that handles money.
What we did do is make it visible. The pre-delisting check now prints, for every service carrying a strike, the source address of each piece of evidence — because the error text cannot tell you whose fault something was, but the address tells you who sent it. In our ledger, 248 of 264 recorded failures came from our own loopback and our own IPv6 block, and 16 came from a single external caller inside one half-hour window, all of them malformed input from that caller's own integration rather than a service defect. On the day a number moves that we did not move, that column is the first thing worth reading.
The counter is finally doing what it says. Three failed deliveries, attributable to the service, in a bounded window, take a service off the shelf. Getting to that sentence took two wrong versions and a read of our own source that we had been treating as a dead file. Both of those are worth writing down; a counter that is almost right drives an action that is entirely wrong.
Notes from running a pay-per-call x402 marketplace. The figures above are read from our production delivery ledger with SQL, not recalled, and the mechanism was confirmed against the deployed source rather than from memory.