Most of what breaks in a payment system is not the payment path. It is the check you built to watch the payment path — a check that was green every time you looked, and would have been green no matter what happened.
We run a pay-per-call API marketplace, and over a few months we collected six incidents with the same shape. Each one is a different mechanism, and each one cost us real time. What they share is worth stating up front:
A signal is worth exactly what its ability to come back negative is worth. If you cannot describe the observation that would make it fail, it is not a measurement — it is decoration that happens to be printed in red when things are bad and green the rest of the time.
Our documented first-run script ended with something like this:
if npx minia2a-cli trial x402-time; then
CALL_OK=1
fiA developer whose five free trials were spent ran it and got, in green: ✓ That was a real signed call, answered by the live gateway. The line above it, in red, said the call had been refused. The script reported a refused call as a success.
The cause was not the wording. minia2a-cli exited 0 when the gateway refused with a 402 — and also 0 on a genuine success. The guard was a tautology; the exit code carried zero information on that path. It looked more reliable than a log line, which is exactly why it survived review.
The fix was to stop asking a proxy and read the producer: capture the CLI output, strip ANSI, take the three-digit code off its own Status: line, and treat only 200 as success. We also fixed the CLI, whose other failure paths already exited non-zero — the 402 branch was the lone outlier.
An endpoint returned trialExhausted: true for callers who had never made a call. That reads like a fact about the caller. It was a constant.
A subtler version: our own stats endpoint published trials.creditsRemaining as 152,180,000 next to trials.creditsIssued as 332,219. A "remaining" 458× larger than its "issued" is not a remainder. Reading the handler explained it — both fields were populated from the same variable, SUM(credits) FROM agents, after the meaning of "remaining" was deliberately repointed to a different quantity. The value was intentional; the name never moved with it, and the result was a public JSON object that contradicted itself.
None of this throws an error. A structurally-pinned field returns 200 and a perfectly well-formed body. If you are consuming someone's API, the useful question is not "is this field present?" but "what value could this field hold that would tell me something?"
When a wallet runs out of free calls, the refusal body read:
Free trial calls exhausted for wallet 0x0ab5… (0/5 total). Pay per call via x402.(0/5) parses as "0 remaining out of 5" and equally as "0 used out of 5". The word exhausted in front of it pushes a reader toward the second reading — so the sentence announces that 0 of 5 calls were used, immediately after saying they are all gone. A CLI printed this verbatim to developers, which is how we found it.
Two numbers with no names are not a measurement. The repair is one line — state the semantics in the sentence (5 of 5 used) rather than relying on a reader to infer them from position.
A related trap: the proposed fix swapped a hardcoded 0 for trial.Remaining. That looks like an improvement and changes nothing, because the variable is structurally zero in that branch. A fix whose output cannot change cannot be verified from outside — which makes "we fixed it" unfalsifiable. If a change is genuinely behaviour-preserving, say so; do not present it as a repair.
We keep a read-only reference copy of a service's source on the machine it runs on. Before quoting line numbers from it, we checked whether it was stale. The directory's mtime said it had not been touched in seven days, and the running binary was two days newer — which would have meant rebuilding from week-old source and silently rolling the service back.
The files told the opposite story. Their mtimes were two minutes before the binary was built. The directory mtime was frozen because a directory's mtime only advances when entries are added, removed, or renamed — overwriting a file in place leaves it untouched.
A timestamp is not evidence of freshness; it is evidence of one specific filesystem event. We were one command away from rebuilding a live payment service on the strength of a number that looked more reliable than the ones next to it.
After fixing the script from incident 1, we checked that the old claim was gone by grepping for it. Hit count: 2.
The sentence was still in the file — now inside a conditional branch, so it no longer printed on the refusal path. A search for a string tells you the string exists. It cannot tell you whether the behaviour changed, and a fix that is about behaviour will often leave the old string in place. The criterion has to be behavioural: run the thing, observe what it prints.
The same mistake has a mirror image on the fixing side: a rewrite rule pinned to an old literal value can only ever fire once. Rules that encode today's numbers rot silently, and they rot in the direction of looking fine.
A liveness probe tested endpoints for 404 and treated everything else as alive. Then a server began answering a dead route with 403, and the probe went blind — reporting health for endpoints that no longer existed. Blindness and health produced identical output.
Status codes are the producer's choice and can change for reasons that have nothing to do with your question. Where the body carries a machine-readable field — a retryable flag, an error code, an accepts array — key on that instead. And write down the negative case explicitly: if the probe cannot distinguish upstream failed from resource genuinely absent, it will eventually report one as the other.
For every check you write, ask: what would this print if the thing it watches were broken?
If the answer is "the same thing it prints now", you have found an always-green check. That is the whole test. It requires no tooling, and it catches every incident above.
Two corollaries we now apply by default:
Prefer the producer's own machine-readable field to any proxy. An exit code, a status code, or a human sentence are all summaries someone else wrote for a different purpose. If the producer gives you a field, read the field.
Make the negative case reachable in a test. We verify by running the failing branch for real — a spent wallet, an absent binary, a real 402 — rather than by inspecting the code that handles it. Reading the code tells you what you intended. Only running it tells you what happens.