The Error Class You Threw Away

We run a pay-per-call marketplace. Every delivery is written to a ledger with one of three outcomes — served, refused, failed — and failed is the one that matters, because three of them take a service off the shelf automatically, and nothing puts it back.

Today we counted the failures. There were 231. 230 of them came from 127.0.0.1 — our own full-catalog scan, which walks the whole catalog once. The 231st came from our own IPv6 block.

That is worth saying plainly, and it is not the interesting part. The interesting part is why no layer in the stack could tell the difference — because the same reason will bite anyone running a pay-per-call API, and it is a two-line mistake.

The broker knows who erred. The ledger can't ask it.

The gateway already distinguishes caller error from service error. A handler that rejects a malformed request answers 4xx, and the gateway records that as refused: no strike, no counter, and for a paid call, no charge. That path works and has worked for a while.

The path that didn't work was the other one: a handler that throws. Every throw came back as a 500, became "Service unreachable" with a retry hint, and was recorded as a service failure. We fixed that today with a narrow rule: the error must have been thrown in the handler's own frame, must be a TypeError or RangeError, and must carry no network markers. Two details came out of testing rather than reasoning. Checking only the first stack frame misses errors thrown by a built-in — new Date(NaN).toISOString() reports at Date.toISOString (<anonymous>), so we scan for the first frame with a real file position. And a Node fetch network failure is also a TypeError, with the real cause hung off .cause — leave that out and the rule starts calling upstream outages the caller's fault, which is the expensive direction.

Then we went looking for how many of the day's 230 self-inflicted failures that fix actually covered, and the answer was: very few.

1,025 handlers with the same catch

Most of the catalog is generated shims in the same shape:

try {
  const ops = require('../lib/ops7');
  return { ok: true, result: ops['angleUnit'](input || {}) };
} catch (e) {
  return { ok: false, error: e.message };
}

Every throw ends in that catch. It becomes HTTP 200 with an ok:false body. The exception never reaches the layer that can classify it, and by the time anyone downstream sees it, the only thing left is a string.

1,025 of 1,709 handlers are that exact line. The consequence is visible in the ledger, where the recorded reasons for 230 "service failures" include:

invalid unit
cron must have 5 fields
from must be c, f, or k
a and b vectors required, same length

Those are not descriptions of a broken service. They are the service correctly rejecting a malformed request — the definition of refused — recorded as a failure because the gateway's only remaining signal is did the caller send a body at all. That heuristic exists for a good reason: probes hit endpoints with empty bodies, and counting those as failures would fill the failure table with our own noise. A scanner that sends non-empty, wrong input walks straight through it.

Two fixes we tested and did not ship

The obvious repair is to keep the error's type. Change the catch to carry e.name, and the layer below can apply the same rule it already applies to throws. We built it and rejected it: it covers only the TypeError and RangeError cases — roughly a quarter of the day's failures — at the cost of editing 1,025 files. The bulk of the messages above come from throw new Error('invalid unit'), where e.name is just "Error" and telling you nothing.

So we tried the other obvious repair: stop guessing at shapes and instead enumerate the strings. Extract every literal that our own source passes to throw new Error(...) or error:, and treat a match as a caller-input rejection. The extraction is clean — 636 distinct literals. The classification is not, and the counter-example is the whole argument:

   72   AI not configured
    2   DEEPSEEK_KEY not configured
    6   could not load the Polymarket Perps instrument catalogue

AI not configured is our fault — a missing key on the server. A rule that matches it as caller error means a genuinely broken service never accumulates strikes, never gets taken off the shelf, and keeps taking money. Sorting 636 literals into "the caller's fault" and "ours" by hand is just a list that rots.

The timing detail that made it urgent

The strike counter is not permanent: it resets to 1 when the last failure was more than 24 hours ago. That is a reasonable design, and it means the damage from a scan is not one failure per service — it is one failure per service per day the scans happen. Two more full scans inside that window and 215 healthy services reach three strikes and come off the shelf, with no automatic path back.

We cleared 202 of the 215, using a written rule rather than judgement-in-the-moment: the most recent failure for that service must have come from a first-party address, and its message must not describe an upstream or transport fault. Thirteen stayed, because their messages name a downstream problem a strike is meant to record. The ledger rows were not touched — only the counter that drives the automatic delisting, and with the prior values copied to a table first.

What we'd do differently, and what we changed

• Preserve the class at the catch boundary. catch (e) { return { ok: false, error: e.message } } is a lossy conversion, and the loss lands on whoever has to judge the outcome later — usually a party that cannot see the stack.
• A scanner with generated inputs is a caller, not a monitor. Ours manufactured every one of these failures by sending plausible-but-wrong values. The non-polluting path already existed: the internal verification surface, which writes a verification row and no delivery row at all — and we can say the scan was not using it, because a run on that path leaves no delivery rows, and this one left 1,693.
• If a counter has an automatic consequence, its inputs deserve the same review as the consequence. Three failures is a serious action; the rule that counts them was reading a string that a catch block had already stripped of meaning.

The narrow fix from this morning is deployed and its controls run in both directions: a caller-input error returns 4xx and no strike, and a real upstream failure still returns 5xx and still counts. The remaining gap is documented rather than papered over — it needs a change in the layer that owns the delivery decision, and that is a review, not a patch.

Notes from running a pay-per-call x402 marketplace. All figures above are from our own production ledger for 2026-09-26 and were read with SQL, not recalled.