A Timeout Is Not an Answer

2026-11-07 · operational notes

Five services in our review queue were reported to have "answered 0". None of them answered anything. The probe gave up after ten seconds and the string 0 came from our own socket layer, not from their server.

The verdict those five got was HOLD, which writes nothing — no listing approved, no listing rejected. So nobody was harmed by the wrong number. What was wrong was the sentence next to it, and the sentence is what the next person reads.

The shape

A probe returns four things: status, headers, body, elapsed. When the request dies before a response exists there is no status to return, so the function returns 0 and puts the exception text where the body would go:

try:
    ...
    status = r.status
except Exception as e:
    return 0, {}, str(e)[:300].encode(), time.time() - started

That is a reasonable sentinel. The bug is one if-statement away. The classifier had branches for 402, 400, 403/429, 404/405/501, and 2xx — and then a catch-all tail for everything else:

return "HOLD", "unpaid call answered %d, not a 402 challenge (%s)" % (status, method)

So 0 fell into "everything else" and was narrated in the same grammar as a real reading. The egress log next to it showed the tell — 0 with an elapsed of 10.02s — but the human-readable line said the service had replied.

Why it matters more than it looks

A wrong reason is not cosmetic when the reason is the artifact. Three properties make this shape expensive:

• It survives review. "answered 0" reads like a service that returns a nonsense status. A reviewer skims it, believes the remote end is broken, and moves on. The actual state — we did not measure — never enters the record.
• It aggregates in the wrong direction. Timeouts correlate with a host being slow, blocked, or geographically unlucky, which correlates with us, not with the seller. Stamp enough of them and a healthy service looks flaky in our history.
• It is indistinguishable from a fix. Once the wording is right, a run that says "no response (timed out)" is actionable — re-probe, change the vantage point, raise the deadline. "answered 0" is a dead end.

Three shapes of the same mistake

This is a family, not an incident. Across our gateway and our tools we have found the same error in three distinct places, and each has its own guard now.

The middle row is the one that costs money. A JSON-RPC node that returns {"result": null} has told you it does not know the balance; printing balance: 0 tells the caller the account is empty. Same bytes upstream, opposite claim downstream. The fix is not a better default — it is refusing to have a default at all: propagate "no value" as its own outcome, so the caller can retry or fail loudly.

The third row is subtler. A shared helper that fetches upstream and returns parsed JSON will happily parse a 502 error page into {} if nobody checks r.ok. Every handler downstream then reports a confident empty result. The helper looked correct because it was correct on 200s; the defect only exists on the path no one exercised.

What the fix actually is

For the reviewer, the change is a branch before the catch-all, and it does not touch the verdict:

if status == 0:
    return "HOLD", ("no response from the endpoint on %s (%s) — the probe never got an "
                    "HTTP answer, so this is a measurement gap, not a reading about the "
                    "gate; re-probe before judging" % (method, err))

The verdict stays HOLD on purpose. A timeout is not evidence about a payment gate, and the safe bucket for "no evidence" is the bucket that writes nothing. Only the sentence changes: from a claim about their server to a statement about our measurement.

Five controls pin the wording and none of them check the verdict — that one was already pinned. They assert: the reason never contains "answered 0"; it carries the transport error when there is one; it says the words "measurement gap"; it does not emit an empty parenthesis when the exception has no text; and it names the method, because "no response on GET" and "no response on POST" mean different things when you go back to re-probe.

Selftest went from 41 controls to 178 over the life of this tool. Every one of them is a fixture driven through the classifier — no network, no third-party requests. That matters for the same reason the bug did: the only trustworthy control is one you can run against a state you constructed, not against a host you hope is up.

The general rule, if there is one: a measurement function should have a return value that cannot be confused with a measurement. 0 is not that value. Neither is null, {}, [], or false — each of them is already a plausible answer to some real question. If your code cannot tell "I did not find out" apart from "I found out it is nothing", it will eventually tell somebody the second thing.