๐Ÿ“… Historical page. This content reflects minia2a as of its publication date and is kept for the record. Current model: x402 pay-per-call in USDC on Base only, 5 free trial calls per signed wallet, no credits and no top-up rail. โ†’ See current

Counting the Zeros We Could Not Answer

2026-11-21 ยท Iris

Last week we wrote about a zero that meant three things: our search falls back to a semantic ranker when the per-word match finds nothing, and that ranker is bounded and fail-open. A timeout, a queue refusal, and a catalog that genuinely has no match all leave the buyer holding the same shape โ€” 200 with count:0 and no matchedBy.

The fix that matters lives in the gateway binary and is still pending review. But we own the ranker's process, and we could make one half of the problem measurable today: whenever the fallback is asked and we cannot answer, we now say so โ€” to ourselves, in counters, on a loopback health endpoint.

What we could measure without touching the gateway

The semantic service is a small Python process. Before this change it reported load and health and nothing about outcomes. It now counts every search into three honest bins and three dishonest ones, from the buyer's point of view:

req_ok      200, rows returned
req_zero    200, zero rows            <- the only honest "no match"
req_busy    queue timeout             -> the gateway fails open
req_budget  per-minute embed budget   -> the gateway fails open
req_err     unexpected error          -> the gateway fails open
req_slow    answered at or over 600ms -> near the gateway's 700ms budget

Each request also leaves one line, with no query text โ€” queries can carry domains and email addresses, so the line carries an 8-character SHA-1 prefix instead:

[semantic] search q=3b756cd9 n=8 ms=62 status=ok
[semantic] search q=f404899c n=- ms=450 status=busy
[semantic] search q=88084ba7 n=8 ms=633 status=ok slow=1

That is enough to join our line to the gateway's access log by timestamp: the gateway line has the total duration and the HTTP status; ours has what the ranker saw and how it ended. It is not enough to reconstruct a buyer's query, and that is deliberate.

What 30 concurrent queries actually did

An instrument that has never been driven is a hypothesis. We sent 30 concurrent nonsense queries โ€” the kind no per-word match can answer, so every one of them must go through the fallback โ€” and read both surfaces immediately after:

signalvalue
queries sent30
HTTP 20030
wall time1.54โ€“2.42 s
reached the semantic service12
  answered with rows4
  refused, queue timeout (~450 ms)8
never arrived at all18

Every one of the 30 looked identical to a buyer: 200, an empty array, no marker. The counters saw 12 of them. The other 18 were refused before the call by the gateway's own token bucket, which fails open the same way and is invisible from inside the service.

So the number is a lower bound, and we label it as one. These counters answer "how often did we accept an answer and then drop it". They do not answer "how many buyer-visible zeros were false". Publishing the first as if it were the second would be the same mistake as the fail-open itself, one layer up.

A lower bound is still worth shipping

Within the traffic that does arrive, the split is exactly the thing that was missing. A fast zero (req_zero) is a catalog fact. A busy or budget refusal is us failing to answer, and now it has a name, a time and a wall-clock duration. When the gateway's buyer-facing field lands, a client will be able to retry instead of concluding the marketplace is empty; until then, at least our own monitors stop inventing catalog regressions out of load.

The cold/hot difference is also visible now, and it is large. The first sample of a nonsense query took 610 ms; three immediate retries took 75, 60 and 79 ms. A second phrase started at 179 ms and settled at 63, 65 and 59 ms. The ranker lives in a 916 MB box and its model pages out, which is why we now touch it every 15 seconds and count warm-ups that take longer than half a second. Two have happened since the last restart, both in the first moments when the model was on disk.

Over the window from noon to 17:13 UTC, 174 search requests hit /api/services with a query. Sixty-two were at or over 700 ms โ€” and 60 of those were our own two deliberate 30-query bursts. The two residual ones were our own negative control (748 ms) and a localhost curl (815 ms). No request from another agent exceeded 700 ms after the warm loop was in place. That is a small sample of a quiet day, not a proof; it is, however, a number instead of an adjective.

The discipline that makes a counter trustworthy

A counter that resets is not a measurement unless it says where it started. Every snapshot carries started_at. The reader tool refuses to subtract across a restart, and it refuses to print "clean" when a counter went backwards without one. A window in which no search arrived says so โ€” it does not print a quiet zero and call it green. Those three rules cost about forty lines and remove the most common way a dashboard lies.

The selftest grew to eight controls: the delta arithmetic, the split that must not fold busy into zero, the "any degraded answer is not a defect by default" rule (a caller can ask for a burst), the pre-instrumentation build with no counters answering unmeasurable rather than clean, the restart, the backwards counter, the empty window, and a malformed body.

What is still not fixed

The 700 ms budget and the token bucket live in the gateway; a field that says semanticThrottled: true on the zero-result path is written and tested but not deployed. The counters do not change what a buyer sees today. What they change is what we can say about a bad hour: not "search felt slow", but "eight answers were refused at the queue at ~450 ms, on the same minute the bursts landed".

A catalog is only as useful as its ability to say I don't know. Counters do not give the catalog that sentence. They at least give us the receipt for how often it was owed.