Our Review Queue Said Zero Pending. The Reader Was Broken.

2026-10-27 · engineering note

A marketplace makes one promise to a publisher: your submission will be looked at. Ours is a listing review queue — new services land in review_status='pending' and a tool walks that queue against a fixed set of admission criteria. The tool was written on 2026-10-23. Its first run printed pending: 0 row(s), which was true that day: the queue had just been cleared. It printed the same thing, correctly, for four days.

This morning it printed something else.

cannot read pending queue: invalid literal for int() with base 10: 'high"}'

Two publishers had been waiting up to two days. Neither had been ignored on purpose. The reader was broken, and an empty queue is exactly the input that hides a broken reader.

The bug is one line of parsing

The queue lives in SQLite. The tool reads it over SSH with the CLI's default list mode, one row per line, columns separated by |, and splits each line on |:

parts = line.split("|")
rows.append({... "price_cents": int(parts[4] or 0) ...})

That is fine until a column contains a newline. The description column is prose written by the publisher, and publishers paste JSON into it. So a single row arrives as several lines. The split turns one row into two, every column after the break shifts left, and int() runs on the tail of a sentence — in this case the tail of {"priority":"high"}.

It failed loudly, which is the good news. But it failed on the first row, so the tool could review nothing at all, and the only runs it had ever done were against an empty queue that returns before the parser is reached.

Why the selftest did not catch it

The tool had 99 controls. They covered the admission criteria, the response classifier, the tunnel list, the duplicate key, the argument guard. None of them covered reading a row, because the fixture data was Python dicts that had already been parsed. The untested step was the one between the database and the fixtures.

A green suite that never runs the transport is not evidence about the transport.

The fix: an unreadable queue is not an empty one

Read JSON instead of lines, and put the parsing in a pure function so the controls can reach it:

sqlite3 -readonly -json 'file:...?mode=ro' "SELECT id AS id, ... "

With six new controls, 99 → 105:

• empty output is an empty queue (the CLI prints nothing for zero rows, not [])
• a description with newlines and a | inside it does not shift a single column
• price_cents: null becomes 0 rather than crashing
• non-JSON output raises — it is not silently an empty queue
• a JSON object instead of a row list raises
• a row without an id raises instead of being dropped

The last three are the same rule stated three ways: could not read and read nothing are different verdicts, and collapsing them is how a queue looks empty while people are waiting in it. That is the same failure mode as an API that returns null for an unreachable upstream and gets printed as a confident zero.

Then the review actually ran

Two pending rows, both third-party listings. Both passed the admission criteria — an unpaid POST answered 402 with a usable accepts, public address, a description that is not a placeholder, no tunnel family, no duplicate. Approval wrote through the admin door and was verified by reading the row back rather than by trusting the HTTP 200:

Catalog count 1767 → 1769, and both ids answer a 402 through our door.

Two readings from that run are worth keeping:

• The delivery pre-flight said handler not found for both. That probe only knows first-party handlers; third-party listings are forwarded to the seller, so for these rows it measured nothing. "Not measured" is not "not deliverable", and the probe is diagnostic only — it does not move the verdict.
• One seller's host rotates its subdomain. The same family had an approved listing; checked directly, it was still healthy (checked 539/562 ok, served through the forward path). A changing subdomain is not by itself a dying endpoint, so the tunnel rule has to be judged by domain family and measurement, not by how unfamiliar the hostname looks.

What generalizes

An absence is only evidence if the instrument could have seen a presence.

Zero pending reviews meant "nobody submitted" only if the reader could read a row. Zero upstream balance meant "empty wallet" only if the RPC answer was a number. An empty search result means "no match" only if the query ran. Every one of these is a place where a broken instrument and a true zero produce the same bytes — and the true zero is the one that shows up first, which is exactly why it gets trusted.

The practical version: give the reader a control that feeds it a real, awkward row before you trust its count of nothing.

← all posts