An Empty Column Is Not an External Caller

2026-10-25 · data report

Every delivery through the gateway writes one row: which service, which path kind, who called it, what came back, and who was at fault when something failed. The table is the only place we can answer "do agents actually use this?" without counting HTTP requests, which — as we wrote earlier — are mostly crawlers.

This week I ran that table to find our most-used endpoints by real external callers. The first query returned a clean, plausible answer. It was wrong, and it was wrong in a way that is worth writing down because the mistake is structural, not careless.

The query and the answer it gave

SELECT service_id, COUNT(*) n, COUNT(DISTINCT agent_hash) agents
FROM delivery_outcomes
WHERE created_at >= datetime('now','-30 day')
  AND ip NOT IN ('…','…')   -- our own hosts, addresses redacted
GROUP BY service_id ORDER BY n DESC;

The top row came back as x402-gas-price: 170 successful deliveries, one distinct agent. Everything else was an order of magnitude smaller. That reads like a real customer with an integration — the kind of signal you build a roadmap on.

It was our own probe. NOT IN over a column that can be empty does not mean "everyone who is not us". It means "everyone who is not us, plus everyone we did not record". And we have exactly one write path that records nothing.

The one path that records no IP, on purpose

/api/gas-price is a free alias. It forwards to the paid service and answers without a wallet, a signature, or a trial. When it was added to the ledger, the reviewer made a deliberate call: do not store the IP.

This is the only completely anonymous write path in this table. Recording an IP here means building a permanent PII row for every crawler, and this path does not need to distinguish sources anyway — no wallet, no trial.

That is a defensible privacy decision, and I am not arguing with it. But look at what it does to the columns. In the whole table, x402-gas-price is the only service with rows whose ip is the empty string — 259 of its 260 rows. Every other service has an address on every row. So the single endpoint whose traffic we know least about is also the one that shows up first in a query written as "not us".

Same table, 30-day window, the rows the query counted as external:

169 rows. Not one of them is a customer. Add the one row that does carry an address and you have the complete population of "our most active external endpoint".

The corrected query is one clause longer, and the row disappears entirely:

WHERE ip != '' AND ip NOT IN (...)

That is the whole fix for reading it. The structural fix is different: if you have to drop the raw address for privacy, store HMAC(ip, secret) instead. It keeps the privacy property the reviewer wanted — you cannot recover an address from it — and it restores the one thing the empty column destroyed: the count of distinct sources. An anonymous path does not have to be an unmeasurable one.

The same shape, on the failure side

While I had the table open I looked at refusals. Seven days, 418 of them. Before reading the breakdown I would have said a third of calls being rejected is a caller-experience problem worth a week of work.

cause   kind    n
caller  trial   413
caller  firstfree 4
caller  paid      1

413 caller errors on the trial path, out of 1,348 deliveries. Then the error texts:

sidecar HTTP 400: {"ok":false,"error":"ext or filename required"…}  216
ext or filename required                                            162
sidecar HTTP 400: {"ok":false,"error":"this service filters by tag"…}  4

378 of 418 come from one service and one message, all from our own host, all carrying the reserved probe digest, in bursts of exactly twelve. Twelve is the row count our post-deploy trial guard writes per run. The refusal rate was not a measure of caller pain. It was a measure of how many times we deployed.

The lesson is not "exclude probes" — we do exclude them, in the metrics that matter. The lesson is that a rate is only meaningful if you know what is in its denominator, and the cheapest way to be wrong is to compute a ratio over a table that has more than one writer.

What did not need fixing

One thing I expected to be broken turned out to be correct. x402-mime-type carries 375 refused deliveries in seven days and still publishes reliability: 100, excellent. That is not a bug. A refusal is a caller error — missing parameter, wrong shape — booked as cause=caller, and seller reliability is computed from the caller's side of that line. A seller is not less reliable because someone called it wrong. Had we moved refusals into the seller's score, a single looping probe would have delisted a healthy service, and the service's owner would have had no way to find out why.

What changed

• The reading: every query over this table now filters ip != '' explicitly, and the empty value is reported as its own bucket instead of being folded into the complement.
• The recording: nothing yet. Storing HMAC(ip) on the anonymous path is a schema change to a table that several guards read, so it is written up rather than smuggled in during a post-mortem.
• The probes: a header change to make our own guard attributable was tested and reverted. The path does not read identity at all, so the header would have changed the request and not the row — a fix that looks like one.

Two columns that can be empty, in a table whose whole purpose is attribution. The empty value is not "unknown, please investigate" in an aggregation — it is a member of whichever set your predicate just defined. Anyone who publishes a usage or error number derived from a ledger should be able to say, in one sentence, what an empty column did to it.

Queries run against the live ledger on 2026-10-25; the handler rationale quoted above is from handler_discovery.go in the deployed gateway source.