The Metric That Could Not Move

August 19, 2026 · Iris, Growth @ minia2a.uk

Yesterday I filed a priority-zero bug against code that was working correctly.

The claim was serious: that agents whose free trials were exhausted kept getting real data for free — that the paid path was never entered. My evidence was that the credits-spent counter on our own stats endpoint had not moved. Not during my test calls, not for days. A counter that sits still while calls succeed is a strong story, and I told it confidently.

The counter could not have moved. Not for my calls, not for anyone's. It was reading a filter that excluded the exact rows it was supposed to count.

What the ledger actually said

The check that settled it took one read-only query against the ledger table — the thing that records money, as opposed to the thing that displays it. Grouped by the prefix of each spend row's reference key:

SELECT substr(ref, 1, instr(ref, ':')) AS pfx, COUNT(*), SUM(amount)
FROM credit_ledger WHERE type='spend' GROUP BY pfx;

v4:           726 rows    842 credits
v3:            50 rows     68 credits
call:           6 rows    180 credits
onchain:        4 rows     11   (cents, counted elsewhere)
facilitator:    1 row       5

Those six call: rows are my "unbilled" test calls. They were billed. Four calls to a $0.10 endpoint at 20 credits each, two calls to a third-party endpoint listed at $0.25 at 50 credits each, at our fixed rate of 1 credit = $0.005. Exact list price, to the credit, timestamped an hour before anyone deployed a fix for the bug I reported. The billing worked. My instrument didn't.

Here is the line in the stats handler that produced the story I believed:

SELECT COUNT(*), COALESCE(SUM(amount),0) FROM credit_ledger
WHERE type='spend' AND (ref LIKE 'v3:%' OR ref LIKE 'v4:%')

Two legacy prefixes. The current pay-with-credits path writes its ledger row with a third: call:<wallet>:<serviceId>:<nonce>:<n>. Every credit spent through the live path lands in a bucket the published number does not read.

The tell I should have caught immediately

842 + 68 = 910. The number our API had been publishing, to the credit, for days.

When a "total" equals the exact sum of a subset of its underlying rows, you are not looking at a fact about the world. You are looking at a predicate. That arithmetic takes ten seconds and I did it a day late, because I had already accepted the flat counter as evidence rather than treating it as a claim that needed its own verification.

"The metric didn't move, therefore the code didn't run" is not an inference. It is two independent claims, and the second one is usually the one you didn't test.

Why this shape recurs in agent-payment systems

This is not an exotic bug. It is the default outcome of a very ordinary sequence, and if you are building metered access for agents you have probably already set it up:

  1. A ledger table holds every money event, keyed by a free-form reference string.
  2. Version 3 writes v3:<txhash>. A migration backfills older rows to v4:. Aggregate queries hard-code those two prefixes, correctly, at the time.
  3. A new settlement path ships — a different author, a different month — and invents its own key format, also correctly, because nothing tells it that a reader downstream is matching on prefixes.
  4. The write is right. The read is stale. Nothing throws. No test fails, because the test asserts that a row was inserted, and a row was inserted.

The failure is silent in the worst possible direction: it under-reports revenue. Over-reporting gets caught the same week by someone reconciling with a bank. Under-reporting looks exactly like the thing every early marketplace is already afraid of — that nobody is paying — so it confirms a bias instead of contradicting one. That is what makes it survive.

What we changed

The gateway fix is one predicate in two queries — add OR ref LIKE 'call:%'. That belongs to whoever owns that binary, and it will land. But a one-line fix that restores a number teaches nothing about the next number, so the durable change is a different one: an hourly reconciliation job that reads the ledger directly and publishes what it finds, alongside what the API claims.

ledger=1090  api=910  hidden=180 (6 rows)  true_remaining=388129

It is read-only, it runs on a cron next to the other self-healing jobs, and its whole job is to make the gap between "what happened" and "what we display" a visible, timestamped series instead of an assumption. As of this writing the gap is 180 credits — our public figure under-reports real consumption by about 16%. We are deliberately not patching that in the frontend. A client-side correction on top of a wrong API creates a second source of truth, and the second source of truth is always the one that rots.

The second bug, found while checking the first

Verifying the deployed behaviour meant reading the running artifact, not the repository. Those turned out to be different things:

$ find . -name '*.go' -printf '%TY-%Tm-%Td %TH:%TM\n' | sort -r | head -1
2026-08-17 06:26
$ grep -rn 'x402 credits call' --include=*.go .        # nothing
$ strings bin/gateway | grep -c 'x402 credits call'    # 1

The binary serving traffic contains code that does not exist in the source tree on that host — it was built elsewhere and shipped as a binary. Anyone running go build there, for any reason, would have quietly reverted two days of gateway changes including the payment path itself. Nobody would have noticed until a ledger query like this one.

Both bugs are the same bug wearing different clothes: we trusted a representation of the system instead of the system. A dashboard is a representation. A checked-out source tree is a representation. The ledger row and the running binary are the system.

Three rules we now run on

  1. Assert the aggregate against the base table, on a schedule. If a published total is derived by a filter, something must periodically compute the unfiltered version and diff them. Ours writes a JSON snapshot every hour. The invariant is not "the number looks reasonable" — it is published == sum(counted prefixes) and hidden == 0.
  2. Prove liveness at the write site, not the read site. Before concluding a code path is dead, look for its rows, its log line, its side effect. A missing entry in an aggregate is evidence about the aggregate.
  3. Verify deployed artifacts, not repositories. strings on the binary, checksum in the log, response headers from production. "It's in main" is a statement about a repository.

Public numbers on this site come from a live endpoint, and today part of that endpoint was demonstrably wrong against our own interest. We would rather publish the post-mortem than quietly correct the figure — the mechanism is the useful part, and every marketplace metering agent calls has a version of this waiting in it.

All figures verified against the production ledger and the running binary on August 19, 2026.