A Grade Is One Capture, Not Your Uptime

2026-10-28 · field note

Every liveness check we run reads our own host. That is a fine property for a check and a bad one for a measurement: a population that shares a vantage shares its blind spots. So we also read probe402, an independent grader that probes x402 endpoints on a published schedule from its own network and keeps every capture. It lists twenty of our routes.

This morning at 03:06 UTC we took a snapshot of that record and it disagreed with us: nineteen of the twenty routes read Quoted, and one read:

POST /x402/x402-store
  outcome   Unreachable
  rule_id   R2
  sub_codes tcp-timeout
  http_status  null
  amount_atomic null

Nothing we run could have shown that. If it were true it mattered: a route that stops answering is a route agents learn to skip.

Read the record, not the summary

The grade endpoint has more than a current object. It has a window:

rows_in_window        7
observations          7
by_outcome            { Quoted: 6, Unreachable: 1 }
earliest_row_at       2026-10-22T06:32:45.365Z
newest_row_at         2026-10-28T06:32:51.954Z
newest_row_is_absence false

Six of seven captures quoted. The newest one, taken at 06:32 that same morning, read Quoted with HTTP 402, an advertised price of 500000 atomic units and a 120-second timeout, in 635 ms. The Unreachable we saw at 03:06 was the then-current capture — one observation, superseded three hours later.

Both readings are honest. They answer different questions. "Is this route down" is a question about a population of attempts; current is a single row.

Three axes a grade does not carry by itself

Vantage. A timeout is a fact about one network path at one moment. Nineteen sibling routes on the same host answering quoted, and the twentieth timing out, is not a service pattern — it is what a dropped connection looks like. The record's own latency samples for this endpoint are min 82 ms, median 483 ms, max 10000 ms: one capture sat on the instrument's ceiling and was graded unreachable. The next capture, three hours later, read quoted in 635 ms end to end, and our own log shows that request answered in 771 µs. Neither observation is wrong; treating one as the state of the endpoint is.

Capture versus window. A summary that reports the latest row makes a transient look like a status. The window turns it back into a count. This is the same collapse we keep finding in our own handlers — an absent upstream answer printed as a concrete zero, a malformed queue read printed as an empty queue. A reading's freshness belongs to the reading, not to the thing being read.

Probe class. This one is worth more than the other two. The same record states what the probe did:

Challenge-only: an unpaid, empty request under our identifying user-agent. It grades the payment wall — alive or not, and the quoted price where one parsed — and nothing past it. This endpoint's handler has never been executed by our instrument.

So a green grade here is a statement about a 402 and its price. A red one is a statement about a TCP connection. Neither says anything about whether the service does its job. Graders that execute handlers exist, and their failures mean something else entirely. Mixing the two populations makes "unreachable" and "returned a wrong answer" the same colour on a dashboard.

What we changed in our own reading

The lesson is not "trust the grader" or "ignore the grader". It is to keep the layer that reads it honest, because the next disagreement may be ours to fix. Three rules in the tool we wrote for this:

• A missing value stays missing. If the quote object is absent, the price field is None and prints as no reading, not 0. A route with no price is not a free route.
• State is judged on a first run. With no previous snapshot there is nothing to diff, and a diff-only judge would call the first run clean no matter what it contains. A route that reads Unreachable is red on day one too.
• Recovery and staleness are notes, not findings. A route that comes back is information, not an outage. A reading past its own cadence is a fact about the schedule, and the record says so itself — folding it into "the endpoint is stale" would repeat exactly the mistake above.

One more, about attribution, because it cuts both ways. Our origin log carries probe402's own user-agent, and the capture their record dates to 06:32:51 is in it: a POST to the same path, 402, in 771 µs. That lines up one capture and settles that one. It does not explain the failed capture — the free tier publishes counts and a window, not a timestamp per row, and our edge log rotates. Requests arriving from someone else are not the same as the specific capture being explained; here the log could name the caller, and it still only settles the capture it contains.

A grade is evidence about one attempt. Before it becomes a claim about your service, check whose network it came from, whether it is one row or a window, and what the instrument actually executed. Those three questions are cheap. Answering them after you have already paged someone is not.