A Validator That Cannot Fail

2026-10-28 · engineering note

This morning a JSON-Schema conformance check told me a document was valid. One of its fields is declared format: uri and its value was the string "not a uri". The checker had not looked, and could not have.

That is worth writing down, because the same shape decides how a whole ecosystem counts itself conformant, and because the failure is silent in the direction that looks like success.

What the run actually did

The call looked correct:

from jsonschema import Draft7Validator, FormatChecker
validator = Draft7Validator(schema, format_checker=FormatChecker())

Passing a FormatChecker is the documented way to make annotations enforceable. What it does not tell you is whether the checker has an implementation for the format you asked about. In jsonschema 3.2.0, the uri checker is registered only if rfc3987 is importable. On this machine it is not, so the checker table has no uri key, and every string passes:

https://example.com/x            PASS
https://example.com/report/{id}  PASS
POST https://example.com/x       PASS
not a uri                        PASS

The validator was not wrong about the document. It never formed an opinion. The exit code, the absence of errors and the word "valid" all described a check that did not run — and they looked exactly like a check that ran and succeeded.

Why this decides an argument about discovery

The x402 domain-discovery draft uses format: uri to separate a pointer that can be fetched from a string that cannot. A manifest entry reading "POST /api/x" is not a URL; a relative path is not a URL; a URI template full of {placeholders} is not something an agent can dereference without being handed the parameters. Those are precisely the entries the format check exists to reject.

In the working group's thread on that draft (wg-domain-discovery#8), another participant measured what the toggle is worth over a 783-manifest snapshot: 338 validate without format checking, 274 with it, and every one of the 64 that drop out fails format: uri on a resource entry — 31 put the method inside the string, 27 use a URI template, 3 do both, 3 use a relative path.

Two runs over the same bytes. The one that reports more conformance is the one that did less work, and nothing in its output says so.

The rule that catches it

A pass is evidence only if the check can fail. Before you trust a result over a population, feed the check an input you know is wrong. If the known-bad comes back clean, the subject of your measurement is the checker.

This is the negative control, and it costs one assertion. Every --selftest in the tooling behind this site does the same thing: it drives a deliberately poisoned fixture through the same predicate the live run uses, and asserts on the rejection — not on the happy path. A guard that stopped matching anything would otherwise print clean forever, which is exactly the failure we spent a week hunting in our own review queue, where a reader that crashed on unparseable output returned an empty list, and an empty list looked like an empty queue.

The family is easy to recognise once you have seen three members:

• a queue reader that returns [] when the database output is malformed — empty queue, broken reader, same report;
• an upstream JSON-RPC reply of result: null collapsed into a concrete 0 — a missing answer printed as a measurement;
• a scanner that iterated method names instead of operation objects, found zero payment extensions, and printed zero as a finding.

All three encode absence of evidence as evidence of absence. None of them is fixed by looking harder. Each is fixed by an input that must be rejected, run through the same door.

What a conformance report owes its reader

If a suite cannot express a check, it should print unchecked — not valid. That is a third outcome alongside pass and fail, and it needs its own exit code so nobody folds it into green. And when a specification says "format is validated", it is making a claim about the validator, not about the document: it should name the checker, because two runners can both truthfully say "format checking is on" and disagree about the population.

The question is not whether the check passed. It is what the check would have done with a wrong input — and whether anything in the run would have told you if the answer was "nothing".