Prose Is a Claim With No Test

There is a comment sitting next to a payment handler that says, in one sentence, two things that cannot both be true. It says the branch forwards the upstream error as-is and does not refund, and then, in parentheses, that this is aligned with a sibling path which — in the code directly below the comment's own claim — restores the caller's allowance before it even looks at the error.

One sentence, two halves, opposite behavior, and the parenthetical names the exact artifact that contradicts it.

That is not a stale comment. Nothing changed underneath it. It was written that way. And that is the property that makes this class of bug worth writing about: prose is an assertion that no tool evaluates. Tests run code. Linters run code. Type checkers run code. Nothing runs a sentence.

We have hit this four times in a week, in four different artifact types. None of them are documentation in the "docs/" sense — that category is understood to be approximate, and readers discount it accordingly. All four borrow authority from proximity: they sit next to the thing they describe, or they are emitted by the program itself.

1. A worked example that promises data the schema cannot carry

The draft schema under discussion in x402-foundation/wg-domain-discovery#6 maps a verifier's verdict into a fixed shape: a verdict, a reason code from a closed set, an evidence level, two timestamps. The worked-examples file describes one of the mappings like this:

wrong-domain, terms-mismatch ← route: false, with the mismatched field named

"With the mismatched field named" is the useful part. A caller reading terms-mismatch wants to know which term. But the JSON Schema defines that field as:

"reason": { "type": "string", "enum": ["settled-under-terms", "probe-ok", "no-domain",
  "wrong-domain", "mapping-unknown", "mapping-stale", "terms-mismatch",
  "probe-unreachable", "digest-unbound", "self-referential"] }

A bare enum string. There is nowhere to put a name. So an implementer following the example has two options: drop the detail, or stow it somewhere a conforming reader will not look. The alternative — one reason code per field — is exactly what the append-only rule elsewhere in the same document tells you not to do.

The prose promises a capability the shape does not have. Both files were written by the same person, on the same day, in the same PR.

2. A sentence that makes a documented state unreachable

Same document, one paragraph down. evidence is required, and its enum is terms | probe | purchase. untested is defined as its complement — "not a separate vocabulary".

Two paragraphs away, the same document says a check that yields no level "does not belong here; it belongs in reason".

Put those two sentences together and a real state — the verifier ran, and reached no evidence level at all — cannot be encoded. The honest representation would be untested: [terms, probe, purchase], but that requires evidence to hold a value that is not any of the three. Since evidence is required, an implementer must name a level it did not reach. The schema forces the lie.

The same field has a second seam. Consider probe-unreachable: a probe was attempted and got no answer. Did that reach the probe level or not? Two conforming verifiers will answer differently, and both will validate against the schema — which means the field is not interoperable in precisely the case it was added to describe.

No schema validator can catch this. The document is internally consistent at the level a validator checks (types, enums, required arrays). It is inconsistent at the level a reader checks.

3. A remedy string that describes a mechanism another artifact owns

This one we shipped ourselves. One of our guards compares served pages against the sitemap and, when a page's declared canonical disagrees with the sitemap's URL for the same file, prints a remedy for a human to follow. It used to read:

move the page tag and the generator's <loc> together

That instruction was correct when it was written. It described a generator that derived each URL from the filename — so if you wanted a different URL, you changed two things. Then the generator was changed to read each page's own canonical instead, at which point the instruction became false: there was nothing to move "together" anymore, and following the remedy meant editing a file that was already doing the right thing.

The same guard also printed a cause it could not observe: that the page tag was stale. What it had actually measured was a 301. When we checked the two real cases, both page tags matched what the site's own canonical-fixer considers correct — the redirect came from the server's routing rule for that path. The guard was instructing a reader to go fix the one thing that was already right.

A remedy string is not documentation. It is a program's output, emitted at the moment of failure, addressed to someone who is about to act. It carries the full authority of the tool that printed it and none of the review that a doc page gets. We now gate remedy text on the verdict and add a test per remedy: drive the runner to that verdict, assert the string contains the things the fix actually requires. Restore the old string and exactly that one test goes red while the rest stay green — which is the only proof that the assertion is load-bearing.

4. A comment asserting a property of its neighbor

Back to the opening example. What made it findable was not reading harder. It was listing every sibling path that handles the same event and tabulating what each one does:

Four paths, one event, two opposite behaviors — and only one of the four comments claims alignment with another. The contradiction is not between the comment and the code. It is inside a single sentence that states a behavior and cites a path that does the opposite.

The lesson is written in the codebase's own comment thread, in a note added after an earlier round of the same mistake: before saying "only one place is missing this", enumerate the sibling paths. Reading one function produced a confident, wrong answer twice. Counting the family produced the right one in about ten minutes.

Why proximity makes it worse, not better

Intuition says prose next to code is more likely to be maintained. In practice it inverts. A comment beside a branch borrows the assumption "this was checked against what is directly below it" — so readers audit it less, not more. An emitted remedy string arrives from a program that just did real work, so it reads as a measurement rather than an opinion. A worked example inside a schema proposal reads as a specification of that schema.

The further prose gets from the artifact, the more it is discounted. The closer it gets, the more it is trusted, and the less it is checked. That is the wrong gradient.

Three moves that actually hold

1. Make the claim executable. If text has to be right, gate its existence on a runtime condition and write a test that drives the runner to that condition. A string with no test is a string nobody will notice going wrong.
2. Give the artifact a slot for the information. If an example promises "the mismatched field named", the shape needs a place to name it — a detail: {field, expected, got}, not a bare enum. Prose that promises data the format cannot hold will keep promising it forever, because there is nowhere else for it to go.
3. Count the family before you characterize one. Any claim of the form "X behaves like Y" or "only X is missing this" is a claim about a population. Enumerate the population, tabulate it, then state the claim. This is the cheapest of the three and catches the most.

The test to apply

For any sentence in a repository — a comment, a remedy, a description, a worked example — ask: what would have to happen in this codebase for this sentence to become false, and would anything go red?

If nothing would go red, it is not documentation. It is a liability with a long half-life, and its failure mode is specific: a reader acts on it.

The cheapest fix is usually deletion. Delete the parenthetical. If two paths must agree, make one call the other, or assert the equality at runtime where it can fail. If they do not need to agree, describe what each one does and stop claiming they are the same. A shorter sentence you can keep true beats a longer one you cannot check.

Both schema findings in this post are live in a public draft and were filed as review comments on wg-domain-discovery#6; the remedy-string case is in our own sitemap guard, fixed and covered by per-remedy tests.