Your 404s Have User-Agents

An endpoint that answers 402 Payment Required is up. It is doing exactly what it is supposed to do. But "up" is not a property of your endpoint — it is a property of the pair (your endpoint, the URL the caller built). If the caller joins two of your published fields in a way you did not intend, the request never reaches a route, your gateway answers 404, and some other system's dashboard records you as having been down.

We found one of those this week, and the only reason we found it is that our gateway logs carry the user-agent on every line.

What the log said

One third-party x402 health monitor has been walking our catalog since September 22. Over that window it sent us 4,067 requests. Two numbers matter:

Of the 771 failures, there were 156 distinct paths. 155 of them had the same shape:

/x402/<A>/<B>

And the shape is not random. <B> is always our service identifier — the same value in every one of the 155. <A> takes two forms, and both of them are URLs we publish:

One probe arrived with a query string attached — ?xml=&text=&input=, which are field names from that endpoint's own input schema. That is not someone guessing at paths. That is a client that believes the concatenated string is the endpoint, and is calling it.

The 156th path was a route we deliberately retired, so we will leave it out of the count.

The controls, and the one-line answer

Before believing any of that, we asked the endpoint. Three reads, no parameters, no state:

402  POST /x402/time
402  POST /x402/x402-time        # one doubled prefix: the gateway tolerates it
404  POST /x402/time/x402-time   # only the concatenation is unroutable

Both legitimate forms answer the challenge. Only the join fails. Whatever the caller did, the thing that broke was the join, not the endpoint.

Where the second segment comes from

Our 402 body carries a discovery extension. One of its fields is routeTemplate. We populate it with our service identifier, which is also, separately, the value the service's signing recipe uses:

"extensions": {
  "bazaar": {
    "discoverable": true,
    "routeTemplate": "x402-time",
    ...
  }
}

That identifier is what appears as <B> in all 155 paths. We are not going to claim we watched the client's source — we did not, and this is the one link in the chain we are marking as inference rather than measurement. What is measured is that the trailing segment of every failing path is a value we publish, and that appending it as a path segment to a URL we also publish produces something that cannot route.

Why would a field named for a route carry an identifier? Because that is what a publisher has in hand. An identifier is assigned once and is stable; a route template is a path shape, and for a service with exactly one static route there is nothing to template — the spec says so, and says the field should be absent in that case. We covered that half in the previous post: the official SDK's validator rejects a value without a leading slash, so a spec-compliant facilitator drops the field and falls back to the concrete URL.

This post is the other half. Dropping the field is what a validator does. Appending it is what a consumer that does not validate does. Neither outcome is what the publisher intended, and only one of them shows up in your logs.

Two ways we nearly reported this wrong

Both are worth keeping, because both are the same mistake in different clothes: reading a slice and calling it the population.

One: the six-second window. The first thing we saw was a burst — 153 distinct paths, all 404, inside six seconds. On that slice, the honest summary was "this monitor has never once reached us successfully." It is also false. Full rotated logs: 3,296 of its 4,067 requests answered 402. A single burst is a census of the burst.

Two: the shape test that was too generous. To count how often the doubled shape appears across every 404 we have ever logged — 1,384,805 lines — we first matched /x402/<A>/<B> loosely. That pattern also matches our own legitimate /x402/proxy/<id> alias namespace, and it inflated the count from 630 to 30,284. A shape predicate has to exclude the family that legitimately wears the same shape, or it is not a shape predicate.

Which gives the finding its scale, and its limit: this is one consumer, not a class of consumers. Everything else that walks /x402/<A>/<B> is hitting our documented alias route and succeeding. One monitor, 19% of its probes, is not a platform-wide outage. It is a single integration reading one of our fields differently than we assumed, and it would have stayed invisible if we had only ever read status codes.

Reproduce it

There is no exotic tooling here. Grep your access log for 404s, group by user-agent, and then group the paths by shape:

grep "| 404 |" gateway.log \
  | grep -oE '"(GET|POST) +"[^"]*"' \
  | sed -E 's/^"(GET|POST) +"//' \
  | sort | uniq -c | sort -rn | head -40

What you are looking for is not the noisy top of that list — scanners asking for /.env and /backups/ will always be there. It is a regular shape: the same number of path segments, the same repeated substring, arriving from one user-agent on a schedule. Regularity means a program built it from a rule, and a rule is something you can fix — on your side or on theirs.

The corollary is the uncomfortable part. A crawler that follows your published links will never produce this class of failure, because it asks for exactly the URLs you wrote down. The failures come from consumers that reconstruct your URLs from metadata — and those are precisely the clients you most want, since reconstruction is what a discovery layer is for. Treat any regular 404 shape as a message about your metadata, not as background noise.