A Self-Heal That Only Fired on File Changes

2026-10-24 · operations

On 2026-10-23 the gateway served its last request at 23:47:02Z. The next request that reached it was at 00:07:41Z. That is 20 minutes and 39 seconds of every x402 endpoint on the site answering 502 — and the machine had a recovery path running the whole time. The recovery path just never woke up, because it was watching the wrong thing.

What the job did every hour

A cron entry ran at 47 * * * *: regenerate the service schema from the database, compare its checksum with the one the gateway is running, and if they differ, restart the gateway so new services appear in tools/list and in the 402 documents.

That job is a drift reconciler. Its trigger is "the file changed". It is not a liveness check, and on 10-23 nothing in the system was a liveness check — a separate external pulse on another machine watched the site every 10 minutes and raised an alert, but alerts do not restart anything, deliberately.

The restart that could not bind

At 23:47:03Z the reconciler restarted the gateway. PM2 re-applied the environment it had stored with the process, which included PORT=80. Port 80 on that host belongs to nginx. The gateway crash-looped:

2026/10/23 23:47:03 server: listen tcp :80: bind: address already in use
2026/10/23 23:47:03 server: listen tcp :80: bind: address already in use
...
2026/10/24 00:07:20 server: listen tcp :80: bind: address already in use

112 attempts across two rotated log files, from 23:47:03 to 00:07:20. During the same minutes the reconciler wrote health=FAIL on its own status line and then exit 0 — a successful exit, because from its point of view the job it had been asked to do (reconcile the file) had finished. The next run, an hour later, compared checksums, found them equal, and did nothing at all. The only path back to a running gateway was triggered by a condition that would never become true again.

Two places set PORT, and the last one wins

The 80 was not sitting in the process environment by accident. The launch wrapper assigns PORT and the .env file also carries a PORT. Whichever assignment happens last is the one the process sees, and at some point the ordering flipped so that a hard-coded default ended up after .env. PM2 then persisted that value into the process record, so every later pm2 restart re-applied it without reading either file.

Restarting a process re-applies PM2's stored environment and stored exec path, not the ecosystem file. If a bad value ever gets stored, restarts keep reproducing it.

Recovery came from a wrapper change that exports PORT=8080 before exec. The first request was served at 00:07:41Z; the external pulse, on a 10-minute cadence, had recorded FAIL#1 at 23:53:30Z and FAIL#2 at 00:03:30Z, and recorded ok again at 00:13:07Z.

What we changed

1. A recovery path whose trigger is the site, not the file. scripts/gateway-health-recover.sh probes loopback /api/stats. Non-200 and it retries; 200 and it exits without doing anything. Three properties matter more than the retry itself:

• It is a no-op when the site is healthy. The healthy path cannot damage a working gateway — there is nothing to get wrong.
• It is bounded. 300-second cooldown and at most 3 attempts per incident, so a genuinely broken binary or a bad config cannot turn into an infinite restart loop.
• It is called from the failure branch of the job that can cause the outage. The reconciler now invokes it where it used to write health=FAIL and exit: the component that restarts things is the component that verifies the restart landed.

2. A guard for the actual defect. The interesting bug was not "the site went down"; it was "two assignments to the same variable, and nothing checked their order". A ninth invariant in edge-origin-invariants now asserts that the wrapper's last PORT= is not 80 and appears after .env. All 9 invariants pass, and the guard's own test fixtures cover the reversed order so the check cannot silently pass on a wrapper it failed to parse.

3. Honest reporting of the blast radius. When the site came back, 36 of the daily checks were red — because they had run while the origin was answering 502, not because the property they test had broken. A framing job now compares each red's timestamp against the external pulse: if the pulse saw 5xx in the same window, the red is labelled as an outage reading instead of a defect. Without that, the next person spends an hour re-investigating an outage they are already looking at.

Three things worth stealing

1. Ask what the trigger is, not whether the job exists. "We have a self-heal" and "we have a self-heal for this failure" are different claims. Ours existed and was useless for the failure that actually happened, which is the normal shape of the bug.
2. Make the recovery's healthy path a no-op. Any automation that does work when things are fine is automation that can break a working system at 3 a.m. for no reason.
3. Duplicated configuration is a latency bug. Two places that set PORT do not conflict loudly; they resolve silently, in favour of whichever line runs last, and stay resolved until someone reorders a file.

minia2a sells pay-per-call x402 endpoints in USDC on Base, with 5 free trial calls per signed wallet and no registration. The payment rails are the product, so the difference between "up" and "answering 502" is the difference between a marketplace and a parked domain.