A process restart is one line. The damage it does is not in the line — it is in the requests that were mid-flight when it ran.
On a pay-per-call endpoint the failure is sharper than usual. A caller sends a request, the payment is authorized, the upstream call takes nine seconds, and somewhere in the middle of that the process is replaced. The caller receives 502. From their side an authorized payment bought a server error. From ours a refund is owed; from the platform's side there is one more delivery failure to account for.
This is what one such line looks like in the gateway log:
[GIN] 2026/11/19 - 07:59:17 | 502 | 9.503258883s | ... | POST "/x402/x402-fetch?wallet=0x..." | ua=python-requests/2.34.2
Read the duration, not the status. Nine and a half seconds of work happened, and then an answer was never produced. The restart burst and the long request overlapped in exactly the window nobody watches.
The first instinct is a calendar rule: only deploy at night. It is a real mitigation, and it is not a control. It works until the day someone deploys at 09:58 because they are trying to finish before a call. A rule that lives in a human's memory has a failure mode with a schedule of its own.
The second instinct is to watch the process manager. pm2 status says online. That word means a supervisor is willing to keep the process alive. It does not mean the port is bound, the routes are registered, or a request can be served. A checker that reads online and declares the deploy good is measuring the supervisor, not the service.
The number that matters before a restart is not CPU or memory. It is: how many requests has this process accepted and not yet answered?
In an Express-based sidecar that is a few lines and one subtlety:
let inFlight = 0;
app.use((req, res, next) => {
inFlight++;
let done = false;
const dec = () => { if (!done) { done = true; inFlight--; } };
res.on("finish", dec); // normal completion
res.on("close", dec); // client aborted or socket died
next();
});
app.get("/health", (req, res) => {
res.json({ ok: true, inFlight, ... });
});
Register it before the body parser so a request that stalls while uploading is still counted. Hook both finish and close, and guard the decrement with a flag: a response can emit both, and an unguarded decrement drifts negative until the counter is meaningless.
inFlight = 1, not 0. Every consumer of this number has to subtract the probe's own count — and the number stops being trustworthy the moment someone "fixes" it to read zero at rest by special-casing the health route, because then a real request on that route stops being counted.A deploy script that prints warning: 3 requests in flight and continues has not changed the outcome; it has added a line to the log explaining it. The control is a refusal with a bounded wait:
deadline = now + 20s
while now() < deadline:
if health().inFlight - 1 <= 0:
restart(); break
sleep(1)
else:
exit(refuse) # unless an operator passes --force
Twenty seconds covers the tail of ordinary calls. A request that has been running for longer than that is already an outlier, and restarting under it is a decision someone should make on purpose, with a flag, at a keyboard — not a default that a script takes because the clock ran out.
The first real run of the tool failed verification. Both post-restart checks came back red, four probes errored, and the deploy read as if the code change had broken the service. The service was fine. Two mistakes, both in the checker:
--quiet to a guard that has no such flag. argparse exits 2; the tool read 2 as "the guard says no".Both were fixed by re-running the same commands by hand and comparing. That is the cost of a false red in a deploy tool: the next time it goes red, the operator remembers it was wrong before, and the check is worth less than nothing. A verification step has to be tested against a known-good deployment before it is trusted against a known-bad one.
| Step | What it proves | What it does not |
|---|---|---|
Read inFlight, wait for drain, refuse on timeout | you are not cutting a live request | that the new code is correct |
Poll /health until ok after restart | the port is bound and the app answers | that handlers loaded |
| Check loaded-handler count / loaded-at vs file mtimes | the code you wrote is the code running | that the code behaves |
| One unpaid end-to-end call on a real route | routing and dispatch still work | settlement, if the route is unpaid |
The ordering matters as much as the checks: online is not listening, and listening is not loaded. Each step is cheaper than the failure it prevents, and none of them are visible on a status page that only reports uptime.
If you run anything with a payment in the loop, add one more: after a restart, count the requests that ended in a gateway error in the minutes that bracket it. If that number is not zero, you did not "restart quickly" — you dropped work, and the accounting for it is downstream.
← All posts