Summary
ant-node's built-in self-updater (UpgradeConfig { check_interval_hours: 1 }) replaced our binary (0.14.2 → 0.14.3) and exited with code 0, but the resulting/replacement process failed at startup (Failed to create dual-stack network nodes: Failed to create transport — a separate transient issue, likely an IPv6-readiness race right after the binary swap). Because the exit code was 0, our Windows service manager (WinSW) — whose onfailure recovery only triggers on non-zero exit codes — treated it as an intentional, successful stop and took no recovery action. The node sat completely offline for 14 hours until a scheduled human check caught it.
Why this matters beyond our setup
Any process supervisor that distinguishes "crash" from "clean stop" by exit code (systemd's Restart=on-failure, WinSW's default onfailure, etc.) will have the exact same blind spot: a self-update that exits 0 but leaves no running node looks identical to an operator-requested shutdown. This is a structural gap in the upgrade flow, not just a Windows quirk — the transport-init race that tipped us over was Windows-specific, but the "no verification, no non-zero signal" behavior applies on every platform.
What we found
ant-node.exe --help documents --stop-on-upgrade: "Exit cleanly on upgrade instead of spawning a new process. Use when running under a service manager... that will restart the process automatically" — this is clearly the intended mechanism for our exact scenario, but:
- It's opt-in and not the default, so anyone running under a service manager without discovering this flag inherits the current risky behavior.
- Even with it set, the exit is still code 0 — so
onfailure-style recovery (the common case for Windows/systemd service wrappers) still won't fire. The service manager needs to be configured to restart on any exit, not just failures, which isn't the conventional default for service wrapper templates.
Suggested improvements (any one of these would have prevented our outage)
- Verify before exiting: before the old process exits (whether self-respawning or under
--stop-on-upgrade), confirm the new binary/process actually completed its transport/network initialization (e.g. wait for a readiness signal or health check) before relinquishing. If the new process fails to come up, the old process should stay alive and report the failure loudly instead of exiting 0.
- Non-zero exit on incomplete handoff: if
--stop-on-upgrade genuinely cannot verify the successor started, exit with a distinct non-zero code so standard onfailure/Restart=on-failure recovery paths trigger automatically — this alone would have restarted our node within 30 seconds instead of 14 hours.
- Document the Windows-service deployment pattern: the README/ops docs for WinSW/NSSM-style deployments should recommend
--stop-on-upgrade explicitly and note that onfailure won't catch exit 0 — point operators at configuring their service manager to restart on any exit, or ship a WinSW XML example that does so.
Our mitigations (for context, not asking you to fix these)
- Added
--stop-on-upgrade to our service arguments.
- Added an external watchdog that checks service status every 10 minutes and force-restarts on
Stopped, independent of exit code — this is now our actual safety net, since (1) and (2) above aren't guaranteed by the current binary behavior.
Happy to share our WinSW config and logs if useful for reproducing the transport-init race that triggered the initial failed handoff.
Summary
ant-node's built-in self-updater (UpgradeConfig { check_interval_hours: 1 }) replaced our binary (0.14.2 → 0.14.3) and exited with code 0, but the resulting/replacement process failed at startup (Failed to create dual-stack network nodes: Failed to create transport— a separate transient issue, likely an IPv6-readiness race right after the binary swap). Because the exit code was 0, our Windows service manager (WinSW) — whoseonfailurerecovery only triggers on non-zero exit codes — treated it as an intentional, successful stop and took no recovery action. The node sat completely offline for 14 hours until a scheduled human check caught it.Why this matters beyond our setup
Any process supervisor that distinguishes "crash" from "clean stop" by exit code (systemd's
Restart=on-failure, WinSW's defaultonfailure, etc.) will have the exact same blind spot: a self-update that exits 0 but leaves no running node looks identical to an operator-requested shutdown. This is a structural gap in the upgrade flow, not just a Windows quirk — the transport-init race that tipped us over was Windows-specific, but the "no verification, no non-zero signal" behavior applies on every platform.What we found
ant-node.exe --helpdocuments--stop-on-upgrade: "Exit cleanly on upgrade instead of spawning a new process. Use when running under a service manager... that will restart the process automatically" — this is clearly the intended mechanism for our exact scenario, but:onfailure-style recovery (the common case for Windows/systemd service wrappers) still won't fire. The service manager needs to be configured to restart on any exit, not just failures, which isn't the conventional default for service wrapper templates.Suggested improvements (any one of these would have prevented our outage)
--stop-on-upgrade), confirm the new binary/process actually completed its transport/network initialization (e.g. wait for a readiness signal or health check) before relinquishing. If the new process fails to come up, the old process should stay alive and report the failure loudly instead of exiting 0.--stop-on-upgradegenuinely cannot verify the successor started, exit with a distinct non-zero code so standardonfailure/Restart=on-failurerecovery paths trigger automatically — this alone would have restarted our node within 30 seconds instead of 14 hours.--stop-on-upgradeexplicitly and note thatonfailurewon't catch exit 0 — point operators at configuring their service manager to restart on any exit, or ship a WinSW XML example that does so.Our mitigations (for context, not asking you to fix these)
--stop-on-upgradeto our service arguments.Stopped, independent of exit code — this is now our actual safety net, since (1) and (2) above aren't guaranteed by the current binary behavior.Happy to share our WinSW config and logs if useful for reproducing the transport-init race that triggered the initial failed handoff.