Skip to content

fix(api): an unreachable worker API must not abort the run, or cry abort while succeeding (#364) - #366

Merged
VijitSingh97 merged 1 commit into
developfrom
fix/status-err-trap-noise
Aug 15, 2026
Merged

fix(api): an unreachable worker API must not abort the run, or cry abort while succeeding (#364)#366
VijitSingh97 merged 1 commit into
developfrom
fix/status-err-trap-noise

Conversation

@VijitSingh97

Copy link
Copy Markdown
Contributor

Closes #364.

Not just noise — it aborted runs on the rigs

#364 is filed as spurious stderr on an exit-0 path, and that is real. But tracing it turned up a
worse sibling on the same root cause, and that one bites on hardware rather than a dev laptop:
an unreachable worker API aborted tune/autotune's sampling loop outright. A miner restarting
under a sweep took the sweep with it. Please don't read this as cosmetic.

Root cause

The API readers had a propagate mode that let curl's exit escape, so an unreachable miner would
"surface upstream". Nothing upstream ever read that status — every caller branches on an empty
body. Letting it escape then broke two ways depending on the bash running the script:

bash 5.2 (rigs, appliance) bash 3.2 (macOS)
Caller not guarded (tune/autotune sampling) aborts, exit 7 aborts, exit 7
Caller guarded with ( ... ) || true (status) quiet prints the abort banner twice

set -E inherits the ERR trap into the $( ) each caller reads through. Bash 3.2 does not carry
the caller's suppressed-errexit context into that child, so the trap fires there once per frame the
failure unwinds through — hence two lines from status while it still printed its correct fallback
and exited 0. 5.2 does carry it, which is why this showed up on dev machines and not on the rigs.

The trap fires once per stack frame, not twice for one command: once for the failing curl inside
_read_api_summary, then again as _read_api_summary returns 7 (curl is its last command) one frame
up in _status_api_summary.

Fix

propagate is gone. Both readers always return 0 with an empty body when the API is unreachable —
the contract every caller already assumed — so neither failure shape is reachable. This fixes
status, tune, autotune and the control channel's post-apply liveness wait in one place instead
of copying #210's guard to each call site; the watchdog's || true is removed because the reader now
owns the guarantee.

Two alternatives considered and rejected:

Verification

Old-vs-new, API refusing connections, ERR trap live:

miner-0 (bash 5.2) HP ProBook 650 G5 appliance (5.2.37) macOS (3.2)
before — unguarded read died, 4 abort lines died, 4 abort lines died, 4 abort lines
after survived, loop completed, 0 survived, loop completed, 0 survived, 0
beforestatus verb 0 (guard holds) 0 (guard holds) 2 lines (as reported)

Note for the reviewer

The tune-side test is driven as a separate bash process deliberately. Run inside a subshell it
inherits the suite's own errexit context, and since 5.2 and 3.2 disagree about whether that context
crosses into a $( ), it silently disarms the failure under test on one platform or the other. Two
earlier drafts of this test passed against the broken script for exactly that reason. Please don't
fold it back into a subshell.

…ort while succeeding (#364)

The API readers had a "propagate" mode that let curl's exit escape, so an
unreachable miner would "surface upstream". Nothing upstream ever read it —
every caller branches on an empty body — and letting it escape broke two ways
depending on the bash running the script:

  * Any caller that was NOT guarded aborted outright. Measured on miner-0 and
    the HP appliance (Linux, bash 5.2): with the API refusing connections,
    tune/autotune's sampling loop and a bare _status_api_summary both died with
    "[ERROR] rigforge aborted ... (exit 7)". An API that went away mid-sweep —
    the miner restarting under you — took the sweep with it. This is the failure
    #210 first hit on miner-0 and papered over with a `|| true` at ONE call site.

  * On bash 3.2 (macOS) even GUARDED callers printed the banner. `status` wraps
    its read in `( ... ) || true` and still emitted two "[ERROR] rigforge aborted
    while running 'status'" lines before printing its correct "worker API not
    reachable" line and exiting 0 — the shape reported in #364. set -E inherits
    the ERR trap into the $( ) the caller reads through, and 3.2 does not carry
    the caller's suppressed-errexit context into that child, so the trap fires
    there once per frame the failure unwinds through. 5.2 does carry it and stays
    quiet, which is why this showed on dev machines and not on the rigs.

The mode is gone. Both readers now always return 0 with an empty body when the
API is unreachable — the contract every caller already assumed — so neither
failure shape is reachable, and #210's guard-inside-the-$( ) idiom is no longer
needed for these readers. The watchdog's `|| true` goes with it: the reader owns
the guarantee now (verified unguarded on both rigs).

`set +e` inside svc_status's subshell was considered and rejected: bash's ERR
trap fires independently of errexit, so it silences nothing.

Coverage: 7 assertions, each verified to FAIL on the pre-fix script. The
tune-side test is driven as a separate bash process on purpose — run in a
subshell it inherits the suite's own errexit context, which disarms the failure
under test on one bash or the other and passes vacuously.

Closes #364.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@VijitSingh97
VijitSingh97 force-pushed the fix/status-err-trap-noise branch from f6de4f7 to 07e9d7e Compare August 15, 2026 22:36
@VijitSingh97
VijitSingh97 merged commit 44ca68d into develop Aug 15, 2026
9 checks passed
@VijitSingh97
VijitSingh97 deleted the fix/status-err-trap-noise branch August 15, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

status prints a spurious '[ERROR] rigforge aborted' to stderr when the worker API is unreachable (exit code is still 0)

1 participant