Skip to content

e2e-real verify: doctor's MSR block intermittently doesn't render, failing both #66 assertions on a healthy rig #367

Description

@VijitSingh97

Hit during the v1.15.1 release gate on miner-3 (2026-08-15). Filing rather than re-running to green, per the
#831 practice of fixing a flake at the harness.

What happened

e2e-real.sh verify failed with:

✗ doctor didn't confirm the MSR mod applied (#66)
✗ doctor's rdmsr register verification did not pass (#66) — see the doctor output above
E2E-REAL (verify): FAIL — 2 failed, 36 passed.

An immediate re-run of the same phase, same rig, same commit, no intervention: PASS — 38 check(s) passed,
with both MSR lines present. So it is intermittent, not a real MSR fault — the rig was healthy throughout
(doctor run by hand reports MSR mod applied — XMRig set the 'ryzen_17h' preset and
MSR registers verified via rdmsr (4/4 match)).

What the evidence shows

In the failing run doctor's output goes straight from msr kernel module loaded to CPU governor — the
entire MSR block is absent, not failing:

✓ kernel lockdown: none — MSR writes are permitted
✓ msr kernel module loaded
✓ CPU governor = performance          <- the two MSR lines should be between these

That block is guarded by [ -n "$log_file" ] && [ -f "$log_file" ] (rigforge.sh, in doctor, around the
"Resolve the worker's xmrig.log once" comment), where log_file="$wr/xmrig.log" and
wr=$(_worker_root_from_config). So on the failing run that guard was false — either wr came back empty
or the log path didn't resolve. _worker_root_from_config returns empty when $CONFIG_JSON is missing.

Ruled out:

I could not pin the trigger before the state was gone, hence this issue rather than a speculative patch.

Why it matters

Both assertions are bad (hard-fail) and they abort the release gate, so a transient makes the gate lie
about a healthy rig — and the natural reaction is to re-run until green, which is exactly how a real MSR
regression would get waved through.

Suggested direction

  • Have the verify phase assert the guard's inputs first (resolve + report the log path) so the failure
    says "couldn't resolve xmrig.log" instead of "MSR mod not applied" — a skipped block and a failed check
    should not look identical.
  • Consider making doctor distinguish "MSR unverifiable (no log)" from "MSR not applied"; today the e2e
    can only grep for the success string, so absence and failure are the same signal.

Incidental

xmrig.log on miner-3 is 122 MB (with a 121 MB .1 beside it) and _msr_log_status awk-scans the whole
file on every doctor. minsize 50M + daily rotation is letting it run well past the threshold. Tailing the
scan (the MSR line is written at miner start, and only the last one matters) would make doctor cheaper on
every invocation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions