Diagnose a hung parallel job: per-rank watchdog and a roll-call report

[lmoresi]

What this is for

A collective entered by only some ranks leaves N-1 ranks stopped in the same
frame and one somewhere else. The signature exists only across the set, so no
rank can see it, and the job sits there until the queue kills it with nothing in
the output naming the rank that went the other way.

#628 catches the cases a static scan can see. It cannot see a collective reached
three frames down through a helper, which is most of them. This catches those,
because it does not reason about the call graph at all --- it observes the
symptom.

Using it

UW_HANG_WATCHDOG=120 mpirun -n 4 python myrun.py
python -m underworld3.utilities.hang_report uw-hang-dumps

ranks [0, 2, 3]  stopped at  discretisation_mesh.py:6954 in get_max_radius
        via  discretisation_mesh.py:6140 in _classify_points_in_domain
        via  swarm.py:3786 in migrate
rank  [1]        stopped at  swarm.py:3801 in migrate

=> 3 of 4 ranks are waiting together at discretisation_mesh.py:6954, and ranks [1]
   are somewhere else. That frame is where the collective is; ranks [1] are
   where the bug is.

The last line is the point. The majority position is the crime scene, not the
crime: those ranks did nothing wrong, they arrived at a collective and waited.
The rank that is not with them is the one that took the branch.

Arming comes from the environment rather than a call in the script, so it is in
force during import and mesh construction. A rank that diverges or dies before
reaching a watch() call reports nothing, and "before the script got going"
covers a lot of ground.

Pieces

• uw.mpi.watch / unwatch / watching / checkpoint --- the per-rank
  watchdog. Nothing collective: by the time anything is stuck the ranks have
  split, and a probe needing all of them either blocks alongside the others or
  becomes one more collective for the divergent rank to miss.
• uw.mpi.ranks_agree --- the positive audit. Compares a label, so it also
  catches ranks that arrive together by different routes, and prints which ranks
  took which path. Useful for bisection, because ranks hang at the first
  collective after the branch that split them, which is routinely unrelated
  code.
• underworld3.utilities.hang_report --- the roll call.
• @collective_operation now re-arms the watchdog and labels the report, so the
  14 declared collectives check in without anything being placed by hand.

checkpoint is a local timer reset: no communication, one test when disarmed,
safe to leave in production code.

Three measurements that shaped it

A Python thread cannot report a rank blocked in MPI. At np=4 against a 4 s
block in comm.allreduce, a re-arming 0.5 s threading.Timer fired zero
times on the blocked ranks; faulthandler.dump_traceback_later produced all
seven dumps. The interpreter lock is held for the duration, so the ranks that
most need to report are exactly the ones that cannot. Reporting has to be in C,
and the consequence is that the destination must be a file descriptor --- an
earlier draft used a StringIO, passed every serial test, and would have
reported nothing in the real case. watch() now refuses a buffer, and the test
asserting that refusal is what keeps the door shut.

The main thread is identified by its outermost frame, not its depth. A rank
blocked two frames into allreduce, with a telemetry thread twelve frames deep
inside requests, was reported as being in the HTTP call.

A rank is placed by the modal position over its recent dumps, not its last
one. A stuck rank shows the same position every period; the final sample can
catch it mid-transition. That had put one of four waiting ranks in a group of its
own, so the report named two culprits instead of one.

Testing

By subprocess, not in-process. dump_traceback_later is a single global slot
that pytest's own plugin already owns, and a rank inside a test framework is one
process looking at itself --- the one thing this analysis cannot be done from.

• test_0053 --- watchdog behaviour, 11 tests, real files throughout.
• test_0054 --- the parser and roll call on synthetic dumps, plus an
  end-to-end run: a real four-rank job with a real rank-local guard, which
  really hangs, is killed by mpirun, and whose dumps must name rank 1. An
  earlier version had the divergent rank join late so the job exited cleanly ---
  that tested a situation nobody is ever in, and made termination the flaky part.

23 passed locally alongside test_0052 from #628.

Limits

• The watchdog is a post-mortem. It makes a silent hang legible; it does not
  prevent one, and it needs a rank count that exposes the divergence.
• The checkpoint label is printed by the secondary Python timer, so it is
  absent on ranks blocked inside MPI --- those give the stack only. The stack is
  the localisation; the label is a convenience.
• ranks_agree is itself collective and can hang if a rank never arrives. That
  case belongs to the watchdog; the two are meant to run together.
• The dump names the calling line, not the C callee: a rank in
  comm.allreduce shows the Python line that made the call.

Underworld development team with AI support from Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lmoresi]

Adversarial review

The weakest claim is coverage, and it is weak by construction. The watchdog
reports where a rank is; it does not know what is collective and never checks.
That is the source of its reach — it sees collectives no static scan can find,
because it reasons about nothing — and equally the reason it proves nothing. A
clean report means no rank stopped for seconds, not that the run is correct.
Read it as a diagnostic, never as a gate.

It needs the hang to actually happen, at a rank count that exposes it. np=2
remains a special case: a mismatched pair often still meets. Everything here is
downstream of a divergence that a two-rank run may never produce.

The label is missing exactly where the reader most wants it. It is printed
by the secondary Python timer, which we measured does not run when the lock is
held — so a rank blocked in MPI gives the stack and nothing else. The stack does
localise, and the frames above it give the call chain, but "rank 3 last saw
step 12" is the line a human reads first and it is absent in the commonest
case. A fix means the label reaching C, which means writing it somewhere
faulthandler can reach; we did not attempt it.

ACCEPTED-style holes now exist in two places. _MACHINERY decides which
threads are the model, by substring. A hang inside a library whose path happens
to contain threading.py would be classed as machinery and dropped. The list is
short and specific, and the outermost-frame rule makes it much less load-bearing
than the depth rule it replaced, but it is a heuristic.

Grouping is by one frame. Two ranks stopped at the same line via different
call paths group together. For the shape this exists to find that is right —
they are in the same collective — but it will merge genuinely different
situations that share a line, and the printed call chain is the only thing that
reveals it.

The environment variable is process-wide and silent when malformed in one
direction only: a non-numeric value prints a warning and leaves the watchdog
disarmed. That is the safe failure, but it is a warning on stderr in a parallel
job, which is close to invisible. A run that thinks it is watched and is not is
the worst state to be in.

What we did not build. No uw subcommand — it is python -m underworld3.utilities.hang_report. No automatic collection of dumps from a
batch job's scratch directory. No integration with the static scan from #628,
though the natural pairing is obvious: the analyser names a frame, and the scan
could say whether that frame is one it already knows is guarded.

Controls. Both directions throughout: the watchdog is asserted to fire when
progress stops and to stay silent when it does not; the buffer refusal has its
own test because the buffer-backed draft passed everything else; the
outermost-frame rule and the modal-position rule each carry a regression test
naming the measurement that produced them. The end-to-end test uses a job that
really hangs, because a job that recovers is not the situation.