tests/parallel/test_0760_swarm_cache_migration.py::test_global_evaluate_after_migration
hangs at np=4 on current development (8b7c8b9). It passes at np=2.
Reproduction
mpirun --timeout 90 -n 4 python -m pytest --with-mpi \
tests/parallel/test_0760_swarm_cache_migration.py::test_global_evaluate_after_migration -q
mpirun reaches its timeout with no test result. The other two tests in the
file pass at np=4:
| test |
np=2 |
np=4 |
test_swarm_cache_valid_after_migration |
pass |
pass |
test_global_evaluate_after_migration |
pass |
hang |
test_global_evaluate_displaced_nodes |
pass |
pass |
Not introduced by anything in flight — reproduced on development itself, and
the same run on bugfix/shrinking-working-set (#609) hangs identically, which is
how it was found.
Why nobody has seen it
scripts/test.sh runs exactly two globs under mpirun:
mpirun -n $PARALLEL_RANKS python -m pytest --with-mpi tests/parallel/test_075*py
mpirun -n $PARALLEL_RANKS python -m pytest --with-mpi tests/parallel/test_10*py
test_0760_... matches neither — test_075* stops at 0759 and test_10* starts
at 1000. The file has never been run in parallel by our own script, at any rank
count. PARALLEL_RANKS also defaults to 0, so ./uw test runs no parallel tests
unless --p is passed.
This is the #570 class exactly: a coverage gap in the glob, hiding a real defect.
The gap is the reason the defect is old and unnoticed rather than new.
Why it matters now
The hanging test is global_evaluate after a swarm migration, which is the
subsystem under active work: #551 (locator anti-scaling), #604 (a located point
returning NaN), #606 (rank-local answers in parallel), and PR #602, which adds a
regression test to this same file and records its validation as
mpirun -n 2 ... 4 passed per rank. At np=2 everything here passes, so that
validation would not have caught this.
Suggested handling
- Fix the globs so
tests/parallel/ is covered by pattern rather than by two
ranges that happen to have a hole between them — the hole will recur
otherwise.
- Run the parallel selection at np=4 as well as np=2. Every test that passes at
np=2 and hangs at np=4 in this subsystem has been a collective reached by some
ranks and not others; two ranks is often too few to expose it.
- Then diagnose the hang itself.
Underworld development team with AI support from Claude Code
tests/parallel/test_0760_swarm_cache_migration.py::test_global_evaluate_after_migrationhangs at np=4 on current
development(8b7c8b9). It passes at np=2.Reproduction
mpirunreaches its timeout with no test result. The other two tests in thefile pass at np=4:
test_swarm_cache_valid_after_migrationtest_global_evaluate_after_migrationtest_global_evaluate_displaced_nodesNot introduced by anything in flight — reproduced on
developmentitself, andthe same run on
bugfix/shrinking-working-set(#609) hangs identically, which ishow it was found.
Why nobody has seen it
scripts/test.shruns exactly two globs undermpirun:test_0760_...matches neither —test_075*stops at 0759 andtest_10*startsat 1000. The file has never been run in parallel by our own script, at any rank
count.
PARALLEL_RANKSalso defaults to 0, so./uw testruns no parallel testsunless
--pis passed.This is the #570 class exactly: a coverage gap in the glob, hiding a real defect.
The gap is the reason the defect is old and unnoticed rather than new.
Why it matters now
The hanging test is
global_evaluateafter a swarm migration, which is thesubsystem under active work: #551 (locator anti-scaling), #604 (a located point
returning NaN), #606 (rank-local answers in parallel), and PR #602, which adds a
regression test to this same file and records its validation as
mpirun -n 2 ... 4 passed per rank. At np=2 everything here passes, so thatvalidation would not have caught this.
Suggested handling
tests/parallel/is covered by pattern rather than by tworanges that happen to have a hole between them — the hole will recur
otherwise.
np=2 and hangs at np=4 in this subsystem has been a collective reached by some
ranks and not others; two ranks is often too few to expose it.
Underworld development team with AI support from Claude Code