Skip to content

test_global_evaluate_after_migration hangs at np=4, and its file is in neither parallel-test glob #611

Description

@lmoresi

tests/parallel/test_0760_swarm_cache_migration.py::test_global_evaluate_after_migration
hangs at np=4 on current development (8b7c8b9). It passes at np=2.

Reproduction

mpirun --timeout 90 -n 4 python -m pytest --with-mpi \
    tests/parallel/test_0760_swarm_cache_migration.py::test_global_evaluate_after_migration -q

mpirun reaches its timeout with no test result. The other two tests in the
file pass at np=4:

test np=2 np=4
test_swarm_cache_valid_after_migration pass pass
test_global_evaluate_after_migration pass hang
test_global_evaluate_displaced_nodes pass pass

Not introduced by anything in flight — reproduced on development itself, and
the same run on bugfix/shrinking-working-set (#609) hangs identically, which is
how it was found.

Why nobody has seen it

scripts/test.sh runs exactly two globs under mpirun:

mpirun -n $PARALLEL_RANKS python -m pytest --with-mpi tests/parallel/test_075*py
mpirun -n $PARALLEL_RANKS python -m pytest --with-mpi tests/parallel/test_10*py

test_0760_... matches neither — test_075* stops at 0759 and test_10* starts
at 1000. The file has never been run in parallel by our own script, at any rank
count. PARALLEL_RANKS also defaults to 0, so ./uw test runs no parallel tests
unless --p is passed.

This is the #570 class exactly: a coverage gap in the glob, hiding a real defect.
The gap is the reason the defect is old and unnoticed rather than new.

Why it matters now

The hanging test is global_evaluate after a swarm migration, which is the
subsystem under active work: #551 (locator anti-scaling), #604 (a located point
returning NaN), #606 (rank-local answers in parallel), and PR #602, which adds a
regression test to this same file and records its validation as
mpirun -n 2 ... 4 passed per rank. At np=2 everything here passes, so that
validation would not have caught this.

Suggested handling

  1. Fix the globs so tests/parallel/ is covered by pattern rather than by two
    ranges that happen to have a hole between them — the hole will recur
    otherwise.
  2. Run the parallel selection at np=4 as well as np=2. Every test that passes at
    np=2 and hangs at np=4 in this subsystem has been a collective reached by some
    ranks and not others; two ranks is often too few to expose it.
  3. Then diagnose the hang itself.

Underworld development team with AI support from Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions