Blame the right rank, rather than require every rank to have dumped - #634
Merged
Conversation
test_0054's end-to-end check asserted the waiting group was exactly [0, 2, 3]. On an oversubscribed CI runner a rank can be scheduled too little to dump inside the window, and the group came back as [0] -- a true report of a slower machine, not a defect. It went red on development after #630 merged. What the tool has to get right is which rank is BLAMED, so that is what is asserted now: the majority group sits at the collective, rank 1 is not in it, the group contains only ranks that entered it, and rank 1 is named as the odd one out. Requiring a particular number of witnesses tested the scheduler. Underworld development team with AI support from Claude Code
At a 1.0 s watchdog and a 25 s window the end-to-end check passed here and failed on CI with the majority located in `importlib._bootstrap`: four oversubscribed ranks take longer to `import underworld3` than the watchdog allowed, so the dump file filled with import frames and the job was killed before the collective produced enough dumps to outvote them. The timings were tuned to a workstation. 5 s watchdog, 75 s window. The assertion also now names the frame the ranks were actually found in -- "not at the collective" without saying where cost a CI round trip to diagnose. Underworld development team with AI support from Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_0054's end-to-end check asserted the waiting group was exactly[0, 2, 3]. On an oversubscribed CI runner a rank can be scheduled too little to dump inside the window, and the group came back as[0]— a true report of a slower machine, not a defect. It went red ondevelopmentafter #630 merged.What the tool has to get right is which rank is blamed, so that is what is asserted now: the majority group sits at the collective, rank 1 is not in it, the group contains only ranks that entered it, and rank 1 is named as the odd one out. Requiring a particular number of witnesses tested the scheduler.
7 passed locally.
Underworld development team with AI support from Claude Code