Give fadvi a gpu label and bump scalex to midcpu - #108
Merged
Conversation
Open
13 tasks
rcannood
force-pushed
the
fix-resource-labels
branch
from
August 21, 2026 07:16
d507258 to
ca73f2d
Compare
rcannood
force-pushed
the
fix-resource-labels
branch
from
August 21, 2026 07:17
ca73f2d to
1ac7c78
Compare
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Two resource problems from the last full run, where 30 tasks were killed at exactly their walltime.
fadviwas never on a GPU. Its labels were[midtime,midmem,midcpu]with nogpu, so SLURM put it on the CPU partition with no--gpus, the log saidGPU available: False, used: False, and it hit the 4 hmidtimeceiling on all six datasets. It now uses[hightime, midmem, midcpu, gpu], matchingdrvi,scvi,scanvi,cellplmandcondo.Four methods want more than 8 h, and do not get it.
uce,mnnpy,batchelor_mnn_correctandscalexall completed ondkd-- in 5 h 05, 23 min, 4 h 57 and 28 min respectively -- and were killed at 7 h 59 on every larger dataset. They stay onhightime: methods never getveryhightime, so every method gets the same 8 h ceiling (fair comparison) and no job ties up a node for a day. Timing out on the larger datasets is a scaling limit of those methods and is recorded as such in the results.scalexdoes go fromlowcputomidcpu, since it is CPU-bound by circumstance rather than by design (see below).Two things deliberately not done here:
scalexpassesgpu=0toscalex.SCALEX()but its engine isopenproblems/base_python:1, which has no CUDA, so it can only ever run on the CPU. Moving it to the pytorch base image is the real fix, but that is blocked on the GPU driver on the de.NBI workers being older than the CUDA the current pytorch base image needs.scimilarityhas the same shape asfadvidid -- pytorch base image, no gpu label -- but it completes fine on the CPU, so it is left alone.Checklist before requesting a review
I have performed a self-review of my code
Check the correct box. Does this PR contain:
Proposed changes are described in the CHANGELOG.md
CI Tests succeed and look good!