Skip to content

Give fadvi a gpu label and bump scalex to midcpu - #108

Merged
rcannood merged 3 commits into
mainfrom
fix-resource-labels
Aug 21, 2026
Merged

Give fadvi a gpu label and bump scalex to midcpu#108
rcannood merged 3 commits into
mainfrom
fix-resource-labels

Conversation

@rcannood

@rcannood rcannood commented Aug 21, 2026

Copy link
Copy Markdown
Member

Describe your changes

Two resource problems from the last full run, where 30 tasks were killed at exactly their walltime.

fadvi was never on a GPU. Its labels were [midtime,midmem,midcpu] with no gpu, so SLURM put it on the CPU partition with no --gpus, the log said GPU available: False, used: False, and it hit the 4 h midtime ceiling on all six datasets. It now uses [hightime, midmem, midcpu, gpu], matching drvi, scvi, scanvi, cellplm and condo.

Four methods want more than 8 h, and do not get it. uce, mnnpy, batchelor_mnn_correct and scalex all completed on dkd -- in 5 h 05, 23 min, 4 h 57 and 28 min respectively -- and were killed at 7 h 59 on every larger dataset. They stay on hightime: methods never get veryhightime, so every method gets the same 8 h ceiling (fair comparison) and no job ties up a node for a day. Timing out on the larger datasets is a scaling limit of those methods and is recorded as such in the results. scalex does go from lowcpu to midcpu, since it is CPU-bound by circumstance rather than by design (see below).

Two things deliberately not done here:

  • scalex passes gpu=0 to scalex.SCALEX() but its engine is openproblems/base_python:1, which has no CUDA, so it can only ever run on the CPU. Moving it to the pytorch base image is the real fix, but that is blocked on the GPU driver on the de.NBI workers being older than the CUDA the current pytorch base image needs.
  • scimilarity has the same shape as fadvi did -- pytorch base image, no gpu label -- but it completes fine on the CPU, so it is left alone.

Checklist before requesting a review

  • I have performed a self-review of my code

  • Check the correct box. Does this PR contain:

    • Breaking changes
    • New functionality
    • Major changes
    • Minor changes
    • Bug fixes
  • Proposed changes are described in the CHANGELOG.md

  • CI Tests succeed and look good!

@rcannood rcannood changed the title Give fadvi a gpu label and raise the walltime of the methods that timed out Give fadvi a gpu label and bump scalex to midcpu Aug 21, 2026
@rcannood
rcannood force-pushed the fix-resource-labels branch from d507258 to ca73f2d Compare August 21, 2026 07:16
@rcannood
rcannood force-pushed the fix-resource-labels branch from ca73f2d to 1ac7c78 Compare August 21, 2026 07:17
@rcannood
rcannood merged commit c098818 into main Aug 21, 2026
1 check passed
@rcannood
rcannood deleted the fix-resource-labels branch August 21, 2026 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant