Fail DeepSSM jobs instead of reporting success - #2622
Merged
Conversation
DeepSSMJob::run() caught exceptions and returned normally, so a failed job was still marked complete: training that died at model initialization logged "Training complete", Run All chained into testing, and the CLI exited zero. Jobs now carry a failed state, which stops the chaining and makes the CLI exit non-zero. DeepSSM also selected the GPU on torch.cuda.is_available() alone, which is true even when the installed PyTorch wheel has no kernels for that card. Probe the GPU with a small kernel launch before using it and fall back to the CPU with a message naming the device capability and the wheel's supported architectures.
A machine with a GPU that PyTorch cannot use is a setup problem to fix, not a reason to silently start a run that would take days. get_device() now raises UnusableGPUError with the diagnostic message rather than returning the CPU. Machines with no GPU at all still run on the CPU as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2621
DeepSSMJob::run()caught exceptions and returned normally, so a failed job was still marked complete: training that died at model initialization logged "Training complete",Run Allchained into testing (which then failed on the missingbest_model.torch), and the CLI exited zero.Jobnow carries a failed state thatPythonWorkerreports instead of the completion message, so Studio does not advance to the next step andshapeworks deepssmstops and exits non-zero.PythonWorkeralso marks the job complete on failure now, which it previously did not do, leaving the CLI's wait loop spinning forever.The underlying failure is that light-the-torch 0.8.0's driver table stops at CUDA 12.6, so it can never select a cu128 wheel — every RTX 50-series user gets a PyTorch with no
sm_120kernels and DeepSSM cannot train at all. Bumped to 0.8.1, which knows CUDA 12.8 through 13.1.net_utils.get_device()now also probes the GPU with a small kernel launch and raisesUnusableGPUErrornaming the device capability, the architectures the wheel supports, and whether a newer or older PyTorch is needed. It does not fall back to the CPU — that would silently turn a fixable setup problem into a run of several days. Machines with no GPU at all still run on the CPU as before.swpip installnow passes--upgrade, without which pip skips files that already exist under--targetand reports success while leaving the old build in place — so the reinstall command in the error message and docs actually replaces PyTorch.Both DeepSSM tests pass on CPU. The failure path was checked by forcing the unusable-GPU branch: the run stops at data loader preparation with the full message in the log, testing does not run, and the CLI exits 1.