feat: detect when the resolved torch/CUDA ships no kernels for the local GPU#3104
Open
golithe wants to merge 1 commit into
Open
feat: detect when the resolved torch/CUDA ships no kernels for the local GPU#3104golithe wants to merge 1 commit into
golithe wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
torch==2.4.1builds green, and every kernel launch fails withCUDA error: no kernel image is available for execution on the devicetorch.cuda.is_available()returnsTrue, and PyTorch's own diagnostic is aUserWarningcog rundoes not work with A100 #389 is the same failure class on an A100 in 2022 (closed when the reporter worked around it); Blackwell makes it current againThis PR adds:
GPUCompatibilityChecktocog doctor, modelled onDockerChecknvidia-smi --query-gpu=compute_cap(lowest across GPUs, since the image must run on the weakest) and compares the resolved torch/CUDA against the oldest release known to ship kernels for it.cog doctoroutput when it fires:Notes for review:
torch._C._cuda_getArchFlags(), which works without a GPU2.7.0+cu118is a genuine 2.7 build with no Blackwell kernelsSeverityWarning, matchingPythonVersionCheck: the image is valid and runs fine on other hardware; it only fails when executed on this machine's GPUgpu: false;COG_SKIP_GPU_CHECK=1skips it when building for different hardware than the local card. Doctor-only, no build-path changesTesting:
evaluateGPUCompat), table-tested without a GPU, including+cu128local-tag pins and the new-torch/old-CUDA casetorch==2.4.1, silent on2.7.1and ongpu: falseOut of scope (deliberately):
MinimumTorchVersionand friends inpkg/dockerfile/base.go)tools/to regenerate them on demand could be a follow-up; fullcompatgenintegration seems like a poor fit, since new capability majors ship every couple of years and the probe needs multi-GB wheel installs. Happy to add the script here or in a follow-up PR if useful