Skip to content

cuda: query the device type with cudaDeviceGetAttribute, not cudaGetDeviceProperties (+8% decode) - #163

Merged
khosravipasha merged 1 commit into
prismfrom
fix/cuda-dev-type-attr
Sep 8, 2026
Merged

cuda: query the device type with cudaDeviceGetAttribute, not cudaGetDeviceProperties (+8% decode)#163
khosravipasha merged 1 commit into
prismfrom
fix/cuda-dev-type-attr

Conversation

@bri-prism

Copy link
Copy Markdown

What

ggml_backend_cuda_device_get_type read the whole cudaDeviceProp to test the integrated flag. It now asks cudaDeviceGetAttribute(cudaDevAttrIntegrated). HIP gets the attribute alias in vendors/hip.h; MUSA already resolves the CUDA name.

Why

The full property read costs about 660 us per call on an H100; the attribute read about 20 ns. The hybrid-attention graph builder queries the device type of every device on every recurrent layer of every graph build, so one decode graph pays it once per recurrent layer, dozens of times. An nsys profile in July put it at 409 cudaGetDeviceProperties calls / 337 ms per llama-bench run against 25 calls / 22 ms for upstream, about 0.3 ms per decoded token, and it was the largest identified piece of the fork's host-side decode gap vs upstream.

The graph-side loop is left as is: with the backend call cheap it costs ~2 us per build, and caching it in the model would have to reason about multiple loaded models.

Verification (L40S, CUDA 12.8, sm_89)

  • Build: clean (three builds of the same tree; this change touches no device code).
  • test-backend-ops test -b CUDA0 -o MUL_MAT: 1283/1283 passed, -o MUL_MAT_ID: 1019/1019 passed on the sibling build of the same tree (exit 0 both).
  • llama-bench, 9B PTQ1_0 and PQ2_0, r=3, two interleaved passes against base 8c0170d19:
9B band pp512 base pp512 fix tg128 base tg128 fix decode
PTQ1_0 5,216 5,209 170.6 183.7 +7.7%
PQ2_0 8,364 8,317 164.6 178.2 +8.2%

Prefill differences are inside the run-to-run spread (about 8% on this card at pp512); decode error bars are 1 to 4 tok/s.

Bit-exact by construction: no kernel changes, only the host query.

…eviceProperties

ggml_backend_cuda_device_get_type read the whole cudaDeviceProp to test one
flag. The full property read costs ~660 us per call; the hybrid-attention
graph builder asks for the device type of every device on every recurrent
layer of every graph build (dozens of calls per graph), which an H100 nsys
profile put at 409 cudaGetDeviceProperties calls / 337 ms per llama-bench run
against 25 / 22 ms for upstream, about 0.3 ms per decoded token.

cudaDeviceGetAttribute(cudaDevAttrIntegrated) answers the same question in
~20 ns. HIP gets the attribute alias; MUSA resolves the CUDA name already.
@khosravipasha
khosravipasha merged commit 3dd7675 into prism Sep 8, 2026
9 of 10 checks passed
@khosravipasha

Copy link
Copy Markdown
Collaborator

Reproduced on our L40S (CUDA 12.8, sm_89). PR head f7ae96b vs its base 8c0170d, separate worktrees, two interleaved passes of llama-bench -p 512 -n 128 -r 3 -fa 1, averaged:

Band pp512 base pp512 PR tg128 base tg128 PR decode
2B PQ2_0 23958 23925 400.4 437.3 +9.2%
4B PTQ1_0 8659 8598 257.2 275.7 +7.2%
9B PTQ1_0 5364 5382 177.2 186.1 +5.0%
9B PQ2_0 8705 8700 171.5 179.9 +4.9%
27B PQ2_0 2772 2784 69.3 71.9 +3.7%

Prefill unchanged within noise; decode gains consistent across both passes and well outside the 1 to 2 tok/s error bars. Smaller models gain most, as expected for a fixed per-token host cost.

Review notes:

  • The hot caller is the per-layer device loop in the Qwen3.5 graph builder (the GDN state-path check), which is fork code. Hoisting that check out of the layer loop would be the tidier fix eventually, but with the attribute query this cheap it no longer matters. Not blocking.
  • The Windows CI failure is the CPU-only server test_completion_unified KV-slot test, unrelated to this change; re-ran the job.

Merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants