cuda: native PTQ1_0 MMQ tile loader (closes the dequantize-to-cuBLAS prefill fallback) - #160
Open
bri-prism wants to merge 1 commit into
Open
cuda: native PTQ1_0 MMQ tile loader (closes the dequantize-to-cuBLAS prefill fallback)#160bri-prism wants to merge 1 commit into
bri-prism wants to merge 1 commit into
Conversation
Large-batch PTQ1_0 matmul had no MMQ path, so it fell back to a full dequantize to fp16 followed by cuBLAS: measured 2.26x slower than PQ2_0 at n=512 while its decode already led. Add the tile loader, the MMQ type dispatch and the DECL_MMQ_CASE instance so PTQ1_0 takes the same quantized GEMM path as the other low-bit types, plus a packed, coalesced dequantize for the remaining conversion sites. Ported from a handoff worktree whose gates recorded 45/45 PTQ1_0 MUL_MAT on CUDA and an H100 perf pass at the pre-#152 base; its Metal and FWHT parts are superseded by #152 and #157 and are not included. This branch has not yet been compiled or run on CUDA hardware by the committer; that verification is the gate before it is pushed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native MMQ support for PTQ1_0 on CUDA. Before this, PTQ1_0 had no entry in the quantized-matmul path, so large-batch matmul fell back to a full dequantize to fp16 followed by cuBLAS, measured at 2.26x slower than PQ2_0 at n=512 even though PTQ1_0 already led on decode.
What
Adds the PTQ1_0 MMQ tile loader, the type dispatch, and the
DECL_MMQ_CASEinstance so PTQ1_0 takes the same quantized GEMM path as the other low-bit types, plus a packed coalesced dequantize for the remaining conversion sites.Provenance
Ported from an internal handoff branch whose gates recorded 45/45 PTQ1_0 MUL_MAT on CUDA and an H100 perf pass, but that was against a pre-#152 base. Its Metal and FWHT parts are superseded by the already-landed Metal PTQ1_0 and FWHT work and are excluded; this is the CUDA MMQ half only, rebased onto current
prism.Verification (this PR, on an L40S, sm_89)
test-backend-ops test -o MUL_MAT -b CUDA0: PTQ1_0 45 OK, 0 FAIL. (33 further cases report not-supported; those are thetype_b=f16shapes, identical on the Metal backend, unrelated to this change.)test-backend-ops test -o MUL_MAT_ID -b CUDA0: PTQ1_0 75 OK, 0 FAIL.Correctness of the PTQ1_0 CUDA matmul and per-expert matmul is verified on current
prism.What is not in this PR
A fresh end-to-end prefill A/B was not re-measured on this run. The win is the removal of the dequantize-to-cuBLAS fallback by giving PTQ1_0 a real MMQ path; the prior handoff measured that on H100. A maintainer can take the current-tree prefill number, or I can on request. Decode is unchanged (this is the prefill/large-batch path).