Skip to content

Fix M5 split-K NAX dispatch cliff - #4237

Closed
PhilipJohnBasile wants to merge 1 commit into
ml-explore:mainfrom
PhilipJohnBasile:agent/fix-4198-splitk-nax
Closed

Fix M5 split-K NAX dispatch cliff#4237
PhilipJohnBasile wants to merge 1 commit into
ml-explore:mainfrom
PhilipJohnBasile:agent/fix-4198-splitk-nax

Conversation

@PhilipJohnBasile

Copy link
Copy Markdown
Contributor

Fixes #4198.

Summary

On M5, the split-K heuristic routes one-M-tile transposed QMMs in the
N=6656..8192 band to the plain Metal qmm_t_splitk kernel. The fallback
qmm path is NAX-capable and is faster for the measured two-way split.

This change:

  • factors the existing fallback-QMM NAX eligibility check so the split-K
    decision uses the exact same predicate;
  • bypasses split-K only on the measured applegpu_g17s M5 Max class for
    single-batch, transposed affine 4-bit/group-64 QMM with one M tile,
    N=6656..8192, K divisible by 128, and a finalized split count of two;
  • leaves all other devices, modes, shapes, and split counts on the existing
    path.

The narrow shape/device gate is intentional. Prior measurements show split-K
still wins in other regions, and this PR only changes the band demonstrated in
#4198. It composes with #4171 if that NAX small-M tiling change lands.

Validation

  • Clean editable build from a8e24f202: passed.
  • git diff --check, clang-format, Black, Python syntax: passed.
  • CPU test_quantized: 37 passed, 3 skipped.
  • Pure C++ dispatch-policy test: 18 assertions passed.
  • Targeted M5 numerical regression, including M=32/N=6656/K=6656 and adjacent
    N/M boundaries: passed.
  • Full M5 Max GPU test_quantized: 37 passed.
  • M5 Max ABBA performance sweep: passed; details below.

M5 Max performance

Apple M5 Max (applegpu_g17s), macOS 27.0, bfloat16 activations, affine
4-bit/group-64 weights. Baseline and candidate were separate clean builds from
the same a8e24f202 source. The Metal library hash was identical; only the host
libmlx differed. Each cell used 64 dependent operations, warmup, and seven
samples per arm in baseline/candidate/candidate/baseline order.

Shape M baseline candidate speedup
K=6656, N=6656 32 251.4 us 196.4 us 1.28x
K=6656, N=7168 32 275.1 us 199.8 us 1.38x
K=6656, N=7680 32 277.4 us 202.6 us 1.37x
K=6656, N=8192 32 299.6 us 201.8 us 1.48x
K=4096, N=8192 32 187.7 us 126.1 us 1.49x
K=5120, N=8192 32 232.0 us 158.5 us 1.46x
K=8192, N=8192 32 361.1 us 241.8 us 1.49x
K=6656, N=8224 (control) 32 204.0 us 204.6 us 1.00x

Across all 33 affected M<=32 cells, speedup was 1.22–1.49x (median
1.37x). Across 24 M>=33 or N=8224 controls, the ratio was 0.992–1.012x
(median 0.999x).

The M=33/M=32 ratio at N=6656/7168/7680/8192 moved from
0.781/0.733/0.732/0.682 on the baseline to 1.005/1.002/1.000/1.007 on the
candidate, eliminating the dispatch cliff. Baseline and candidate ABBA arm
drift stayed below 1.4% and 1.9%, respectively.

@PhilipJohnBasile
PhilipJohnBasile marked this pull request as ready for review August 13, 2026 16:56
@zcbenz zcbenz added the await verification This pull request is non-trivial and requires a human expert to verify its correctness. label Aug 13, 2026
@zcbenz

zcbenz commented Aug 16, 2026

Copy link
Copy Markdown
Member

Thanks for the PR but currently we are only focusing on real cases happened during inference otherwise we wouldn't be able to review all the improvements, so I'm closing this PR.

@zcbenz zcbenz closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

await verification This pull request is non-trivial and requires a human expert to verify its correctness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qmm_splitk's 512-threadgroup target routes M<=32 transposed qmm to the non-NAX split-K kernel on M5 (M=33 is 21% faster than M=32)

2 participants