Optimize BF16 quantized matmul with FP16 tiles - #4216
Closed
justinchuby wants to merge 3 commits into
Closed
Conversation
This was referenced Aug 12, 2026
Use vectorized BF16-to-FP16 loads and wider tiles for affine INT4 matrix-matrix workloads while preserving BF16 I/O. Keep the path opt-in and cover AOT/JIT tile dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5f4a437b-39c5-44ac-9bd7-e726eed87db0
justinchuby
force-pushed
the
optimize/bf16-qmm-fp16-tiles
branch
from
August 12, 2026 22:04
39835ce to
5c42446
Compare
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 68eb37ce-1aa5-4ef3-a9e9-ef0176f326e2
Member
|
Thanks for the PR, while it is a nice improvement we prefer users to convert to fp16 themselves rather than us implicitly do the conversion. |
Author
|
Thanks - explicit cast verified to have matching performance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enable with
MLX_BF16_QMM_FP16=1;MLX_BF16_QMM_FP16_TILE=32selects the narrow comparison path.Performance
M1 Max, M=512, K=6656, N=19968, affine INT4 group size 32:
The mixed path is ~1.77x faster than stock BF16 and effectively matches pure FP16 while preserving BF16 I/O.
Validation
python/tests/test_quantized.py: 33 tests / 2896 subtests passMLX_METAL_JIT=ONbuilds