Keep the winograd gemm at full float32 precision - #4242
Conversation
| self.assertTrue(np.allclose(c_mx, c_np, atol=atol)) | ||
|
|
||
| def test_conv_2d_winograd_float32_precision(self): | ||
| # The aligned conv takes winograd and the unaligned one does not, so |
There was a problem hiding this comment.
curious why need to use sub process for this test ?
There was a problem hiding this comment.
mlx_tests.py sets MLX_ENABLE_TF32=0 at import, before mlx.core is imported, for the whole
suite:
# Use regular fp32 precision for tests
os.environ["MLX_ENABLE_TF32"] = "0"and enable_tf32() reads the environment once into a static:
inline bool enable_tf32() {
static bool enable_tf32_ = get_var("MLX_ENABLE_TF32", 1);
return enable_tf32_;
}So by the time any test runs, tf32 is off and the value is fixed for the life of the process.
Setting os.environ inside the test does nothing, and this change is a no-op with tf32 off, so an
in-process test would vacuously pass.
The subprocess runs the conv with MLX_ENABLE_TF32=1, which is the default a user gets. It
reports 2.38 before the change and 0.0023 after.
The same two facts mean the suite currently has no coverage of the default
tf32 configuration at all, which is why a 4859x accuracy difference on float32 conv went
unnoticed.
A float32 conv2d on hardware with neural accelerators returns roughly float16 accuracy, but only
when the channel counts are multiples of 32.
conv.cppsends 3x3 stride 1 convs to winograd whenC % 32 == 0andO % 32 == 0(plus sizeconditions). Winograd runs its batched gemm through
steel_matmul, which takes the nax path forfloat32 whenever
MLX_ENABLE_TF32is set, and that is the default. The winograd transforms thenamplify what tf32 rounded away. The implicit gemm path never calls
steel_matmul, so it isunaffected, which is why the same conv is accurate at 336 channels and not at 352.
N=4, 64x64, 3x3, padding 1, C = O, error against a cpu float32 reference:
Setting
MLX_ENABLE_TF32=0drops the float32 winograd error to 0.00693, a 600x change, and thesame shape on an M4 Pro, which has no neural accelerators, measures 0.00693 as well. Both point
at the same place, so this makes winograd ask for full float32 rather than depending on a global
that is about matmul.
steel_matmul_axpbydecidesuse_naxfromenv::enable_tf32()internally, so this adds anallow_tf32parameter defaulted to true and threads it to that decision. Every existing calleris unchanged; winograd passes false. The flag only has an effect for float32 inputs, since the
condition it guards is already false for every other dtype.
The float16 and bfloat16 rows above are a separate problem and are not addressed here. Those come
from the winograd intermediates being allocated in the input dtype, and fixing them means
doubling the working set, which is the opposite of what #4102 is trying to do.
Cost
N=8, 64x64, C = O = 352, calibrated, min of 7.
25% on the float32 winograd path and nothing anywhere else. It stays about 2x faster than the
implicit gemm the same conv falls to with an unaligned channel count.
Test
mlx_testssetsMLX_ENABLE_TF32=0for the whole suite andenable_tf32()reads theenvironment once into a static, so a test in the suite cannot see this. The new test runs the
conv in a subprocess with tf32 on and compares an aligned conv against an unaligned one. It
reports 2.38 before this change and 0.0023 after.
test_conv,test_blas,test_ops,test_autogradandtest_nnall pass, which is worthsaying because
matmul.hhas a lot of callers.This touches the
steel_matmulcall that #4102 moves, so whichever lands second needs a one linerebase.