Add SIMD emulation of mul_add_precise for f64 on SSE4.2 - #324
Conversation
awxkee
left a comment
awxkee
left a comment
There was a problem hiding this comment.
LGTM
Dekker product with Veltkamp's split is no doubts impossible for edge cases which the paper covers only with a blanket "provided that underflow and overflow do not occur", and instead it use fast2mult using FMA in annex to compute FMA, what is a little bit ironic. Since values that are impossible to split are gated here, everything else seems correct to me.
|
Thanks a lot! |
Shnatsel
changed the title
mul_add_precise for SSE4.2mul_add_precise for f64 on SSE4.2
…es, but only 0.7x on subnormal/huge values
The kernel now magnitude-sorts both error-free additions and uses Fast2Sum, shortening their dependency chains. The correction predicate also omits redundant exponent classification under the existing safe-range proof. Also tried and rejected an exponent-bit correction kernel and a one-blend Fast2Sum formulation because they regressed the real benchmark. Here's what llvm-mca thinks: | CPU model | Original throughput | Optimized throughput | Modeled total cycles | |---|---:|---:|---:| | Zen 4 | 11.5 → 9.5 | 17% better | 41.1 → 36.1 | | Nehalem | 24 → 20 | 17% better | 53.1 → 49.1 | | Tremont | 47 → 37 | 21% better | 108.0 → 87.1 |
…ance by 25% in time and 33% in throughput on Zen4 benchmarks and showing llmv-mca improvements across the board
ab19a76 to
9e4a6cf
Compare
|
Now that #323 is merged, I've rebased it on main and this is now ready for review |
LaurenzV
left a comment
LaurenzV
left a comment
There was a problem hiding this comment.
I obviously haven't tried to verified this. 😄
There was a problem hiding this comment.
Could we again move the ignored tests to the bottom?
| b: f64x2<Sse4_2>, | ||
| c: f64x2<Sse4_2>, | ||
| ) -> f64x2<Sse4_2> { | ||
| let a_raw: __m128d = a.into(); |
There was a problem hiding this comment.
For such a long function, is it really good to inline everything? Should we maybe do an inline never and use simd.vectorize here somehow?
There was a problem hiding this comment.
Function calls are quite expensive with SIMD arguments so I don't think #[inline(never)] is a good idea. But I've been thinking about inlining and vectorize recently and I do believe there is a way to do it that'll be a good fit.
There was a problem hiding this comment.
Upon further consideration, kernel! already desugars into a function declaration with #[inline] and #[target_feature] on it, but not #[inline(always)].
Outlining this sounds like a good idea because it could reduce register pressure and avoid spills. However, functions calling SIMD are quite slow because all the SIMD vectors need to be spilled to the stack, they cannot be passed in registers. A change to pass them in registers was made in Rust 1.87 but it broke inlining across functions with compatible but not identical #[target_feature] annotations and caused massive regressions.
I don't think it's a good idea to force register spills here out of fear it might cause register spills if we don't force them.
3 participants
The implementation is based on a 2025 paper by Graillat and Muller, Emulation of the FMA and the correctly-rounded sum of three numbers in rounding-to-nearest floating-point arithmetic. Integrating cutting edge research here!
To the best of my knowledge this doesn't fix any libm or musl bugs, so this isn't necessary for correctness, it's just an optimization.
This is upwards of 3x faster than scalar on normal values; huge values and subnormals fall back to scalar and only get about 0.7x the usual scalar performance. It's still worth it because subnormals are rare and slow even in hardware.
@awxkee I'd appreciate it if you could take a look