Use VBIT for the arm64 NEON AndNot popcount kernel

[lemire]

Speeds up the arm64 NEON AndNot cardinality kernel (_popcntMaskSliceNEON) by 1.28x, by expressing s &^ m in one instruction per vector instead of two.

Credit for the observation goes to David Sparks, who pointed out that arm64 has a dedicated and-not in Advanced SIMD and that the kernel was not using it.

The problem

_popcntMaskSliceNEON computes sum(popcount(s[i] &^ m[i])). Every other kernel in the file (And, Or, Xor) combines its two inputs with a single vector op before counting. The Mask kernel could not, because it was written on the assumption that NEON has no and-not, so it synthesised one:

VMOVI $255, V15.B16            // V15 = all ones, materialised once

VEOR V15.B16, V4.B16, V4.B16   // V4 = ~m
VEOR V15.B16, V5.B16, V5.B16
VEOR V15.B16, V6.B16, V6.B16
VEOR V15.B16, V7.B16, V7.B16
VAND V4.B16, V0.B16, V0.B16    // V0 = s & ~m
VAND V5.B16, V1.B16, V1.B16
VAND V6.B16, V2.B16, V2.B16
VAND V7.B16, V3.B16, V3.B16

Eight vector operations per 64-byte block, plus a register tied up holding all-ones, to express something the hardware does in four.

The fix

arm64 does have a vector and-not — BIC — but the Go assembler has no mnemonic for the vector form (its arm64 mnemonic table has VBIT, VBIF, VBSL and VBCAX, but no VBIC). VBIT fits just as well and is spelled:

VEOR V15.B16, V15.B16, V15.B16 // V15 = 0

VBIT V4.B16, V15.B16, V0.B16   // V0 = s &^ m
VBIT V5.B16, V15.B16, V1.B16
VBIT V6.B16, V15.B16, V2.B16
VBIT V7.B16, V15.B16, V3.B16

What VBIT does

BIT is "bitwise insert if true". It is a bit-by-bit select between the destination register and one source, under the control of the other source:

Vd<i> = Vm<i> ? Vn<i> : Vd<i>        i.e.   Vd = (Vm & Vn) | (^Vm & Vd)

Wherever the control register Vm has a 1 bit, the corresponding bit of Vn is inserted into Vd; wherever Vm has a 0 bit, Vd keeps what it had. It is normally used for merging two vectors under a mask.

Hold Vn at zero and it degenerates into something more useful here: every bit set in Vm is cleared in Vd, and every other bit of Vd survives. That is precisely Vd &^= Vm. So with Vd = s and Vm = m, one VBIT applies the mask in place.

(Note that in Go assembly the destination is written last, so VBIT Vm, Vn, Vd corresponds to ARM's BIT Vd, Vn, Vm.)

V15 is still reserved, but it now holds zero — as the Vn operand — rather than all-ones for the invert. It is zeroed with VEOR instead of VMOVI.

Why this shows up as a real speedup

The inner loop goes from 8 combining ops to 4, i.e. 13 vector ALU ops per 64-byte block down to 9.

That matters because this kernel was ALU-bound, not load-bound. On an Apple M4 the NEON unit retires 4 vector ALU ops per cycle, and the loop's two VLD1s (8 uops of 16 bytes) can issue in fewer cycles than the arithmetic needs, so removing four ops removes four ops' worth of wall clock rather than being absorbed by spare issue slots. The measured throughput lands almost exactly where the op count predicts.

Benchmarks

Apple M4 Max, Go 1.24.4, darwin/arm64. Medians of 6 runs.

kernel, 1024 words (one bitmap container)	before	after
popcntMaskSlice	137.7 ns	107.3 ns	1.28x
popcntAndSlice	106.3 ns	106.0 ns	1.00x
popcntOrSlice	106.3 ns	106.4 ns	1.00x
popcntXorSlice	106.3 ns	106.1 ns	1.00x
popcntSlice	73.2 ns	73.1 ns	1.00x

62 GB/s → 80 GB/s on the Mask kernel. The gain holds down to short slices (1.16x at 64 words) and is neutral, not negative, on the sub-8-word tail path.

Container-level AndNot gains far less — around 1%. andNotBitmap and iandNotBitmapSurely call this kernel for the cardinality and then make a second pass, a scalar Go loop writing bc.bitmap[k] &^ value2.bitmap[k] over 1024 words. That write loop, not the cardinality pass, is where the time goes. Worth knowing before anyone reads too much into the 1.28x; a vectorised write loop is the larger prize and is not attempted here.

Correctness

• Full existing suite passes on darwin/arm64.
• Additionally differential-tested against bits.OnesCount64 over lengths 0–33000 (including every boundary at 1, 7, 8, 9, 15, 16, 17, 63, 64, 65, 1023, 1024, 1025, 8191, 8192, 8193) with both random and all-ones inputs, and stress-tested past the INNERMAX drain boundary where the 16-bit lane accumulators would overflow if mis-sized.

BIT is baseline ARMv8-A Advanced SIMD — no feature detection needed, same as everything else in this file. On other implementations (Neoverse, Cortex-A) BIT is rate-matched with AND, so the win should carry, though I have only measured M4.

Alternatives measured and rejected

Recorded here so they don't get re-proposed:

• UADALP (pairwise-add-long-and-accumulate) to fold byte lanes directly into the 16-bit accumulator would save one instruction per iteration. It has no Go assembler mnemonic, and hand-encoding it measures slower: an isolated throughput probe puts it at ~2.4 instructions/cycle against 4.0 for VADD/VCNT/VUADDW/VBIT. It costs the plain popcount kernel ~20%, and the regression is independent of how many accumulator chains it is split across (tested with 1, 2 and 4). A note to this effect is added to the file header. It is a small win (~4%) on the two-input kernels, which are load-bound, but not worth a raw instruction-word encoding.
• Removing the integer accumulator so the sum stays in NEON registers end to end: neutral at container sizes and a regression on short slices (0.63x at 7 words), because it puts two serialised VUADDLV horizontal reductions on the critical path where the current code pipelines one per word. Short slices are reachable in practice via popcntSlice(bc.bitmap[:(x+1)/64]) in rank.
• CSEL-based block-count setup in place of the compare-and-branch: correct and one instruction shorter, but perf-neutral (0.99–1.02x) since it runs once per call. Not worth churning hand-written assembly for.
• Raising INNERMAX from 1024 to 2047: safe (2047x32 = 65504 fits a 16-bit lane), but changes nothing — a bitmap container is 128 blocks, so the drain runs once per call either way.