Skip to content

Use CSEL to compute the NEON popcount inner-loop batch size - #548

Closed
lemire wants to merge 1 commit into
masterfrom
neon-csel-inner-count
Closed

Use CSEL to compute the NEON popcount inner-loop batch size#548
lemire wants to merge 1 commit into
masterfrom
neon-csel-inner-count

Conversation

@lemire

@lemire lemire commented Aug 13, 2026

Copy link
Copy Markdown
Member

Follow-up to a review suggestion on the AVX2 popcount PR, applied to the NEON side.

What

Each of the five NEON popcount routines opened its outer block loop by clamping the outstanding 64-byte-block count to INNERMAX with a compare and a forward branch:

	MOVD INNERMAX, R4
	CMP R4, R3
	BHS <label>
	MOVD R3, R4
<label>:
	SUB R4, R3, R3

This replaces that with a branch-free CSEL sequence, factored into a SPLITBLOCKS macro to sit alongside the file's existing FOLD4/ZEROPART/DRAIN macros:

	SUBS INNERMAX, R3, R6
	MOVD INNERMAX, R4
	CSEL CS, R4, R3, R4     // R4 = min(R3, INNERMAX)
	CSEL CS, R6, ZR, R3     // R3 = max(R3-INNERMAX, 0)

The SUBS computes the leftover count and sets the flags that choose between the two cases, so it is four instructions instead of five, branch-free, and it removes five now-unused labels (slinner, andinner, orinner, xorinner, maskinner).

R6 is the scratch register rather than R5, because R5 holds the slice length across the vector loop in the four two-input routines.

Performance: none, and that is expected

This is a code-size/clarity change. The sequence runs once per INNERMAX batch; since a roaring bitmap container is 1024 words = 128 blocks and INNERMAX is 1024 blocks, that is once per call, against an inner loop that saturates the NEON pipes for ~280 cycles. Measured on an arm64 laptop, -count 6:

benchmark before after
PopcntSlice1024NEON 70.3 ns/op 69.8 ns/op
PopcntAndSlice1024NEON 104.0 ns/op 103.9 ns/op

Both differences are inside run-to-run noise. Please do not merge this expecting a speedup.

Testing

go test ./... passes on arm64, as does go vet.

⚠️ Worth flagging: the existing tests do not cover this code path. neonTestLengths tops out at 1025 words = 128 blocks, so the outer block loop never iterates more than once and the CS side of both CSELs is never taken. I confirmed this by mutation: flipping CSEL CS to CSEL CC still passes TestNEONPopcntDifferential.

I verified this change locally with a temporary test at lengths that cross the batch boundary (8188/8191/8192/8193/8200, 16383–16385, 24576, 32775) plus an all-ones saturated case, differentially against the popcnt*SliceGo reference implementations. That temp test passes on this branch and fails on the mutant. It is not included here, since the request was to keep this PR to the CSEL change alone — happy to add it as a follow-up if you would like the coverage upstream.

The five NEON popcount routines each opened their outer block loop by
clamping the outstanding block count to INNERMAX with a compare and a
forward branch:

	MOVD INNERMAX, R4
	CMP R4, R3
	BHS <label>
	MOVD R3, R4
<label>:
	SUB R4, R3, R3

Replace it with a branch-free CSEL sequence, factored into a SPLITBLOCKS
macro alongside the file's existing ZEROPART/DRAIN/FOLD4 macros:

	SUBS INNERMAX, R3, R6
	MOVD INNERMAX, R4
	CSEL CS, R4, R3, R4
	CSEL CS, R6, ZR, R3

The SUBS computes the leftover block count and sets the flags that pick
between the two cases in one go, so this is four instructions instead of
five and drops a branch. It also removes five now-unused labels
(slinner, andinner, orinner, xorinner, maskinner).

This is a code-size and clarity change, not a speed one. The sequence
runs once per INNERMAX batch, and since a roaring bitmap container is
1024 words = 128 blocks, that is once per call, against an inner loop
that is saturating the NEON pipes for ~280 cycles. Benchmarks on an
arm64 laptop show no change beyond run-to-run noise:

	PopcntSlice1024NEON      70.3 ns/op -> 69.8 ns/op
	PopcntAndSlice1024NEON  104.0 ns/op -> 103.9 ns/op

R6 is used as the scratch register rather than R5, which holds the slice
length across the vector loop in the two-input routines.

Suggested by a reviewer on the AVX2 popcount PR.
@lemire lemire closed this Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant