Skip to content

Improve roaring64 BSI value access and min/max - #544

Merged
lemire merged 2 commits into
masterfrom
bsi-value-access-minmax
Aug 11, 2026
Merged

Improve roaring64 BSI value access and min/max#544
lemire merged 2 commits into
masterfrom
bsi-value-access-minmax

Conversation

@guymolinari

@guymolinari guymolinari commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds a typed batch value-read API for roaring64.BSI and optimizes BSI min/max selection. It keeps existing public method signatures intact while adding an int64-oriented read path and replacing row-by-row MinMaxBig value reconstruction with bitmap plane pruning.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Performance improvement
  • Code refactoring
  • Documentation update
  • Test improvements
  • Build/CI changes

Changes Made

What was changed?

  • Added GetValues(columnIDs []uint64) ([]int64, []bool) for aligned batch reads from int64-width BSIs.
  • Reused the existing batch-read request/indexing helper so duplicate column IDs and missing values are handled consistently with GetBigValues.
  • Updated MinMaxBig to prune candidate rows by sign and value bit planes, then decode one surviving value, instead of reconstructing every candidate row value.
  • Added correctness tests for GetValues, duplicate/missing reads, subset min/max, and large-value min/max.
  • Added focused benchmarks for typed value reads and BSI min/max.

Why was it changed?

Both paths avoid common row-by-row BSI value loops. GetValues lets int64 callers avoid allocating big.Int values for every row. The MinMaxBig change keeps the existing API but changes the implementation to use bitmap algebra directly, which is much cheaper for large found sets.

How was it changed?

  • GetValues uses the same aligned found-set/position map used by GetBigValues, then fills primitive int64 and bool slices from intersected bit planes.
  • MinMaxBig first intersects the requested found set with the BSI existence bitmap. It then narrows candidates by sign plane and descending value planes according to the requested MIN or MAX, and finally calls GetBigValue on one remaining candidate.
  • Wider-than-int64 values continue to use the existing generic big-value read path where required.

Testing

go test ./roaring64 -run 'TestBSI64Get|TestMinMax' -bench 'BenchmarkBSI64(GetValues|MinMaxBig)' -benchmem -count=1
go test ./roaring64 -run '^$' -bench 'BenchmarkBSI64(GetBigValues|GetValues|GetBigValueLoop|MinMaxBig)' -benchmem -count=3
go test ./... -count=1

Formatting

gofmt was run on the touched Go files.

Fuzzing

Not run for this change. The touched code is covered by deterministic correctness tests over signed values, missing values, duplicate inputs, subset found sets, and wider-than-int64 BSI values.

Performance Impact

Environment: linux/amd64, Go toolchain from local development machine, Intel Core i7-1255U.

BenchmarkBSI64MinMaxBig uses 1,000,000 rows with low-cardinality int64 values and computes both min and max per operation.

Before, at current master (438e356) with only the benchmark fixture copied in:

BenchmarkBSI64MinMaxBig-12    4    394940968 ns/op    345738228 B/op    13600268 allocs/op
BenchmarkBSI64MinMaxBig-12    3    350017694 ns/op    345738018 B/op    13600267 allocs/op
BenchmarkBSI64MinMaxBig-12    4    295964200 ns/op    345736852 B/op    13600260 allocs/op

After:

BenchmarkBSI64MinMaxBig-12    850    1587877 ns/op    1641409 B/op    850 allocs/op
BenchmarkBSI64MinMaxBig-12    885    1549372 ns/op    1641407 B/op    850 allocs/op
BenchmarkBSI64MinMaxBig-12    570    1872720 ns/op    1641403 B/op    850 allocs/op

Candidate value-read benchmarks on a 100,000-row fixture:

BenchmarkBSI64GetBigValuesLargeFixture-12       28174978-45175203 ns/op    ~8213 KB/op    200357 allocs/op
BenchmarkBSI64GetValuesLargeFixture-12          22678089-32531669 ns/op    ~4321 KB/op    458 allocs/op
BenchmarkBSI64GetBigValueLoopLargeFixture-12    73701615-101170173 ns/op   ~35602 KB/op   1299901-1299902 allocs/op

Breaking Changes

None.

Compatibility

  • Existing public method signatures are unchanged.
  • GetValues is additive.
  • GetBigValues remains available for callers that need arbitrary-width big.Int values.
  • MinMaxBig preserves the existing API and result type.
  • For an empty effective found set, MinMaxBig preserves the previous sentinel behavior.

Related Issues

None.

Additional Notes

The PR is intentionally split into two commits so the additive typed read API and the min/max implementation change can be reviewed independently.

@lemire
lemire merged commit 957d6e8 into master Aug 11, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants