gocroaring

Well-tested Go wrapper for CRoaring (a C/C++ implementation of Roaring Bitmaps)

Roaring bitmaps are used by several important systems:

• Apache Lucene and derivative systems such as Solr and Elastic,
• Metamarkets' Druid,
• Apache Spark,
• Netflix Atlas,
• LinkedIn Pinot,
• OpenSearchServer,
• Cloud Torrent,
• Whoosh,
• Pilosa,
• Microsoft Visual Studio Team Services (VSTS),
• eBay's Apache Kylin.

Roaring bitmaps are found to work well in many important applications:

Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods (Wang et al., SIGMOD 2017)

The original java version can be found at https://github.com/RoaringBitmap/RoaringBitmap

There is a native Go version at https://github.com/RoaringBitmap/roaring

This code is licensed under Apache License, Version 2.0 (ASL2.0).

Copyright 2016 by the authors.

Benchmarking

See https://github.com/lemire/gobitmapbenchmark for a comparison between this wrapper and the Go native version.

References

• Daniel Lemire, Owen Kaser, Nathan Kurz, Luca Deri, Chris O'Hara, François Saint-Jacques, Gregory Ssi-Yan-Kai, Roaring Bitmaps: Implementation of an Optimized Software Library arXiv:1709.07821
• Samy Chambi, Daniel Lemire, Owen Kaser, Robert Godin, Better bitmap performance with Roaring bitmaps, Software: Practice and Experience Volume 46, Issue 5, pages 709–719, May 2016 http://arxiv.org/abs/1402.6407 This paper used data from http://lemire.me/data/realroaring2014.html
• Daniel Lemire, Gregory Ssi-Yan-Kai, Owen Kaser, Consistently faster and smaller compressed bitmaps with Roaring, Software: Practice and Experience (accepted in 2016, to appear) http://arxiv.org/abs/1603.06549

Requirements

Go 1.24 or better, and a C compiler (cgo). The CRoaring sources are bundled with the package, so there is nothing else to install:

• go get github.com/RoaringBitmap/gocroaring

The bundled CRoaring version is reported by gocroaring.CRoaringVersion.

32-bit and 64-bit bitmaps

The package wraps both C APIs. gocroaring.Bitmap stores 32-bit integers and wraps roaring_bitmap_t; gocroaring.Bitmap64 stores 64-bit integers and wraps roaring64_bitmap_t. The two types offer the same operations, with the free functions of the 64-bit type carrying a 64 suffix (Or64, And64, Read64, and so on).

rb := gocroaring.New64()
rb.AddRange(1<<40, 1<<40+1000)
rb.RunOptimize()
fmt.Println(rb.Cardinality(), rb.Contains(1<<40+5))

Memory management

Bitmaps and iterators hold memory allocated by C. That memory is released automatically once the Go value becomes unreachable: we register a cleanup with runtime.AddCleanup rather than a finalizer, which lets the garbage collector reclaim a bitmap in a single cycle instead of two, and makes creating and discarding bitmaps about 20% cheaper than the finalizer-based approach we used previously. You may still call Free to release the C memory eagerly; doing so cancels the cleanup, and calling Free twice is harmless.

Iteration reads values from C in blocks rather than one at a time, so walking a bitmap through Iterator() is more than an order of magnitude faster than paying for a crossing of the Go/C boundary per value. Use NewIterator() when you need to move backwards or seek.

The CRoaring entry points are declared #cgo nocallback (none of them calls back into Go) and, where the function does not keep the buffer it is given, #cgo noescape (so the Go buffers we hand to C are not forced onto the heap). The frozen views are deliberately left out of the noescape list, since they do retain their buffer.

Example

Here is a simplified but complete example:

package main

import (
	"fmt"

	"github.com/RoaringBitmap/gocroaring"
)

func main() {
	// example inspired by https://github.com/fzandona/goroar
	fmt.Println("==roaring==")
	rb1 := gocroaring.New(1, 2, 3, 4, 5, 100, 1000)
	rb1.RunOptimize() // improves compression
	fmt.Println("Cardinality: ", rb1.Cardinality())
	fmt.Println("Contains 3? ", rb1.Contains(3))

	rb2 := gocroaring.New()
	rb2.Add(3, 4, 1000)
	rb2.RunOptimize() // improves compression

	rb1.And(rb2)
	// prints {3,4,1000}
	fmt.Println(rb1)

	rb3 := gocroaring.New(1, 5)
	rb3.Or(rb1)

        // prints 1, 3, 4, 5, 1000
        i := rb3.Iterator()
        for i.HasNext() {
          fmt.Println(i.Next())
        }
        fmt.Println()

	fmt.Println(rb3.ToArray())
	fmt.Println(rb3)

	rb4 := gocroaring.FastOr(rb1, rb2, rb3) // optimized way to compute unions between many bitmaps
	fmt.Println(rb4)

	// next we include an example of serialization
	buf := make([]byte, rb1.SerializedSizeInBytes())
	rb1.Write(buf) // we omit error handling
	newrb, _ := gocroaring.Read(buf)
	if rb1.Equals(newrb) {
		fmt.Println("I wrote the content to a byte stream and read it back.")
	}

	fmt.Println(rb1.Stats()) // show the cardinality and the numbers of each type of container used.
}

Documentation

Current documentation is available at http://godoc.org/github.com/RoaringBitmap/gocroaring

Mailing list/discussion group

https://groups.google.com/forum/#!forum/roaring-bitmaps

Compatibility with Java RoaringBitmap library

You can read bitmaps in Go, Java, C, C++ that have been serialized in Go, Java, C, C++.