bzip2 llm rewrite (~1.5-2x speedup)

[dweiss]

TL;DR; I am a human but I used LLMs to rewrite bzip2 code in commons-compress and it's 2x as fast. I thought you'll find this interesting - it isn't meant to be merged in as-is but is sure as hell interesting (at least to me!).

So, I've been working with (huge) bz2 files from Java level ([1]) - their decompression takes... a good while (why wikimedia doesn't use zstd for this is another question I'll leave aside). Compression/decompression code is a great self-contained example of hairy stuff and I thought - it'd be interesting to burn some watts and give LLM a task to improve the current implementation.

In short - I gave it a simple prompt of : "Would it be possible to speed up apache-commons bzip2 implementation (especially the decompression) using new Java language features? Consult with implementations in other languages, if needed."

And worked iteratively from there. It was very much unsupervised in what it chose to optimize and how. Eventually though, this is what I see (paths a bit redacted, disk cache-primed):

$ time java -cp commons-io-2.22.0.jar:commons-compress-1.29.0-SNAPSHOT.jar BZip2SumTest.java enwiki_general-20260823-00000.json.bz2
enwiki_general-20260823-00000.json.bz2: 3,364,783,751 bytes, sum = 300,473,491,475 (47.00 s, 71.6 MB/s)

real    0m47.469s
user    0m49.976s
sys     0m0.236s

$ time java -cp commons-io-2.22.0.jar:commons-compress-1.28.0.jar BZip2SumTest.java enwiki_general-20260823-00000.json.bz2
enwiki_general-20260823-00000.json.bz2: 3,364,783,751 bytes, sum = 300,473,491,475 (91.01 s, 37.0 MB/s)

real    1m31.506s
user    1m32.751s
sys     0m0.211s

$ time bunzip2 -c enwiki_general-20260823-00000.json.bz2   | wc -c
3364783751

real    1m10.402s
user    1m9.625s
sys     0m3.266s

Which are... crazy numbers but they're consistent with what my eyes see... I don't know if it optimized for my particular CPU/ cache layout or not but I found it interesting that (a) it's correct and works all across the board, (b) it is much, much faster than the code currently in apache-commons.

I wonder what people think and if these results reproduce for others (I haven't tried this yet across any other known compression data sets like the silesia corpus, etc.).

I also include the "description" of changes from the LLM from those interested [2].

[later] After a round of improvements, experiments and tweaks, including parallel stream decompression, it now looks like this -

[1] https://dumps.wikimedia.org/other/cirrus_search_index/20260823/index_name%3Denwiki_content/
[2] LLM summary of changes.

[ecki]

Wow, that’s a neat improvement. (I have a feeling the single byte and Boolean Arrays of the old code hurt, and the new one dient even use crazy optimisations. In that regard, the native Tool Performance is shocking unoptimized.

Did you disclose your agent, model, harness and skills?

[dweiss]

(I have a feeling the single byte and Boolean Arrays of the old code hurt, and the new one dient even use crazy optimisations.

I agree - it doesn't look too terrifying. Sure - it's complex and uses some assumptions (cache and cpu parallelism friendliness) but it's not revolutionary. It is still a significant speed improvement. What was a bit shocking to me is this - this time on a mac (completely different architecture):

$ curl https://dumps.wikimedia.org/other/cirrus_search_index/20260823/index_name%3Denwiki_general/enwiki_general-20260823-00000.json.bz2 > /tmp/enwiki.bz2
$ mvn package dependency:copy-dependencies

$ time java -cp target/dependency/commons-io-2.22.0.jar:target/commons-compress-1.29.0-SNAPSHOT.jar ./temp/BZip2SumTest.java /tmp/enwiki.bz2
/tmp/enwiki.bz2: 3,364,783,751 bytes, sum = 300,473,491,475 (34.71 s, 96.9 MB/s)
java -cp  ./temp/BZip2SumTest.java /tmp/enwiki.bz2  36.12s user 0.21s system 103% cpu 34.989 total

$ time bunzip2 -c /tmp/enwiki.bz2 | wc -c
 3364783751
bunzip2 -c /tmp/enwiki.bz2  44.51s user 0.43s system 99% cpu 44.946 total

Wild. It is much faster than the native tool.

Did you disclose your agent, model, harness and skills?

I didn't want this to look like an ad for a particular vendor... you can sort of see it in the intermediate commits - I used claude and fable/ high. I don't have any particular skills set up but I'm sure it collects and reuses some knowledge I've used in the past sesssions... hard to tell what it does know about one and what it doesn't these days.

[dweiss]

I (the llm) also applied the same (?) concepts to the C version - https://github.com/dweiss/bzip2

It is faster than before (first is patched, recompiled, second is the original):

$ time ./bzip2 -d -c enwiki_general-20260823-00000.json.bz2 | wc -c
3364783751

real    0m43.440s
user    0m42.915s
sys     0m3.073s

$ time bzip2 -d -c enwiki_general-20260823-00000.json.bz2 | wc -c
3364783751

real    1m8.521s
user    1m7.824s
sys     0m3.212s

[rmuir]

in my experience with lucene, you can get even faster if you can use the builtin CRC in the openjdk, rather than writing your own.

I believe they have two versions, one with the zlib polynomial, the other with the CRC-32C. These are implemented as hotspot intrinsics and are massively optimized, e.g. they will use instructions available such as VPCLMULQDQ and so on?

[dweiss]

Yes, there's more options but this is stuck at Java 8 (!). It'd have to be more complex to discover the builtin and call it but certainly something to try if this is to be taken further.

I haven't used bz2 in a long time (zstandard is so much faster and offers similar compression ratios) but if it can be improved, why not do it. :)

[dweiss]

Or did you mean the java.util.zip.CRC32? This hasn't always been accelerated though, has it? I'd have to look back and see - I'm not familiar with it.

[rmuir]

java.util.zip.CRC32 is the one i mean indeed. It is in java 8. This is the one lucene uses. I spent a ton of time benchmarking those parts. It is insanely fast, just give it enough buffer so that the vectorization can happen (don't give it like just a few bytes at a time). I think 1K or so is enough.

[rmuir]

Or did you mean the java.util.zip.CRC32? This hasn't always been accelerated though, has it?

this depends upon the the JDK version the end user is using (not the project's javac compiler). But just for history, they started hw-accelerating it in java 8: https://bugs.openjdk.org/browse/JDK-7088419. On arm i don't know the timeline but also they improve it, so if you use a newer JDK you'll take advantage automatically.

[dweiss]

Seems like the crc32 flavor in bzip uses different byte ordering so it can't be easily applied here (?). Claude did generate bit/byte reversals to use crc32, of course, but the net gain after that is zero.

[dweiss]

Here are the numbers - literally the same as before. Could be that the llm went sideways somewhere. Still fast though!

time java -cp target/dependency/commons-io-2.22.0.jar:target/commons-compress-1.29.0-SNAPSHOT.jar ./temp/BZip2SumTest.java /tmp/enwiki.bz2
/tmp/enwiki.bz2: 3,364,783,751 bytes, sum = 300,473,491,475 (33.85 s, 99.4 MB/s)
java -cp  ./temp/BZip2SumTest.java /tmp/enwiki.bz2  35.49s user 0.18s system 104% cpu 34.125 total

[dweiss]

I've reverted the crc32 patch because it was introducing more complexity for no real benefit. This code isn't to be merged anyway - it's more of an... exploratory experiment. Thanks for the suggestion though!

[garydgregory]

Hi @dweiss
Commons Compress already depends on Commons Codec where you'll find org.apache.commons.codec.digest.PureJavaCrc32 and org.apache.commons.codec.digest.PureJavaCrc32C. How about reusing that and dropping the Compress version?

[dweiss]

Commons Compress already depends on Commons Codec where you'll find org.apache.commons.codec.digest.PureJavaCrc32 and org.apache.commons.codec.digest.PureJavaCrc32C.

Good to know, thank you. Unfortunately this won't work here because bzip2 uses a different byte/bit ordering from standard crc32 (or crc32c) implementations. So it's very specific to this algorithm. Here's the LLM explaining the details:

• PureJavaCrc32C: wrong polynomial (Castagnoli). It can't produce bzip2's CRC under any transformation. Unusable.
• PureJavaCrc32: right polynomial, wrong bit orientation (reflected/LSB-first; bzip2 is MSB-first). Usable only via reverse32(crc(reverse8(M))), which costs a
  bit-reversal pass per byte — and since it's plain Java slicing-by-8 with no intrinsic behind it, you'd pay that pass on top of essentially the same table
  computation the tree already does. Net effect ≈ 1.6× slower CRC than the current class, for the sake of deleting ~60 lines.

[dweiss]

Ok. I've taken a look at what the LLM produced and it really isn't that hairy/ crazy - mostly engineering tweaks. All the existing tests (including https://sourceware.org/git/?p=bzip2-tests.git;a=summary) pass with flying colors.

I am willing to clean this up from LLM cruft (benchmarks it used to navigate through solutions, excessive comments, etc.) if there is a desire for this to be merged in - let me know. bzip2 is still used these days, even if better alternatives exist, so let's make the decompression faster?

[garydgregory]

FWIW, this my current priority is fixing the pile of bugs that have recently reported internally. This is nice to have but not a priority ATM.

[dweiss]

FWIW, this my current priority is fixing the pile of bugs that have recently reported internally. This is nice to have but not a priority ATM.

Absolutely, not a problem at all. I've cleaned up some LLM cruft and added some convenience parallel block decompression/compression stuff on top because it's handy for my use cases. Works beautifully.

I agree with you that it's not any priority since bzip2 is not in any mainstream use (again - not sure why wikimedia folks would pick it as their bulk data compressor; seems like a waste of compute power).

I'm fine with leaving this on a branch or even closing with won't commit status. It has been fun to explore the possibilities.