Fix ZSTD_DCtx leak when a frame fails to decode

[Watson1978]

Summary

Zstd.decompress creates a ZSTD_DCtx and frees it once the scan loop is done, but decode_one_frame raises whenever libzstd reports an error, so the free is skipped and the context is lost.

The size matters because libzstd allocates the context's inBuff and outBuff from the frame header, before decoding any block. The header is attacker-supplied, so the caller decides how much is leaked per failed call — up to the default window cap.

Measured with a valid header followed by a body that fails to decode: 200 such calls grow RSS by ~435 MB, about 2.2 MB per call, and it does not come back. A header declaring the maximum default window leaks far more. Anything that decompresses untrusted bytes can be walked into OOM by repeating a single malformed request.

Fix

Run the scan loop under rb_ensure, so the context is freed on every path. The scratch buffer in decode_one_frame gets the same treatment — it leaked as well if rb_str_cat raised while appending output.

One consequence is worth calling out. set_decompress_params used to free the context itself before raising; with an ensure also owning it, that would be a double free. It now raises and leaves the context to its owner:

• Zstd.decompress — the ensure added here
• StreamingDecompress — the TypedData free callback, so sd->dctx is assigned before the call rather than after

Verification

Under AddressSanitizer, over three paths: the failing decode, a rejected dict: argument, and the same rejection through StreamingDecompress.

Before this change the failing decode reports one leaked context per call:

Direct leak of 4,798,800 byte(s) in 50 object(s) allocated from:
  ZSTD_createDCtx  decompress/zstd_decompress.c:313
  rb_decompress    ext/zstdruby/zstdruby.c:113

After it, no allocation from ZSTD_createDCtx is reported on any of those paths, and none of them raises an ASan error — in particular the ownership change does not introduce a double free.

Tests

The two new specs walk the failure paths and assert the raise; they do not try to measure memory. Zstd.decompress(x, dict: 123) still raises ArgumentError with the same message as before.

Compatibility

No API or behavior change. The same inputs raise the same errors; only the memory that was previously abandoned is now released.

Note on overlap

This touches the same loop in decode_one_frame as #143, so whichever lands first will leave the other with a textual conflict. They are independent changes and I am happy to rebase this one on top of that.

🤖 Generated with Claude Code