Fix ZSTD_DCtx leak when a frame fails to decode

[Watson1978]

Summary

Zstd.decompress creates a ZSTD_DCtx and frees it after decode_one_frame returns, but decode_one_frame raises whenever libzstd reports an error, so the free is skipped and the context is lost.

The size matters because libzstd allocates the context's inBuff and outBuff from the frame header, before decoding any block. The header is attacker-supplied, so the caller decides how much is leaked per failed call — up to the default window cap.

Measured with a valid header followed by a body that fails to decode: 200 such calls grow RSS by ~435 MB, about 2.2 MB per call, and it does not come back. A header declaring the maximum default window leaks far more. Anything that decompresses untrusted bytes can be walked into OOM by repeating a single malformed request.

Fix

Run the decode under rb_ensure, so the context is freed on every path. This also covers the scratch buffer, which leaked too if rb_str_cat raised while appending output.

decode_one_frame now owns the context it is given, so rb_decompress no longer frees it. One ordering detail is load-bearing: set_decompress_params frees the context itself before raising, so it has to run before the ensure takes ownership — otherwise the two would double free.

Tests

The new spec only walks the failure path and asserts the raise; it does not try to measure memory. The leak detection belongs to rake spec:valgrind (#147), which reports this on the path the spec exercises:

189,976 (95,976 direct, 94,000 indirect) bytes in 1 blocks are definitely lost
  malloc
 *ZSTD_createDCtx (at .../zstdruby.so)
 *rb_decompress   (at .../zstdruby.so)

With this change that report is gone and the task exits 0.

Compatibility

No API or behavior change. The same inputs raise the same errors; only the memory that was previously abandoned is now released.

Note on overlap

This touches the same loop in decode_one_frame as #143, so whichever lands first will leave the other with a textual conflict. They are independent changes and I am happy to rebase this one on top of that.

🤖 Generated with Claude Code