Raise instead of hanging on a truncated frame in Zstd.decompress

[Watson1978]

Summary

Zstd.decompress hangs forever on a truncated / incomplete frame.

decode_one_frame loops until ZSTD_decompressStream returns 0. For a truncated frame, libzstd keeps returning a non-zero "need more input" hint while consuming and producing nothing, so the loop never terminates. Because ZSTD_decompressStream is called directly with the GVL held, this freezes the whole Ruby VM at 100% CPU: other threads stop, GC checkpoints are never reached, Timeout::timeout and Thread#kill have no effect, and the process does not even respond to SIGTERM (only SIGKILL stops it).

A header-only frame reproduces it:

Zstd.decompress("\x28\xB5\x2F\xFD")  # hangs forever (must be SIGKILLed)

This is reachable from any input source that feeds attacker-controlled compressed bytes to Zstd.decompress, for example a compressed network payload.

Raising here matches Zstd::StreamingDecompress#decompress, which already stops once the input is exhausted and returns gracefully on the same input.

This is a regression

Before 856938d (Fix Zstd.decompress glitches), rb_decompress called ZSTD_getFrameContentSize up front. For a truncated frame that returns ZSTD_CONTENTSIZE_ERROR, so the same input raised "not compressed by zstd" instead of hanging. The ZSTD_CONTENTSIZE_UNKNOWN path went through decompress_buffered, whose loop condition was while (input.pos < input.size) and therefore also terminated.

856938d replaced both paths with the current for (;;) loop, whose only exit conditions are a libzstd error and ret == 0. Neither holds for a truncated frame. The hang first shipped in v2.0.1 and is present through the current v2.0.7.

Scope: what this PR does not change

This PR does not touch GVL handling. ZSTD_decompressStream is still called directly, with the GVL held, exactly as it is on main.

Releasing the GVL here would additionally make the loop interruptible, and it would make one-shot decompression consistent with Zstd.compress and StreamingDecompress#decompress, which already go through the rb_thread_call_without_gvl wrappers. I deliberately left it out of this PR because it is a separate decision with its own history: #133 proposed exactly that change and was closed the same day pointing at #112, whose root cause turned out to be GC compaction and GC'd pointer lifetime (fixed in #116). Releasing the GVL while libzstd holds a RSTRING_PTR into the input String belongs in its own discussion, and it should not gate a fix for an unkillable hang. I am happy to open a separate PR or issue for it if that is wanted.

Compatibility

The only inputs whose behavior changes are ones that currently hang forever. Every input that currently returns a value still returns the same value, and every input that currently raises still raises. Valid single and concatenated frames are unaffected.

Known related issue, not fixed here

The new rb_raise unwinds past ZSTD_freeDCtx(dctx) in rb_decompress, so the ZSTD_DCtx leaks. This is pre-existing — the existing ZSTD_isError branch a few lines above has the same problem, and libzstd may have already allocated a window buffer of up to ~128 MB by then. Wrapping the decode in rb_ensure (or wrapping the DCtx in a TypedData object) fixes all of these paths at once. I kept it out of this PR to keep the diff minimal, and I am glad to send it separately.

🤖 Generated with Claude Code