Fix use-after-free of the CDict/DDict referenced by a stream

[Watson1978]

Summary

ZSTD_CCtx_refCDict and ZSTD_DCtx_refDDict only borrow the pointer they are given — zstd.h says the dictionary "must outlive its usage within CCtx". StreamingCompress and StreamingDecompress keep the context in their struct for the life of the object, but nothing kept the Ruby CDict / DDict object reachable, so a dictionary the caller did not store was collected while the context still pointed at it.

stream = Zstd::StreamingCompress.new(dict: Zstd::CDict.new(bytes, 5))
# nothing references the CDict any more
stream << data   # reads freed memory

The freed allocation holds the dictionary's entropy tables and its dictContent pointer and size. Once the slot is reused — a same-sized Ruby String is enough — those become whatever now occupies that memory, which is why this is a memory-safety bug and not only a crash.

Verification

Built with -fsanitize=address and run under LD_PRELOAD=$(gcc -print-file-name=libasan.so).

Before this change, both new specs report heap-use-after-free:

ERROR: AddressSanitizer: heap-use-after-free
    #0 ZSTD_CCtx_init_compressStream2  compress/zstd_compress.c:6363
    #1 ZSTD_compressStream2            compress/zstd_compress.c:6480
freed by thread T0 here:
    #3 ZSTD_freeCDict                  compress/zstd_compress.c:5737
    #4 rb_data_free                    gc.c:1213
    #5 rb_gc_obj_free                  gc.c:1359

ERROR: AddressSanitizer: heap-use-after-free
    #0 ZSTD_DDict_dictContent          decompress/zstd_ddict.c:49
    #1 ZSTD_decompressBegin_usingDDict decompress/zstd_decompress.c:1606
    #2 ZSTD_decompressStream           decompress/zstd_decompress.c:2214
freed by thread T0 here:
    #2 ZSTD_freeDDict                  decompress/zstd_ddict.c:217

In both traces the free comes from the GC's own free callback, so the dictionary is being collected exactly as expected — it simply has no owner. After this change the full suite runs clean under ASan, with zero AddressSanitizer reports.

Scope

Only the streaming classes are affected. The one-shot Zstd.compress and Zstd.decompress paths borrow a dictionary too, but it stays reachable from the caller's frame for the whole call — including across rb_thread_call_without_gvl, since the GC scans the machine stack of every thread — so they need no ownership and are left alone.

A String dictionary needs no handling on either path: ZSTD_CCtx_loadDictionary and ZSTD_DCtx_loadDictionary pass ZSTD_dlm_byCopy, so libzstd owns its own copy.

set_compress_params / set_decompress_params now return the dictionary object so the streaming initializers can retain it. Their existing callers ignore the return value and are otherwise unchanged.

Why the existing specs did not catch this

The dictionary specs — including the ones that already call GC.compact — hold the dictionary in a let, which keeps it reachable for the whole example. The two new specs deliberately keep no reference to it, then run GC.start / GC.compact and allocate into the freed slot before using the stream.

Compatibility

No API or behavior change. The dictionary object is now reachable from the stream, so it is collected after the stream instead of possibly before it.

🤖 Generated with Claude Code