Skip to content

feat(io): optional pure-Rust parallel gzip input decoding (rapidgzip-core) - #225

Open
BenjaminDEMAILLE wants to merge 2 commits into
scverse:mainfrom
BenjaminDEMAILLE:feat/rapidgzip-parallel-decode
Open

feat(io): optional pure-Rust parallel gzip input decoding (rapidgzip-core)#225
BenjaminDEMAILLE wants to merge 2 commits into
scverse:mainfrom
BenjaminDEMAILLE:feat/rapidgzip-parallel-decode

Conversation

@BenjaminDEMAILLE

@BenjaminDEMAILLE BenjaminDEMAILLE commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Refs #224 (dependency discussion), #223 (the thread-budget measurements).

Adds rapidgzip-core 0.3 (BSD-3-Clause AND MIT,
COMBINE-lab) behind a cargo feature, for parallel decoding of gzipped --readFilesIn.

Stacked on #219. The first commit here is that PR's multi-member gzip fix; review only the
second. Rebases cleanly once #219 lands.

Correction to my earlier comment on #223. I first evaluated the crate literally named
rapidgzip on crates.io, which is an FFI wrapper whose -sys crate vendors ~20 MB of C++ and
builds it with CMake. That is a different project from the pure-Rust rapidgzip-core used here.
Everything I said about a C++/CMake build requirement applies to that crate, not this one.

Why this crate is cheap to take

Pure Rust. Its dependencies are bon, crossbeam-deque and libz-rs-sys, and that last one is
zlib-rs, the same inflate backend flate2 already uses in this project. No C toolchain, no
CMake, nothing new on the Windows path.

It implements the rapidgzip marker/window algorithm: DEFLATE blocks are decoded speculatively
before their back-references are known, then patched once they are.

Decoder throughput

73 MB level-6 .fq.gz (427 MB out), 16 logical cores, median of three:

decoder throughput
flate2 + zlib-rs, 1 thread 1552 MB/s
rapidgzip-core, 1 thread 1581 MB/s
rapidgzip-core, 4 threads 1872 MB/s
rapidgzip-core, 8 threads 3356 MB/s
rapidgzip-core, 12 threads 4051 MB/s

No penalty at one thread, and 2.2x a single flate2 thread at eight.

Why it is still off by default

Arithmetic, not doubt about the decoder. Aligning 2 M reads takes ~3.0 s at --runThreadN 8, so we
consume decompressed FASTQ at ~140 MB/s while one flate2 thread already supplies
~1550 MB/s. Decode is running at under a tenth of its capacity.

Measured end to end, 20 Mb genome, 2 M reads, --runThreadN 8:

decode wall peak RSS
flate2 3.03s 1305 MB
rapidgzip-core, auto (2 workers) 3.15s 1667 MB

Slightly slower and ~360 MB more resident, which is what spending threads on an already-idle stage
looks like. So: --features rapidgzip to compile it in, RUSTAR_GZ_DECODE_THREADS=N to actually
engage it (0 keeps flate2), matching the existing RUSTAR_* knobs rather than adding a
non-STAR CLI flag.

The feature becomes the right call when the consumer can drain a single inflate thread, roughly
7 M reads/s for this file shape. That is the regime piscem/salmon are in, which is exactly why
they pair a parallel decoder with a broker that splits one thread budget by measured busy time. For
this aligner the same measurement says the bottleneck is elsewhere: the serial writer stage is
~34% of wall (#223).

Implementation

  • Decoder::open owns the file and returns a Read + Send stream, dropping straight into the
    Box<dyn BufRead + Send> the parser wants. Reaching EOF verifies every member footer, so error
    behaviour on a corrupt file is not weakened relative to flate2.
  • Any failure to configure or open through it logs a warning and falls back to flate2.
  • Worker count is process-wide state set once in run(); FastqReader::open is called from a dozen
    places that have no business each deciding a decompression policy.

Correctness

  • test_rapidgzip_decodes_identically_to_flate2: same two-member gzip read down both paths, decoded
    records compared. Covers the member boundary that fix(io): decode multi-member gzip input instead of truncating it #219 fixes.
  • End to end: BAM payload byte-identical with the feature on and off.
  • Gate green both ways: cargo fmt --check, clippy -D warnings with and without the feature,
    cargo test 631 / cargo test --features rapidgzip 632, 0 failed.

If you would rather not carry the dependency at all, --readFilesCommand covers the input-bound
case with no dependency and I will close this.

BenjaminDEMAILLE and others added 2 commits August 7, 2026 18:36
`flate2::read::GzDecoder` stops at the end of the first gzip member and
reports EOF. A `.gz` written as several concatenated members was
therefore read partially, with no error and no warning: bcl2fastq
output, `cat a.fq.gz b.fq.gz > merged.fq.gz`, and every BGZF file are
all multi-member.

Measured on the bundled fixtures before the fix: a two-member
`test/reads.fq` gzip (3 reads, 12 lines, identical to the single-member
file under `gunzip`) reported `Number of input reads | 2`.

Switches the four read paths to `MultiGzDecoder`: FASTQ input, the solo
barcode whitelist, solo counting, and the `emptydrops` binary.

Adds `test_fastq_reader_gzip_multi_member`, which builds a two-member
gzip and asserts both reads come back. Verified it fails on the old
decoder and passes on the new one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `rapidgzip-core` 0.3 (BSD-3-Clause AND MIT) behind a cargo feature,
for parallel decoding of gzipped `--readFilesIn`.

It is pure Rust. Its dependencies are `bon`, `crossbeam-deque` and
`libz-rs-sys`, and that last one is zlib-rs, the same inflate backend
`flate2` already uses in this project, so the feature adds no C or C++
toolchain requirement. It implements the rapidgzip marker/window
algorithm: DEFLATE blocks are decoded speculatively before their
back-references are known, then patched once they are.

Decoding a 73 MB level-6 .fq.gz (427 MB out) on 16 logical cores,
median of three:

    decoder                        throughput
    flate2 + zlib-rs, 1 thread     1552 MB/s
    rapidgzip-core, 1 thread       1581 MB/s
    rapidgzip-core, 4 threads      1872 MB/s
    rapidgzip-core, 8 threads      3356 MB/s
    rapidgzip-core, 12 threads     4051 MB/s

Note there is no penalty at one thread, unlike an FFI decoder.

It is nonetheless off by default, and inert until
RUSTAR_GZ_DECODE_THREADS is set, for a reason that is arithmetic rather
than doubt about the decoder: aligning 2 M reads takes ~3.0 s at
--runThreadN 8, so we consume decompressed FASTQ at ~140 MB/s while one
flate2 thread supplies ~1550 MB/s. Decode runs at under a tenth of its
capacity. Turning it on measured 3.15 s against 3.03 s and ~360 MB more
resident, which is what spending threads on an idle stage looks like.
The feature becomes the right call when the consumer can drain a single
inflate thread, roughly 7 M reads/s for this file shape.

Implementation notes:

* `Decoder::open` owns the file and returns a `Read + Send` stream, so it
  drops straight into the `Box<dyn BufRead + Send>` the parser wants.
  Reaching EOF verifies every member footer, so error behaviour on a
  corrupt file is not weakened relative to flate2.
* Any failure to configure or open through it logs a warning and falls
  back to flate2, so this can never turn a working run into a failing one.
* The worker count is process-wide state set once from `run()`, rather
  than a parameter, because `FastqReader::open` is called from a dozen
  places that have no business each deciding a decompression policy.

`test_rapidgzip_decodes_identically_to_flate2` reads the same two-member
gzip down both paths and asserts the decoded records match. End to end,
the BAM payload is byte-identical with the feature on and off.

Refs scverse#224, scverse#223.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@BenjaminDEMAILLE
BenjaminDEMAILLE force-pushed the feat/rapidgzip-parallel-decode branch from 40617e2 to eaa12ab Compare August 11, 2026 19:04
@BenjaminDEMAILLE BenjaminDEMAILLE changed the title feat(io): optional rapidgzip feature for parallel gzip input decoding feat(io): optional pure-Rust parallel gzip input decoding (rapidgzip-core) Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant