CASSANDRA-21134: Direct I/O for background SSTable writes

[samueldlightfoot]

CASSANDRA-21134: Direct I/O for background SSTable writes

Summary

Opt-in O_DIRECT write path for background SSTable producers, bypassing the OS page cache for write-once read-never data. Memtable flushes remain buffered (hot data benefits from the cache).

background_write_disk_access_mode: direct    # default: standard
direct_write_buffer_size: 1MiB                # aligned up to FS block size; auto-grows to chunk_length

Gated by (1) config, (2) table compression enabled, (3) an OperationType allowlist (DataComponent#DIRECT_WRITE_SUPPORT). Selection is central in DataComponent.buildWriter; producers are unchanged.

Performance

Benchmark results are attached to the JIRA. Significant p99 read latency improvements under throttled compaction.

Operations covered (DIO eligible)

OperationType	End-to-end test
WRITE	CQLSSTableWriterDaemonTest (parameterised on disk mode)
COMPACTION	CompactionsTest (parameterised on disk mode)
MAJOR_COMPACTION	CompactionsTest.testCompactionWithSizeLimitedRewriter
CLEANUP, GARBAGE_COLLECT, TOMBSTONE_COMPACTION, UPGRADE_SSTABLES	CompactionsTest (transitive)
ANTICOMPACTION	AntiCompactionTest.testAntiCompactionWithCompressedTableAndDirectWrites
STREAM	StreamingDirectWriteTest

The allowlist is exhaustive: any new OperationType with writesData == true that is not classified fails static initialization (AssertionError).

Operations NOT covered

Path	Classification	Reason	Coverage
FLUSH (memtable)	UNSUPPORTED_POLICY	Just-flushed data is hot — keep it in the page cache.	DataComponentDirectWriteSelectionTest
SCRUB	UNSUPPORTED_CORRECTNESS	tryAppend needs mark() / resetAndTruncate(), which DIO cannot satisfy.	DataComponentDirectWriteSelectionTest
Zero-Copy Streaming	n/a (path bypass)	Entire-SSTable streaming bypasses DataComponent.buildWriter.	StreamingDirectWriteTest (disables ZCS)
Uncompressed writers	n/a (path bypass)	Only CompressedSequentialWriter has a DIO subclass.	DataComponentDirectWriteSelectionTest (compression gate)

Removing an UNSUPPORTED_CORRECTNESS entry requires code changes; UNSUPPORTED_POLICY is a policy decision.

Key code

• io/DirectIoSupport.java — eligibility enum (SUPPORTED / UNSUPPORTED_CORRECTNESS /
  UNSUPPORTED_POLICY / NOT_APPLICABLE).
• io/sstable/format/DataComponent.java — selection, allowlist, exhaustiveness check;
  per-op first-activation log.
• io/compress/DirectCompressedSequentialWriter.java — new writer; aligned buffers,
  mark()/resetAndTruncate() unsupported.
• io/compress/CompressedSequentialWriter.java — refactored so the DIO subclass can
  override the write-chunk path; writeChunk contract documented and asserted.
• config/Config.java, config/DatabaseDescriptor.java — new knobs, validation, startup
  wiring; buffer size aligned to FS block size, auto-grown to chunk length.
• service/StartupChecks.java — fails fast if direct is requested on a platform/FS
  that does not support O_DIRECT.

Tests introduced

• Property-based DIO-writer sweep — write/read integrity and on-disk byte-identity
  vs. the buffered writer over compressors × chunk lengths × random payload sizes;
  seed-logged for repro (DirectCompressedSequentialWriterTest).
• Parameterised buffer-size tests — three regimes pinning the distinct branches of
  flushCompleteBlocks (DirectCompressedSequentialWriterTest).
• Selection-matrix tests — per-OperationType eligibility, allowlist exhaustiveness,
  compression gate, config-mode gate (DataComponentDirectWriteSelectionTest).
• End-to-end coverage per allowlist arm — WRITE (CQLSSTableWriterDirectWriteTest),
  STREAM (in-JVM dtest, StreamingDirectWriteTest), and the compaction family +
  ANTICOMPACTION (extended CompactionsTest, AntiCompactionTest).
• Regression guards — constructor channel-leak protection, non-power-of-two
  block-size rejection, once-per-JVM undersized-buffer warn, SCRUB-gating canaries
  (DirectCompressedSequentialWriterTest).
• Resource-leak detection — BufferPoolMXBean check that the off-heap aligned
  buffer is returned on close (DirectCompressedSequentialWriterTest).
• Config validation — new YAML knobs (mode parsing, buffer-size bounds, defaults)
  in DatabaseDescriptorTest.

Not in scope

• Uncompressed SSTable writers.
• ZCS streaming.

Reviewer notes

Findings from the Cassandra bug-hunting skills (Opus 4.7 xhigh & kimi-k2.6:cloud) were addressed prior to
review.

[aweisberg]

Mostly just nits but at least one or two things worth doing and then I shared a pastebin with you of an analysis of the DirectCompressedSequentialWriterTest code and boundary coverage that is worth considering.

[aweisberg]

Technically haven't exercised the checks of the paths of isDirectIOSupported which would require being able to intercept that and provide test answers to whether paths support that. Maybe something that can be done with jimfs.

The auto path which we discussed might be worth changing.

[samueldlightfoot]

I think the best we could do is a custom hook that captures what open options were requested, but not via JimFS, since getBlockSize throws an exception. Let me know how strongly you feel on this one.

My plan with auto was to have it resolve to standard now, and then if/when we decide to use direct as the default, we can have it check DIO support and fallback to standard if not.

[aweisberg]

Curious why Streaming needed a dedicated test. Incoming buffers is just incoming buffers right? Not complaining about the coverage though.

[samueldlightfoot]

It doesn't increase coverage; the logic is already tested. I tried to cover all operations with a test for said op - happy to drop the test if you don't think worthwhile.