HDDS-15857. Fix checksum calculation in KeyValueHandler for variable-sized chunks by chungen0126 · Pull Request #10764 · apache/ozone

chungen0126 · 2026-07-14T14:59:12Z

What changes were proposed in this pull request?

Problem

Currently, the getChecksums method in KeyValueHandler.java fails or returns incorrect checksums when chunks have variable sizes or are smaller than bytesPerChunk.

The original code assumes all chunks (except the last one) have a fixed size equal to bytesPerChunk. This assumption is incorrect when chunks have different lengths, leading to wrong index calculations.

Crucially, when checksum verification is enabled, the DataNode must return the actual ChunkInfo (which contains chunk lengths) to the client. Without this metadata, the client cannot perceive the boundaries of each varying chunk, making it impossible to correctly map the offsets and compute/verify the checksums.

Solution

This PR resolves the issue by tracking exact chunk boundaries on both the server and client sides:

Server-Side (KeyValueHandler & Protocol Updates)

Correct Indexing in getChecksums: Rewrote the checksum slice calculation in KeyValueHandler#getChecksums. Instead of calculating chunk indexes using a fixed bytesPerChunk division, we now iterate over chunks, keep track of the cumulative offset (currentChunkOffset), and determine overlap intervals between the read range and each chunk. This ensures we correctly map and collect the target checksums.
Send Chunk Info: Modified ContainerCommandResponseBuilders#getReadBlockResponse and KeyValueHandler#readBlockImpl to populate and return the chunkInfoList to the client when performing a block read.

Client-Side (StreamBlockInputStream Checksum Verification)
Boundary-Aware Verification: In StreamBlockInputStream#onNext, if the response contains chunkInfoList and checksum verification is enabled:

Iterate through each chunk to identify the exact overlapping region between the read block and the chunk boundaries.
Calculate correct start and end checksum indexes using bytesPerChecksum relative to that specific chunk's start.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15857

How was this patch tested?

Added testGetChecksumsWithVaryingChunkSizes and testGetChecksumsWithSmallChunks in TestKeyValueHandler.java.
Added testSmallChunksWithLargeChecksum in TestStreamRead.java.

CI: https://github.com/chungen0126/ozone/actions/runs/29381590591

…sized chunks

chungen0126 · 2026-07-15T01:56:13Z

Hi @szetszwo , @sodonnel ,During my review of the current Stream Read architecture, I identified a checksum calculation issue that affects both the client and server sides.This issue occurs when a client actively triggers a flush or hsync. In such cases, the resulting chunk size does not perfectly align with bytesPerChunk, even though the checksum for this chunk has already been generated. Consequently, we can no longer simply rely on the fixed bytesPerChunk length to determine which chunk corresponds to the checksum currently being calculated.Here is a breakdown of the issues on both sides:

Client-side Issue: Currently, the response returned by the DataNode does not contain any chunk-related information. As a result, the client cannot correctly partition the retrieved data to verify the checksums.
Server-side Issue: The server cannot return the correct set of checksums. For example, if bytesPerChecksum is 1024, and we have chunk1 (data: a) and chunk2 (data: b). Under the current logic, the response will only return the checksum data for chunk1.

To properly verify checksums on the client side, I believe it is essential to return chunkInfo. If we go down this path, we should refine the structure of ReadBlockResponseProto, as the top-level checksumData might become redundant.

Since modifying Protobuf definitions after a major release is highly problematic due to backward compatibility/wire-protocol locking, this issue could potentially block or impact the upcoming Ozone 2.2 release. I'd love to hear your thoughts on this.

Copilot

Pull request overview

This PR fixes streaming-read checksum handling when block chunks are not uniform (variable-sized or smaller than bytesPerChunk) by making checksum selection and verification aware of real chunk boundaries, and by extending the read-block protocol to optionally return chunk metadata needed for correct verification.

Changes:

Reworked server-side KeyValueHandler#getChecksums to select checksum slices based on chunk overlap rather than fixed bytesPerChunk indexing.
Extended ReadBlockResponseProto to optionally include chunkInfoList, and wired server/client to use chunk boundaries during checksum verification.
Added unit and integration tests covering varying/small chunk sizes with larger bytesPerChecksum.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/read/TestStreamRead.java	Adds integration coverage for small chunks with larger checksums during streaming read.
hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto	Adds optional `chunkInfoList` to `ReadBlockResponseProto` for boundary-aware verification.
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandler.java	Adds unit tests validating checksum slicing with varying/small chunk sizes.
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java	Updates checksum slicing logic and read-block response building to support variable chunk sizes.
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ContainerCommandResponseBuilders.java	Populates chunk info into read-block responses (needs conditional sending for perf/compat).
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/storage/TestStreamBlockInputStream.java	Updates test response protos to include chunk info for checksum verification paths.
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/StreamBlockInputStream.java	Adds boundary-aware checksum verification using `chunkInfoList`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

        if (verifyChecksum) {
          ChecksumData checksumData = ChecksumData.getFromProtoBuf(readBlock.getChecksumData());
-          Checksum.verifyChecksum(data, checksumData, 0);
+          if (readBlock.hasChunkInfoList()) {
+            verifyChecksumForReadBlock(data, checksumData, readBlock);
+          } else {
+            throw new IOException("Checksum data is missing for block " + getBlockID());
+          }


+        int bytesPerChecksum = checksumData.getBytesPerChecksum();
+        long blockOffset = readBlock.getOffset();
+        long readLength = data.remaining();
+        long currentChunkOffset = 0;
+        int checksumIndex = 0;
+        int dataOffset = 0;
+
+        for (ContainerProtos.ChunkInfo chunk : readBlock.getChunkInfoList().getChunksList()) {
+          long chunkStart = currentChunkOffset;
+          long chunkEnd = chunkStart + chunk.getLen();
+


+    long currentChunkOffset = 0;
+    for (ContainerProtos.ChunkInfo chunk : chunks) {
+      long chunkStart = currentChunkOffset;
+      long chunkEnd = chunkStart + chunk.getLen();
+
+      long overlapStart = Math.max(blockOffset, chunkStart);
+      long overlapEnd = Math.min(blockOffset + readLength, chunkEnd);
+


  public static ContainerCommandResponseProto getReadBlockResponse(
-      ContainerCommandRequestProto request, ChecksumData checksumData, ByteBuffer data, long offset) {
+      ContainerCommandRequestProto request, ChecksumData checksumData,
+      ByteBuffer data, long offset, List<ChunkInfo> chunkInfoList, boolean verifyChecksum) {

-    ContainerProtos.ReadBlockResponseProto response = ContainerProtos.ReadBlockResponseProto.newBuilder()
+    ContainerProtos.ReadBlockResponseProto response = ReadBlockResponseProto.newBuilder()
        .setChecksumData(checksumData.getProtoBufMessage())
        .setData(ByteString.copyFrom(data))
        .setOffset(offset)
+        .setChunkInfoList(ChunkInfoList.newBuilder().addAllChunks(chunkInfoList).build())
        .build();


+      OzoneConfiguration conf = cluster.getConf();
+      OzoneClientConfig clientConfig = conf.getObject(OzoneClientConfig.class);
+      clientConfig.setStreamReadBlock(true);
+      clientConfig.setStreamBufferFlushDelay(false);
+      final OzoneConfiguration steamReadConf = new OzoneConfiguration(conf);
+      steamReadConf.setFromObject(clientConfig);
+
+      try (OzoneClient streamReadClient = OzoneClientFactory.getRpcClient(steamReadConf)) {


jojochuang · 2026-07-15T05:45:00Z

Bugbot review

Stream checksum verify misaligns chunks — hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/StreamBlockInputStream.java:638-677 · Severity: high

With checksumVerify enabled (the default), verifyChecksumForReadBlock verifies each chunk overlap without expanding to per-chunk checksum boundaries or indexing into each chunk's stored checksum list. After a prefix of non-checksum-aligned chunks, block-offset alignment can still land inside a chunk mid-checksum-unit, so verification compares freshly computed checksums against the wrong stored entries and stream reads fail with OzoneChecksumException.

Generated-by: Cursor Bugbot

chungen0126 added 10 commits July 14, 2026 21:05

fix server side bug

6f964d6

HDDS-15857. Fix checksum calculation in KeyValueHandler for variable-…

a65be93

…sized chunks

update test

3779fad

Merge branch 'master' into HDDS-15857

8202225

throw exception if checksum data is missing

e9b3183

fix checkstyle

e0ae947

fix findbug

ec1d56b

fix checkstyle

86cf0f3

fix bugs

445579d

revert TestOzoneContractFSO

4745196

chungen0126 marked this pull request as ready for review July 15, 2026 01:56

chungen0126 requested review from sodonnel and szetszwo July 15, 2026 01:56

jojochuang requested a review from Copilot July 15, 2026 05:10

Copilot started reviewing on behalf of jojochuang July 15, 2026 05:11 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-15857. Fix checksum calculation in KeyValueHandler for variable-sized chunks#10764

HDDS-15857. Fix checksum calculation in KeyValueHandler for variable-sized chunks#10764
chungen0126 wants to merge 10 commits into
apache:masterfrom
chungen0126:HDDS-15857

chungen0126 commented Jul 14, 2026 •

edited

Loading

Uh oh!

chungen0126 commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

jojochuang commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

chungen0126 commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Problem

Solution

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

chungen0126 commented Jul 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

jojochuang commented Jul 15, 2026

Bugbot review

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chungen0126 commented Jul 14, 2026 •

edited

Loading