[fix](clucene) Support block WAND in composite readers#402
Open
hoshinojyunn wants to merge 1 commit into
Open
Conversation
### What problem does this PR solve? Issue Number: close #62955 Related PR: None Problem Summary: Implement block postings APIs in MultiTermDocs so composite readers globalize physical block docids, expose their block statistics, and cross child-reader boundaries. A cross-reader skip must invalidate the caller block cache even when the new child has fewer postings than the skip interval and reports no local block skip. Preserve short physical tail blocks without merging them across readers. Add composite-reader tests for 511/512/tail block layouts and transitions into short readers. ### Release note None ### Check List (For Author) - Test: Unit Test (be/ut_build_ASAN/bin/cl_test testReadRange) - Behavior changed: Yes (composite readers report cross-reader block state changes) - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #62955
Related PR: None
Problem Summary:
The
fix/multi-reader-block-wandbranch adds block-WAND support for compositereaders (
MultiReader/MultiTermDocs). The block APIs previously worked onlyfor a segment reader, so a caller using a composite reader could not read block
postings or use block-level statistics.
This change:
Implements
readBlock,skipToBlock,getMaxBlockFreq,getMaxBlockNorm,and
getLastDocInBlockinMultiTermDocs.Converts child-reader-local doc IDs and block boundaries into global doc IDs,
while preserving each physical block, including a short tail block, rather
than merging blocks across child-reader boundaries.
Reports a reader transition from
skipToBlockeven when the destinationchild has fewer postings than the skip interval and therefore cannot perform
a local skip. This invalidates the block-WAND caller's cached block metadata.
Fixes the
SegmentTermDocs::skipToBlockposting count after a skip-listlookup in the first physical block:
DefaultSkipListReader::skipTo()returns-1when no skip entry precedesthe target, which means the target is in the first physical block and the
frequency stream is positioned at its beginning. At that point no posting
has been consumed, so
countmust be0, not-1.For example, consider 1,149 postings stored as blocks of 511, 512, and 126
postings. A
skipToBlock()target in the first block returnsnewCount == -1. Ifcountremains-1, the subsequent block reads make it 510, 1,022,and 1,148, respectively. The reader then believes one posting remains after
the 126-posting tail and tries to read beyond the physical postings. Starting
from
count = 0instead yields 511, 1,023, and 1,149, so iteration stops atthe correct boundary.
Regression coverage in
TestReadRange.cppverifies:readBlock()output, global doc IDs, and block metadata againstnext()for two readers with 1,149 and 1,134 documents, including expected511/512/tail physical block layouts.
skipToBlock()before and across reader boundaries, including a target in ashort tail block.
first reader and after a block skip in a larger first reader.
Release note
None
Check List (For Author)
Test
be/ut_build_ASAN/bin/cl_test testReadRangeBehavior changed:
cross-reader block state transitions.
Does this need documentation?
Check List (For Reviewer who merge this PR)