Skip to content

[fix](clucene) Support block WAND in composite readers#402

Open
hoshinojyunn wants to merge 1 commit into
apache:clucenefrom
hoshinojyunn:fix/multi-reader-block-wand
Open

[fix](clucene) Support block WAND in composite readers#402
hoshinojyunn wants to merge 1 commit into
apache:clucenefrom
hoshinojyunn:fix/multi-reader-block-wand

Conversation

@hoshinojyunn

@hoshinojyunn hoshinojyunn commented Jul 15, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #62955

Related PR: None

Problem Summary:

The fix/multi-reader-block-wand branch adds block-WAND support for composite
readers (MultiReader / MultiTermDocs). The block APIs previously worked only
for a segment reader, so a caller using a composite reader could not read block
postings or use block-level statistics.

This change:

  • Implements readBlock, skipToBlock, getMaxBlockFreq, getMaxBlockNorm,
    and getLastDocInBlock in MultiTermDocs.

  • Converts child-reader-local doc IDs and block boundaries into global doc IDs,
    while preserving each physical block, including a short tail block, rather
    than merging blocks across child-reader boundaries.

  • Reports a reader transition from skipToBlock even when the destination
    child has fewer postings than the skip interval and therefore cannot perform
    a local skip. This invalidates the block-WAND caller's cached block metadata.

  • Fixes the SegmentTermDocs::skipToBlock posting count after a skip-list
    lookup in the first physical block:

    count = newCount < 0 ? 0 : newCount;

    DefaultSkipListReader::skipTo() returns -1 when no skip entry precedes
    the target, which means the target is in the first physical block and the
    frequency stream is positioned at its beginning. At that point no posting
    has been consumed, so count must be 0, not -1.

    For example, consider 1,149 postings stored as blocks of 511, 512, and 126
    postings. A skipToBlock() target in the first block returns newCount == -1. If count remains -1, the subsequent block reads make it 510, 1,022,
    and 1,148, respectively. The reader then believes one posting remains after
    the 126-posting tail and tries to read beyond the physical postings. Starting
    from count = 0 instead yields 511, 1,023, and 1,149, so iteration stops at
    the correct boundary.

Regression coverage in TestReadRange.cpp verifies:

  • Composite readBlock() output, global doc IDs, and block metadata against
    next() for two readers with 1,149 and 1,134 documents, including expected
    511/512/tail physical block layouts.
  • skipToBlock() before and across reader boundaries, including a target in a
    short tail block.
  • Reader transitions into a child with only 11 postings, both from a short
    first reader and after a block skip in a larger first reader.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test: be/ut_build_ASAN/bin/cl_test testReadRange
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. Composite readers now expose block-WAND APIs and report
      cross-reader block state transitions.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: close #62955

Related PR: None

Problem Summary: Implement block postings APIs in MultiTermDocs so composite readers globalize physical block docids, expose their block statistics, and cross child-reader boundaries. A cross-reader skip must invalidate the caller block cache even when the new child has fewer postings than the skip interval and reports no local block skip. Preserve short physical tail blocks without merging them across readers. Add composite-reader tests for 511/512/tail block layouts and transitions into short readers.

### Release note

None

### Check List (For Author)

- Test: Unit Test (be/ut_build_ASAN/bin/cl_test testReadRange)
- Behavior changed: Yes (composite readers report cross-reader block state changes)
- Does this need documentation: No
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant