fix(fts): validate complete parameters across segments

[BubbleCal]

What is the bug?

Segmented FTS query planning compared InvertedIndexDetails from each index manifest. That representation is intentionally lossy and omits tokenizer state such as custom_stop_words, so segments with different persisted tokenizer configurations could be combined and queried with inconsistent semantics.

What incorrect behavior does the bug cause?

A segmented index could silently collapse distinct states such as omitted custom stop words, an explicit empty list, and a non-empty replacement list. A first implementation fixed correctness by reading every segment's metadata before execution, but that bypassed the scalar-index cache and added two I/O operations to both cold and fully prewarmed queries.

How does this PR fix the problem?

• Keep manifest InvertedIndexDetails validation in planning for capability checks such as positions.
• Open every selected FTS segment through the existing scalar-index cache before choosing a tokenizer.
• Compare the complete persisted InvertedIndexParams from the opened indexes, including custom_stop_words.
• Fail closed with both segment IDs when any complete parameter set differs.
• Cover Match, FlatMatch, Phrase, and FlatMatchFilter paths through the same validation boundary.
• Preserve shallow-clone base_id behavior through the public index-store path.
• Add actual multi-segment query coverage for None versus [], [] versus non-empty lists, legacy details, and shallow clones.

Validation

• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings
• Prewarmed FTS query I/O regression: 1 passed, 0 reads after prewarm
• Cold inverted-index I/O regression: 1 passed, below the existing 18-read bound
• cargo test -p lance segment_params -- --nocapture (4 passed)
• cargo test -p lance canonicalize_inverted_details_accepts_legacy_empty_details -- --nocapture (1 passed)
• cargo test -p lance-index params_legacy -- --nocapture (2 passed)
• FlatMatchFilter persisted-tokenizer regression: 1 passed

Coordination

This is the canonical latest-main fix. Sophon currently pins Lance v10.0.0-beta.5, so its parent PR uses the same patch backported onto that exact pin instead of upgrading the full Lance dependency set.

• LanceDB SDK/source contract: feat(fts): support custom stop-word lists lancedb/lancedb#3734
• Sophon end-to-end support: https://github.com/lancedb/sophon/pull/6932

[codecov]

Codecov Report

❌ Patch coverage is 56.75676% with 16 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/io/exec/fts.rs	56.75%	13 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!