fix(index): invalidate stale index cache entries after index replacement

[geserdugarov]

What

CreateIndex commits can retire existing index metadata, for example when optimize_indices(...retrain...) replaces an IVF index with a new UUID. If the old index had been prewarmed, the session index cache could still hold root entries and IVF partition entries for the removed UUID.

A later read in the same session could then observe stale cached index state that no longer matches the manifest. This clears cache entries for removed index UUIDs after the commit succeeds.

Changes

• Add exact-key invalidation to LanceCache via invalidate_with_key and invalidate_unsized_with_key.
• Extend CacheBackend with invalidate_entry and implement it for the Moka backend and existing serializing test backends.
• On successful Operation::CreateIndex commits, invalidate cache entries for each removed index:
  • legacy vector index root entry
  • IVF state root entry
  • IVF partition sub-cache under uuid/
  • corresponding uuid-fri_uuid/ entries when a fragment-reuse index is active
• Use the rebased transaction’s removed_indices, so conflict resolution invalidates the indices actually retired by the committed manifest.

Notes

• Exact-key invalidation is needed because root index cache entries are keyed by UUID, while partition entries live under a cache prefix. Prefix invalidation alone cannot remove one root entry without risking unrelated cache entries.
• Invalidation happens only after the manifest write succeeds.
• This does not change index metadata, params, protos, or Python / Java APIs.

Tests

• Cache unit tests for exact-key invalidation, sibling key/type preservation, prefix scoping, nested sub-cache preservation, missing-key no-op behavior, and unsized cache keys.
• An index integration test that prewarms an IVF-PQ index, retrains it through optimize_indices, and verifies the removed index UUID no longer has cached root or partition entries.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[geserdugarov]

@Xuanwo, @wjones127, hi!
If you don't mind, could you please review this PR related to index cache entries?

[codecov]

Codecov Report

❌ Patch coverage is 98.41897% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-core/src/cache/mod.rs	98.48%	2 Missing ⚠️
rust/lance/src/index/vector/ivf/v2.rs	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!