feat(mem_wal): return the sealed generation from force_seal_active

[hamersaw]

Why

wait_for_flush_drain loops until the frozen-watcher set is empty, re-collecting the set on each round — so it also waits on every memtable frozen while it waits, including ones the size/interval trigger and backpressure freeze concurrently. On a table under sustained write load that set may never empty, which makes any post-seal ack unbounded on exactly the workload that asks for one.

The contract a caller actually needs is narrower: everything written before my seal is in L0. That is a predicate on one generation, not on the queue.

What

force_seal_active now returns the SealedGeneration it froze — None when the active memtable was empty (a no-op seal). A caller waits on the shard manifest's current_generation exceeding it: bounded by construction, unaffected by concurrent freezes, and it makes the ack literally true instead of incidentally true.

Also adds IndexStore::describe, reporting the in-memory index set (kind + column) for diagnostics. An absent HNSW entry on a vector column is the whole explanation for a brute-force fresh-tier search, and there was no way to see it from outside the store.

Compatibility

force_seal_active's return type changes from Result<()> to Result<Option<SealedGeneration>>. Existing .await?; call sites compile unchanged (the value is simply dropped); only code that binds or asserts on the unit value needs touching. No format or on-disk change.

Consumer

Sophon's WAL uses this to make POST /v1/table/{name}/flush bounded — it currently calls wait_for_flush_drain and inherits the unbounded-latency problem above.

Notes for review

No new tests here: the change is a return value, and the bounded-wait semantics it enables are exercised on the consumer side (a test that seals, then keeps writing hard enough that the frozen set never empties, and asserts flush returns once its own generation lands — against wait_for_flush_drain that test hangs, which is the point of it). Happy to add a mem_wal test pinning that the returned generation matches the frozen memtable's if you want it covered here too.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 2.77778% with 35 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/mem_wal/index.rs	0.00%	35 Missing ⚠️

📢 Thoughts on this report? Let us know!