fix(mem-wal): resolve writer shard spec identity

[u70b3]

Summary

• resolve the default dataset-writer shard_spec_id (0) to the MemWAL index's sole automatic sharding spec before creating or claiming a shard manifest
• accept an explicit id only when it matches that sole spec, and require manually sharded indexes to keep id 0
• reject metadata with multiple automatic specs because the current writer model supports at most one
• cover the original fresh-shard bug at the public Dataset::mem_wal_writer boundary plus invalid manual, mismatched, and multiple-spec inputs

Scope

This PR fixes the current writer path only. MemWAL initialization already creates either no spec (manual sharding) or one automatic spec with id 1; the writer must record that same identity instead of its unresolved default 0.

It does not add monotonic allocation, spec revisions, activation semantics, metadata validation for hypothetical producers, or a file-format change. Those require a separate end-to-end design if a re-shard API is introduced.

This PR complements but is not a prerequisite for #7949: this PR resolves the table identity before a fresh manifest is created, while #7949 defensively rejects changing an identity already stored in a manifest.

Testing

• cargo fmt --all -- --check
• cargo test -p lance --lib test_writer_shard_spec_resolution_rejects_invalid_identity -- --nocapture
• cargo test -p lance --lib test_mem_wal_writer_uses_automatic_sharding_spec -- --nocapture
• cargo clippy --all --tests --benches -- -D warnings

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[lance-gatekeeper]

Gate recommendation: maintainer decision required.

The fresh-shard writer fix is sound, but the PMC must choose whether the format should make activity implicit in the maximum append-only ID (simple, but commits future cutover and rollback semantics) or persist an explicit active ID (more metadata, but independent activation). This substantive format change requires the documented three binding +1 votes and one-week period; no completed vote is linked. Please link that vote before accepting the contract.

[lance-gatekeeper]

Gate recommendation: maintainer decision required.

The fresh-shard writer fix is sound, but the PMC must choose whether the format should make activity implicit in the maximum append-only ID (simple, but commits future cutover and rollback semantics) or persist an explicit active ID (more metadata, but independent activation). This substantive format change requires the documented three binding +1 votes and one-week period; no completed vote is linked. Please link that vote before accepting the contract.

[codecov]

Codecov Report

❌ Patch coverage is 97.74194% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-table/src/system_index/mem_wal.rs	96.70%	0 Missing and 3 partials ⚠️
rust/lance/src/dataset/mem_wal/write.rs	98.49%	0 Missing and 2 partials ⚠️
rust/lance/src/dataset/mem_wal/api.rs	98.03%	0 Missing and 1 partial ⚠️
rust/lance/src/index/mem_wal.rs	97.14%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!