feat(rollout): count-triggered self-merge of MemWAL shards into base table

[beinan]

Summary

Follow-up to #126. In 0.6.0, rollout MemWAL appends accumulate flushed generations under _mem_wal/{shard}/ forever — there was no path to fold them back into the base table, so read cost grew linearly with append count (every list/get unions all generations). This PR adds an opt-in, size-triggered self-merge (the "external compactor" path Lance's MemWAL LSM design anticipates).

Set --rollout-merge-after-generations N (env ROLLOUT_MERGE_AFTER_GENERATIONS, or RolloutStoreOptions::merge_after_generations). After an append leaves this instance's own shard with ≥ N un-merged generations, the same add synchronously:

1. reads each flushed generation (a self-contained Lance dataset under _mem_wal/{shard}/),
2. Dataset::appends their rows into the base table, and
3. commit_updates the shard manifest to drain flushed_generations to empty — keeping replay_after_wal_entry_position so a reopened writer never re-replays merged WAL entries.

N = 0 (default) preserves the 0.6.0 pure-accumulation behavior.

Why it's safe

• No peer fencing. The merge claim_epochs the shard, bumping its writer epoch — but each instance merges only the shard it writes, which has no other live writer under the server-id sharding model (feat(rollout): MemWAL ingest with server-id sharding #126 / spec §3). The instance's next add re-claims the current epoch.
• Crash-safe. Rollout rows are immutable and deduped by id at read time. A crash between step 2 and step 3 (rows in base, manifest not yet drained) just means a reader sees the rows via both base and the still-listed generation and dedups them — no double counting. The next merge drains the manifest.
• No coordinator. Self-merge per instance: no cronjob, no admin endpoint, no cross-instance lock. The load balancer keeps spraying appends; each instance keeps its own shard compact.

Benchmark

bench_merge_read_amplification (200 single-row generations, local FS, release):

Read (list scan of 200 rows)	Latency
over 200 un-merged generations	~654 ms
after merge into base table	~50 ms
speedup	~13×

Slowest single add (merge-on-every-append at N=1): ~20 ms.

Changes

• RolloutStoreOptions::merge_after_generations + RolloutStore::{maybe_merge_own_shard, merge_own_shard}, triggered in add() after close().
• Server: --rollout-merge-after-generations flag → AppState → rollout route.
• Threaded through the unified facade (unified_rollout.rs).
• specs/rollout-deployment.md §6.1 documents the mechanism, safety, trade-off, and benchmark; §9 checklist updated.

Test plan

• below_threshold_no_merge — generations accumulate, no merge under threshold
• merge_at_threshold_drains_shard_and_preserves_reads — shard drains, rows present exactly once, inline artifact bytes still fetchable post-merge
• merge_is_visible_to_a_fresh_reader_instance — merged rows land in base, visible to a newly-opened reader
• core suite (106 pass) + server suite (26 pass), clippy clean, fmt clean

🤖 Generated with Claude Code