Introduce shared per-stream lifecycle ownership for deletion and mutations

[coderabbitai]

Summary

Introduce a shared per-stream lifecycle lock or generation boundary for stream deletion, stream creation, retention updates, and hot-tier mutations.

The current handlers can check Stream::is_deleting(), then await object-store or metastore work. During that await, deletion can finish and a stream with the same name can be recreated. The original operation can then update persistent state for the recreated stream.

Rationale

PR #1770 makes stream deletion asynchronous and uses durable tombstones. It prevents deletion cleanup from removing a recreated in-memory entry with Streams::delete_if_still_deleting. It also clears stream stats before the tombstone is removed.

The remaining race is different. Name-based retention and hot-tier operations do not retain lifecycle ownership across their awaited persistent operations. Deletion and recreation can therefore change the stream incarnation before those operations finish.

Required changes

• Define a per-stream lifecycle ownership mechanism. This can be a keyed lock, a stream generation token, or an equivalent design.
• Apply the mechanism to stream creation and deletion paths.
• Apply the mechanism to retention and hot-tier mutation paths.
• Recheck deletion ownership after the mechanism is acquired.
• Keep ownership through all object-store and metastore operations that are scoped by (tenant_id, stream_name).
• Before physical deletion and tombstone cleanup, verify that the deletion still owns the same stream incarnation.
• Preserve the existing durable-tombstone semantics and background deletion recovery behavior.

Affected areas

• src/storage/object_storage.rs
  • spawn_stream_deletion
  • tombstone reconciliation in sync_all_streams
• src/handlers/http/logstream.rs
  • stream create and delete handlers
  • retention mutation handler
  • hot-tier create and delete handlers
• src/parseable/mod.rs
  • stream creation and update paths
• src/parseable/streams.rs
  • stream lifecycle state, if the selected design needs generation tracking

Acceptance criteria

• A retention update cannot modify a stream that was deleted and recreated with the same name while the update awaited persistent storage.
• A hot-tier create or delete operation cannot modify a recreated stream after the original stream deletion completes.
• A deletion worker cannot delete data or clear lifecycle state for a new stream incarnation with the same (tenant_id, stream_name).
• Stream creation remains blocked while the durable tombstone exists.
• Restart recovery and periodic tombstone reconciliation preserve single deletion ownership.
• Add focused tests that force interleavings between deletion, recreation, retention, and hot-tier mutations.
• Existing stream deletion, tombstone recovery, and LocalFS idempotency behavior remains intact.

References

• Pull request: Make stream deletion asynchronous, resumable across restarts #1770
• Review discussion: Make stream deletion asynchronous, resumable across restarts #1770 (comment)
• Requested by: @prabhaks