fix(platform-wallet): close asset-lock resume broadcast race#4016
fix(platform-wallet): close asset-lock resume broadcast race#4016thepastaclaw wants to merge 7 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
39348ea to
0efd12c
Compare
|
⛔ Blockers found — Sonnet deferred (commit d59ddc4) |
thepastaclaw
left a comment
There was a problem hiding this comment.
Preliminary review — Codex only
The pre-broadcast status promotion fixes the intended single-resume interleave, but the newly added rejection rollback is not owned by the resume attempt that performed the promotion. Concurrent resume calls can therefore downgrade another successful broadcast to Built, allowing the create-side rejection cleanup to delete the row and release inputs for a transaction that may already be propagating.
Validated blockers were found in the Codex precheck. Sonnet is deferred until a fresh Codex revalidation clears the blocker gate.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed),gpt-5.6-sol— security-auditor (completed),gpt-5.6-sol— rust-quality (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet: not run (deferred by blocker gate)
🔴 1 blocking
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-platform-wallet/src/wallet/asset_lock/sync/recovery.rs`:
- [BLOCKING] packages/rs-platform-wallet/src/wallet/asset_lock/sync/recovery.rs:275-305: Rollback can clobber a concurrent successful resume
The write lock protects only the individual status mutation, not the promote/broadcast/rollback sequence across the broadcast await. Two callers can resume the same `Built` lock: caller A promotes it to `Broadcast`, while caller B snapshots either `Built` or `Broadcast` and successfully broadcasts the transaction. If A then receives `Rejected`, this filter still sees `Broadcast` because B remains at that status while waiting for proof, so A downgrades the shared row to `Built`. If the original create-side broadcast subsequently returns `Rejected`, `build.rs:530-550` sees `Built`, removes the row, and releases the funding reservation even though B handed the transaction to the network. Concurrent calls are supported by the public Rust and FFI resume entry points, and neither the manager nor the catch-up path serializes operations per outpoint. Tie rollback to exclusive attempt ownership with a per-outpoint single-flight guard or generation token, or conservatively retain `Broadcast` on rejection; the existing `Broadcast` arm already re-broadcasts on future resumes.
Note: GitHub does not allow PastaClaw to submit an approval or request changes on their own PR, so the canonical verifier result is transported as a COMMENT review.
…t_lock `resume_asset_lock`'s `Built` arm snapshotted the tracked row under a read lock, dropped it, then awaited `broadcaster.broadcast(&tx)` before advancing the status to `Broadcast`. During that window a concurrent `create_funded_asset_lock_proof` that received `Rejected` from its own broadcast would see the row still at `Built`, remove it, and release the funding reservation — while resume was still handing the same transaction to the network. Advance the row to `Broadcast` under the write lock BEFORE calling `broadcast`, so `untrack_asset_lock`'s guard fires (row + reservation preserved) or the advance itself fails and resume returns before broadcasting. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0efd12c to
afe2611
Compare
|
Addressed the blocking resume-race finding in amended commit
Validation: focused test passed; |
thepastaclaw
left a comment
There was a problem hiding this comment.
Preliminary review — Codex only
The prior rollback race is fixed: resume-side rejection no longer performs an unowned Broadcast -> Built mutation. One blocking concurrency issue remains because the new pre-broadcast promotion can use a stale Built snapshot to overwrite a proof-bearing final status and persist the regression.
Validated blockers were found in the Codex precheck. Sonnet is deferred until a fresh Codex revalidation clears the blocker gate.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed),gpt-5.6-sol— security-auditor (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet: not run (deferred by blocker gate)
🔴 1 blocking
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-platform-wallet/src/wallet/asset_lock/sync/recovery.rs`:
- [BLOCKING] packages/rs-platform-wallet/src/wallet/asset_lock/sync/recovery.rs:269-285: Stale Built snapshot can overwrite a concurrently attached proof
Two callers can both snapshot `Built` at lines 221-252. After the first caller promotes the row, broadcasts, obtains a proof, and stores `InstantSendLocked` or `ChainLocked` at lines 353-361, a delayed second caller still executes this pre-broadcast promotion. `advance_asset_lock_status` assigns the requested status unconditionally (`tracking.rs:180`), so it downgrades the finalized row to `Broadcast`; passing `None` leaves the existing proof attached. The later changeset is last-write-wins, so the inconsistent `Broadcast + Some(proof)` state can also be persisted. If the second broadcast returns `Rejected` or `MaybeSent`, these return paths retain that regression. A subsequent resume selects the `Broadcast` arm, ignores the attached proof, and waits for the transaction record again, potentially timing out or waiting indefinitely despite already having a valid proof. This failure path is introduced by moving the status write before broadcast: a stale caller whose broadcast fails now mutates shared state where it previously returned before the write. Make the `Built -> Broadcast` transition a compare-and-set under the wallet write lock; if the current row has already advanced, re-dispatch from its current status and use any attached proof instead of overwriting it.
Note: GitHub does not allow PastaClaw to submit an approval or request changes on their own PR, so the canonical verifier result is transported as a COMMENT review.
…romotion `resume_asset_lock` snapshots the tracked row under a read lock and drops it before the pre-broadcast promotion, so two callers can both observe `Built`. If the first one broadcasts, obtains a proof and finalizes the row to `InstantSendLocked` / `ChainLocked` (step 3), the delayed second caller still ran the unconditional `Built -> Broadcast` write: `advance_asset_lock_status` assigns the status regardless of the current value, and passing `proof: None` leaves the existing proof attached. That downgrades a finalized row to the inconsistent `Broadcast + Some(proof)` state, and since changesets are last-write-wins it gets persisted too. A later resume then takes the `Broadcast` arm, ignores the attached proof, and waits for a proof it already holds — unbounded for the user-facing funding flows. The `Rejected` / `MaybeSent` return paths kept the regression rather than unwinding it. Add `promote_built_to_broadcast`, a compare-and-set under the wallet write lock: it only promotes while the row is still `Built`, and otherwise mutates nothing and hands back the row's current status and proof. `resume_asset_lock` re-dispatches from those values, so a stale caller takes the already-have-a-proof arm instead of overwriting it. An untracked outpoint still errors, preserving the existing guard against re-broadcasting a rejected row whose funding reservation was released. `advance_asset_lock_status` keeps its unconditional semantics (its other call sites are monotonic proof-attaching advances) with a doc note pointing racing callers at the new helper. Regression test drives the interleave deterministically via a two-sided test-only gate: the resume signals once it has taken its `Built` snapshot, the test finalizes the row to `ChainLocked` with a proof, then releases the resume. Verified to fail without the compare-and-set (the stale caller downgrades the row and re-broadcasts) and pass with it. Co-Authored-By: Claude <noreply@anthropic.com>
Blocking finding addressed: stale
|
| Check | Outcome |
|---|---|
cargo fmt -p platform-wallet -- --check |
clean |
cargo check -p platform-wallet |
pass |
cargo clippy -p platform-wallet --all-targets |
no new warnings (2 pre-existing useless use of vec! in identity/network/withdrawal.rs, untouched) |
cargo test -p platform-wallet --lib asset_lock |
25 passed, 0 failed |
cargo test -p platform-wallet --lib |
471 passed, 0 failed |
cargo check -p platform-wallet-ffi |
pass (FFI consumer unaffected) |
Scope kept to this concurrency issue and its regression test; unrelated issue #4227 not touched.
…ast write The previous commit closed the resume-side stale promotion, but the create path kept the same unconditional writer. `broadcast_funded_asset_lock` tracks the row as `Built` (step 2), then awaits `broadcaster.broadcast(&tx)` — an unbounded network call — before assigning `Broadcast` on the way out. A concurrent `resume_asset_lock` can pick the same outpoint up inside that window: both the FFI catch-up scanner (`asset_lock_manager_catch_up_blocking`) and the funding resolver's `FromExistingAssetLock` arm drive one for any tracked row. If it broadcasts, obtains a proof and finalizes the row to `InstantSendLocked` / `ChainLocked` (its step 3), the original call's `advance_asset_lock_status(.., Broadcast, None)` then downgrades that finalized row — and since `advance_asset_lock_status` leaves the existing proof attached when passed `None`, it recreates the inconsistent `Broadcast + Some(proof)` state and persists it (changesets are last-write-wins). A later resume takes the `Broadcast` arm, ignores the attached proof and waits for one it already holds — unbounded for the user-facing funding flows. Same end state the resume-side fix addressed, reached from the other writer. Route this call through `promote_built_to_broadcast` too, so the promotion only fires while the row is still `Built`. When it has advanced, its status and proof are strictly further along than anything this call could write, so they are left untouched and the successful broadcast is still reported as success. Unlike `resume_asset_lock` there is nothing to re-dispatch: this create-only half neither re-broadcasts nor waits, since the proof wait lives in `wait_for_funded_asset_lock_proof`, the caller's next step. `sync::tracking` becomes `pub(super)` so `build.rs` can name `BuiltPromotion`. With this, no unconditional `Built -> Broadcast` writer remains. The two promotion sites (create + resume) both compare-and-set; the four other `advance_asset_lock_status` call sites (create/resume proof attach, and the shielded / platform-address IS->CL upgrades) are all monotonic terminal writes carrying `Some(proof)` and never write `Broadcast`. Regression test drives the interleave deterministically: the broadcaster finalizes the row to `ChainLocked` with a proof in place — standing in for the winning resume — and then returns `Ok`. Verified red on the old implementation (row read back `Broadcast`, and independently the persisted `Broadcast + Some(proof)` changeset assertion fires) and green with the compare-and-set. The finalize is in-memory only, so any persisted inconsistent changeset could only have come from the create path itself. Co-Authored-By: Claude <noreply@anthropic.com>
Follow-up: the create-side
|
| Site | Semantics | Safe? |
|---|---|---|
build.rs:580 create promotion |
now CAS via promote_built_to_broadcast |
fixed here |
recovery.rs:297 resume promotion |
CAS via promote_built_to_broadcast |
fixed in cf4d33de00 |
tracking.rs:232 |
the CAS itself — only assignment of Broadcast, guarded on status == Built under the write lock |
by construction |
build.rs:632 create proof attach |
Some(proof), terminal IS/CL status |
monotonic |
recovery.rs:407 resume proof attach |
Some(proof), terminal IS/CL status |
monotonic |
shielded/fund_from_asset_lock.rs:386 |
ChainLocked + Some(proof) (IS→CL upgrade) |
monotonic |
platform_addresses/fund_from_asset_lock.rs:243 |
ChainLocked + Some(proof) (IS→CL upgrade) |
monotonic |
recovery.rs:181,183 |
resolve_status_with_in_memory — classification for a first insert of an untracked row, guarded by contains_key re-check under the write lock (first writer wins) |
not a promotion |
advance_asset_lock_status retains its unconditional semantics; every remaining call site writes a terminal status with Some(proof) and never writes Broadcast. No unconditional Built -> Broadcast writer remains.
Regression coverage
create_broadcast_does_not_downgrade_a_row_finalized_during_the_broadcast drives the interleave deterministically: FinalizeDuringBroadcastBroadcaster finalizes the row to ChainLocked with a proof inside the broadcast await — standing in for the winning resume, inline rather than via a second task so the ordering is staged, not scheduler-dependent — then returns Ok. It asserts the row was Built on entry, so it fails loudly if the flow it targets ever changes shape.
The finalize is in-memory only, so any persisted Broadcast + Some(proof) changeset could only have come from the create path itself.
Verified against the old implementation — both assertions fire independently:
- status assertion:
left: Broadcast, right: ChainLocked— "a row finalized during the broadcast must not be downgraded to Broadcast on the way out" - with that assertion removed, the persistence assertion fires on its own: "no changeset may persist a Broadcast row with a proof attached"
Green with the compare-and-set. It also asserts exactly one broadcast call, pinning the "no re-dispatch, no re-broadcast" contract for this create-only method.
Validation
cargo fmt --all --check— cleancargo check -p platform-wallet/-p platform-wallet-ffi— cleancargo clippy -p platform-wallet --all-targets— 2 warnings, bothuseless use of vec!inidentity/network/withdrawal.rs(untouched by this change); confirmed identical at the base commit, so pre-existing- focused asset-lock tests — 26/26 pass
- full
platform-walletlib suite — 472/472 pass - pre-commit hook equivalents run manually (
pre-commitis not installed locally):cargo fmt --all, trailing-whitespace and line-ending checks on all three changed files — clean
Still a draft; no scope beyond the flagged path (unrelated #4227 untouched).
…ersist enqueue The `Built` -> `Broadcast` compare-and-set fixed the in-memory half of the race but left a persistence-ordering window open. `promote_built_to_broadcast` mutated the row under `wallet_manager.write()`, returned a `Broadcast` changeset, and released that lock before the caller ran `queue_asset_lock_changeset`. On the multi-thread runtime another flow could take the wallet lock in that window, finalize the same row to `InstantSendLocked` / `ChainLocked` with a proof, and enqueue that snapshot first -- after which the delayed promoter enqueued its older `Broadcast + None` snapshot last. Nothing downstream repairs the inversion: `FFIPersister::store_round` serializes rounds by acquisition order (so it faithfully preserves the reversal), `AssetLockChangeSet::merge` is last-write-wins per outpoint, and Swift's `persistAssetLocks` upsert overwrites `statusRaw` / `proofBytes` unconditionally. Memory stayed finalized while the durable row regressed to `Broadcast` with no proof -- and because the load path treats `statusRaw < 2` as still-pending, a restart resumed a lock whose proof it already had. Both the create and resume promotion callers had the window. Add `AssetLockManager::status_persist_serial` and hold it across mutation -> enqueue in every asset-lock status writer, folding the enqueue into the mutators so no caller can reorder the pair: - `track_asset_lock` (insert / Built) - `untrack_asset_lock` (rejected-row removal) - `promote_built_to_broadcast` (both Built -> Broadcast callers) - `advance_asset_lock_status` (proof attachment, IS -> CL upgrades) - `consume_asset_lock` (terminal Consumed) - `recover_asset_lock_blocking` (blocking_lock, matching its blocking_write) The mutex is acquired before `wallet_manager` everywhere and never the reverse, and is not held across the unbounded `broadcast` / `wait_for_proof` awaits -- only the two adjacent steps need ordering. A dedicated mutex rather than extending the wallet write guard because the persister call is synchronous and reenters Swift via FFI on iOS. Call semantics and error behavior are unchanged; the mutators still return their changeset for inspection. Regression tests (both fail without the mutex, pass with it): - `promotion_cannot_enqueue_a_stale_snapshot_after_a_concurrent_finalize` parks a promoter in the post-CAS / pre-enqueue window via a new test-only gate, finalizes and enqueues the same row from another task, releases the promoter, then asserts the durable row. Pre-fix it observes `Broadcast` where memory holds `ChainLocked`. The new `durable_asset_lock` helper folds stored rounds using the real downstream semantics (arrival order, LWW merge, unconditional upsert, removal delete), so it models what a restart reads. - `resume_promotion_enqueues_under_the_shared_ordering_mutex` pins that the resume path routes through the same primitive, so the fix cannot regress to covering only the create caller. Co-Authored-By: Claude <noreply@anthropic.com>
Persistence-order race between status mutation and persist enqueue (
|
| Layer | Behavior | Effect on a reversed pair |
|---|---|---|
wallet_manager write guard |
dropped on return from the mutator | window opens |
queue_asset_lock_changeset → persister.store() |
no sequencing beyond call order | reversal propagates |
FFIPersister::store_round (persistence.rs:723) |
store_round.lock() — serializes by acquisition order |
faithfully preserves the reversal |
AssetLockChangeSet::merge (changeset.rs:956) |
extend = last-write-wins |
stale Broadcast + None wins |
Swift upsert (PlatformWalletPersistenceHandler.swift:187) |
unconditional statusRaw / proofBytes overwrite |
proof erased on disk |
| Swift load | filters statusRaw < 2 |
restart reloads the stale row and re-drives it |
So a finalizer that reaches InstantSendLocked/ChainLocked + proof and enqueues after the promoter's CAS but before its enqueue leaves memory correct and durable state regressed.
Design — one shared ordering primitive, enqueue folded into the mutator
Added AssetLockManager::status_persist_serial: tokio::sync::Mutex<()> (mirrors the existing build_persist_serial precedent in the same struct), and moved queue_asset_lock_changeset inside each status mutator under that mutex. Patching call sites would have fixed one caller while leaving others free to reorder; folding the enqueue in makes the pairing structural.
Why a dedicated mutex rather than widening the wallet write guard: queue_asset_lock_changeset calls the persister synchronously, which reenters Swift over FFI on iOS. Holding wallet_manager.write() across host I/O would park every unrelated wallet reader behind it. The mutex is not held across the unbounded broadcast / wait_for_proof awaits.
Lock ordering, documented on the field and uniform across all holders:
status_persist_serial → wallet_manager.write() → drop(wallet) → enqueue → drop(serial)
Acquired before wallet_manager, never the reverse — no cycle.
Ordering audit — every mutation+enqueue path
| Path | Transition | Covered by |
|---|---|---|
track_asset_lock |
insert Built |
status_persist_serial |
untrack_asset_lock |
rejected-row removal | status_persist_serial |
promote_built_to_broadcast |
Built → Broadcast (create and resume callers) |
status_persist_serial |
advance_asset_lock_status |
proof attachment; IS→CL upgrades incl. both external fund_from_asset_lock.rs sites |
status_persist_serial |
consume_asset_lock |
terminal Consumed |
status_persist_serial |
recover_asset_lock_blocking (phase 3) |
sync recovery | status_persist_serial.blocking_lock() |
No path intentionally excluded.
Tests — negative control run
Two deterministic tests in build.rs, plus a durable_asset_lock helper on CapturingPersistence that replays changesets in arrival order with real downstream semantics (LWW merge, unconditional upsert, removal delete) so assertions reflect what a restart actually reads.
promotion_cannot_enqueue_a_stale_snapshot_after_a_concurrent_finalize— parks a promoter on a#[cfg(test)]post-CAS gate, spawns a finalizer toChainLocked + proof, sleeps 50ms to give it a real chance to win, releases the promoter; asserts both memory and durable state areChainLockedwith the proof retained.resume_promotion_enqueues_under_the_shared_ordering_mutex— holds the mutex, starts a resume, asserts nothing enqueued while blocked; releases and asserts the durable row advanced pastBuilt. Pins the resume path to the same primitive.
Negative control (mutex temporarily disabled) — both fail with exactly the predicted symptom:
promotion_cannot_enqueue_a_stale_snapshot_after_a_concurrent_finalize
left: Broadcast right: ChainLocked
"the durable row must not regress below the finalized in-memory status"
resume_promotion_enqueues_under_the_shared_ordering_mutex
left: 3 right: 2
Mutex restored; both pass.
Validation
| Check | Result |
|---|---|
cargo fmt -p platform-wallet -- --check |
clean |
cargo test -p platform-wallet --lib asset_lock |
28 passed, 0 failed |
cargo test -p platform-wallet --lib |
474 passed, 0 failed |
cargo check -p platform-wallet-ffi |
clean |
cargo test -p platform-wallet-ffi --lib |
181 passed, 0 failed (incl. concurrent_stores_never_interleave_rounds, persistence restore/round-trip) |
cargo clippy -p platform-wallet --lib --tests |
2 warnings, both pre-existing |
| repo pre-commit hook | ✅ Rust checks passed |
The 2 clippy warnings are useless use of vec! in wallet/identity/network/withdrawal.rs, an untouched file — confirmed pre-existing by stashing the change, re-running clippy (still 2), and unstashing.
Residual notes
recover_asset_lock_blockingusesblocking_lock(), which panics if ever called from inside a tokio async context. This matches the pre-existingblocking_writein the same method and its documented contract, but it is a second place that contract is now load-bearing.- The guarantee is per-
AssetLockManagerinstance, which assumes one manager per wallet id — howPlatformWallet::newconstructs it (shared across sub-wallets viaArc).
Scope held to this race: no changes to #4227 or broader persistence hardening.
6 files changed, 566 insertions(+), 60 deletions(-).
thepastaclaw
left a comment
There was a problem hiding this comment.
Preliminary review — Codex only
The published stale-Built finding is fixed: create and resume now use a write-locked compare-and-set, re-dispatch from the current row, and serialize each same-manager mutation with its persistence enqueue. Two blocking lifecycle races remain: terminal proof writers can downgrade ChainLocked to InstantSendLocked, and the new ordering mutex is not shared across stale and replacement manager handles for the same wallet ID. No same-instance lock-order inversion, reentrant persistence deadlock, async/blocking misuse, or uncovered production mutation path was found.
Validated blockers were found in the Codex precheck. Sonnet is deferred until a fresh Codex revalidation clears the blocker gate.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed),gpt-5.6-sol— security-auditor (completed),gpt-5.6-sol— rust-quality (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet: not run (deferred by blocker gate)
🔴 2 blocking | 🟡 1 suggestion(s)
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-platform-wallet/src/wallet/asset_lock/sync/tracking.rs`:
- [BLOCKING] packages/rs-platform-wallet/src/wallet/asset_lock/sync/tracking.rs:379-382: Terminal proof updates can regress ChainLocked to InstantSendLocked
`status_persist_serial` preserves mutation/enqueue order, but proof acquisition and validation happen before that mutex is acquired and this assignment remains unconditional. Caller A can obtain and validate an InstantSend proof while the transaction record still has InstantSend context, then be descheduled; after a ChainLock arrives, caller B can store `ChainLocked` with its stronger proof, and caller A can subsequently overwrite the row with `InstantSendLocked` and the stale proof. The mutex then guarantees that this downgrade is also the last durable write. This is reachable when create, resume, catch-up, or funding-resolution flows concurrently wait on the same outpoint, and it can restore a proof that Platform has already rejected or that later expires. Compare the requested transition with the current row while holding the lock, preserve `ChainLocked` over `InstantSendLocked`, and return or otherwise use the effective current proof so a losing caller does not continue with its stale local proof.
In `packages/rs-platform-wallet/src/wallet/asset_lock/manager.rs`:
- [BLOCKING] packages/rs-platform-wallet/src/wallet/asset_lock/manager.rs:144-148: Per-manager persistence mutex does not serialize stale handles for the same wallet
`status_persist_serial` is initialized separately in every `AssetLockManager::new`, but the FFI lifecycle permits multiple live manager generations for one wallet ID. `platform_wallet_get_asset_locks` stores an independent cloned `Arc` handle, while `platform_wallet_manager_remove_wallet` removes only the wallet from the Rust manager and does not invalidate subordinate handles. Re-importing the same mnemonic creates a fresh `AssetLockManager` for the same ID over the same shared `WalletManager` and persister; the stale manager also starts addressing the replacement wallet because every access is keyed only by `wallet_id`. The two managers can therefore mutate the same row under independent ordering mutexes: one can mutate a `Built` row to `Broadcast`, the other can store `ChainLocked + proof`, and the first can enqueue its older snapshot last. `FFIPersister::store_round` preserves that call order, so durable state regresses despite memory remaining finalized. Share the ordering primitive per wallet ID across manager generations, or generation-check and invalidate stale handles and in-flight operations.
In `packages/rs-platform-wallet/src/wallet/asset_lock/build.rs`:
- [SUGGESTION] packages/rs-platform-wallet/src/wallet/asset_lock/build.rs:2218-2222: Mutex regression tests rely on scheduler timing
The yield and 50 ms sleep give the spawned finalizer an opportunity to run but do not establish that it reached the mutex before the promoter is released. The resume test has the same issue at lines 2384-2402: if the spawned resume is not polled until after `drop(serial)`, its pre-release assertion passes even if the resume path bypasses `status_persist_serial`. The later durable-state checks only prove that the tasks eventually ran. Add a test-only notification immediately before each attempted mutex acquisition and wait for it before releasing the competing task or asserting blockage, making both negative controls independent of scheduler timing.
Note: GitHub does not allow PastaClaw to submit an approval or request changes on their own PR, so the canonical verifier result is transported as a COMMENT review.
| /// Lock ordering: acquire this BEFORE `wallet_manager`, never the | ||
| /// reverse. Every holder follows the same | ||
| /// `status_persist_serial → wallet_manager.write() → drop(wallet) → | ||
| /// enqueue → drop(serial)` shape, so no cycle exists. | ||
| pub(super) status_persist_serial: tokio::sync::Mutex<()>, |
There was a problem hiding this comment.
🔴 Blocking: Per-manager persistence mutex does not serialize stale handles for the same wallet
status_persist_serial is initialized separately in every AssetLockManager::new, but the FFI lifecycle permits multiple live manager generations for one wallet ID. platform_wallet_get_asset_locks stores an independent cloned Arc handle, while platform_wallet_manager_remove_wallet removes only the wallet from the Rust manager and does not invalidate subordinate handles. Re-importing the same mnemonic creates a fresh AssetLockManager for the same ID over the same shared WalletManager and persister; the stale manager also starts addressing the replacement wallet because every access is keyed only by wallet_id. The two managers can therefore mutate the same row under independent ordering mutexes: one can mutate a Built row to Broadcast, the other can store ChainLocked + proof, and the first can enqueue its older snapshot last. FFIPersister::store_round preserves that call order, so durable state regresses despite memory remaining finalized. Share the ordering primitive per wallet ID across manager generations, or generation-check and invalidate stale handles and in-flight operations.
source: ['codex']
There was a problem hiding this comment.
Resolved in c3ca845 — Per-manager persistence mutex does not serialize stale handles for the same wallet no longer present.
Auto-resolved by the review system based on the latest commit diff. If you believe this was closed in error, reopen the thread.
There was a problem hiding this comment.
Correction: this finding remains open at d59ddc4f. The sequential remove/re-import path is improved, but remove_wallet still drops the inner entry and the outer self.wallets entry in separate critical sections. A concurrent same-ID registration can install a replacement between them; the unqualified outer removal then removes that active replacement generation without deactivating it, allowing a later re-import to recreate managers with independent persistence mutexes. The preliminary review for this head carries the blocker forward with the current code path and evidence.
…l signal
Two asset-lock concurrency tests used a 50 ms sleep as their only
evidence that the competing task had reached the status-persistence
ordering boundary. Elapsed time is not a rendezvous:
- `durable_row_after_promote_finalize_interleave` released the parked
promoter after the delay whether or not the finalizer had run. If the
finalizer had not been scheduled yet, even the pre-fix implementation
could enqueue in the non-regressing order and the test would falsely
pass.
- `resume_promotion_enqueues_under_the_shared_ordering_mutex` read "no
new changeset after 50 ms" as proof the resume was blocked on the
mutex, but an unscheduled resume produces the same silence — including
if the resume path stopped taking the mutex altogether.
Add a test-only waiter gauge on `status_persist_serial`, maintained by
the new `lock_status_persist_serial` helper that every async acquirer
now routes through. It is incremented before the `lock().await` and
RAII-dropped on acquisition, so a non-zero count means a task is queued
at the boundary and cannot get past it while the test holds the lock.
Because the counter lives in the lock helper, an implementation that
stops taking the mutex never increments it and the waiting test fails
loudly instead of passing on silence.
The interleave helper now waits for the finalizer to either enqueue or
block on the mutex (mirroring the existing `build_serial_gate` test's
regressed-or-serialized pattern), so the release happens only after the
finalizer provably had its chance to go first.
Test-only: production behavior is unchanged, and the gauge and helper
compile out of non-test builds.
Negative control: removing the five `status_persist_serial` acquisitions
makes both tests fail 5/5 runs — the first on the durable-row ordering
assertion (Broadcast vs ChainLocked), the second on the new rendezvous
panic ("saw 0"), which is precisely the case the sleep version misread
as success.
Co-Authored-By: Claude <noreply@anthropic.com>
Deterministic rendezvous for the two ordering testsFollow-up addressing the one blocking review finding: two concurrency tests used a 50 ms sleep as their only evidence that the competing task had reached the status-persistence lock boundary. Diagnosis (verified against
|
… removed
`PlatformWallet` hands out `Arc<AssetLockManager<_>>` clones and the FFI
parks them in its own handle storage, so a handle can outlive
`PlatformWalletManager::remove_wallet`. Wallet ids are deterministic in
(seed, network), so re-importing the same mnemonic recreates the very
same id over a fresh `PlatformWalletInfo` and a fresh
`AssetLockManager` with its own `status_persist_serial`. Because the
manager resolves everything through the shared `WalletManager` by
`wallet_id` alone, the retained handle silently re-attached to the
replacement wallet — and old and new managers then mutated and
persisted the same asset-lock rows under *different* mutexes,
reintroducing across instances exactly the stale-snapshot enqueue
reversal serialization closes within one. The `untrack` arm is the
sharpest: its changeset's `removed` set DELETES the durable row, so a
stale handle could erase a lock the replacement wallet had just
tracked.
Give each manager an `active` flag and flip it in `deactivate()` while
HOLDING that manager's `status_persist_serial`. That is what makes the
retirement a barrier rather than a hint, in both directions:
- it cannot take the mutex until any in-flight mutate→enqueue unit has
released it, so removal never interrupts one halfway (row mutated in
memory, changeset never handed to the persister);
- every such unit re-reads the flag once it holds that same mutex, so
an operation that started before the removal and was parked on an
unbounded `broadcast` / proof wait fails before it can touch a wallet
row or the persister.
`track_asset_lock`, `untrack_asset_lock`, `consume_asset_lock`,
`promote_built_to_broadcast`, `advance_asset_lock_status` and the
blocking `recover_asset_lock_blocking` all take the check under the
mutex, before their wallet lookup. The two tracking primitives now
return `Result`; in `build.rs`'s rejected-broadcast path a refusal is
treated like the existing untrack guard — the funding reservation is
NOT released, since the wallet that took it is gone and `wallet_id`
now resolves to a replacement that never reserved those inputs. Public
entry points (`resume_asset_lock`, `broadcast_funded_asset_lock`, the
top of `recover_asset_lock_blocking`) also pre-check, but only as an
early-out; those checks are documented as non-authoritative because a
removal can land during the very next await.
`remove_wallet` resolves the `PlatformWallet` first and deactivates
with no other lock held, then drops the shared `WalletManager` entry.
Order matters: re-registration must `insert_wallet` into that shared
manager, which still holds this wallet's entry at that point, so no
replacement can exist until deactivation is done. Deactivating under
`wallet_manager` would invert the documented
`status_persist_serial -> wallet_manager` order and deadlock.
`WalletNotFound` idempotency (which the FFI maps to ok) is unchanged.
Two regression tests, both rendezvousing on the production
`status_serial_waiters` gauge rather than on elapsed time:
- `deactivation_waits_for_the_in_flight_unit_then_refuses_every_later_mutation`
parks a promoter post-CAS, proves `deactivate` comes to rest at the
ordering boundary while the flag is still set, proves the in-flight
unit completes and reaches the persister, then proves every later
primitive is refused and enqueues nothing.
- `retained_asset_lock_manager_cannot_touch_a_reimported_wallet` keeps
a handle across removal, re-imports the same mnemonic, asserts the
id collides and the manager is a different instance, then proves the
retained handle can neither mutate the replacement's in-memory row
nor enqueue a single round through the shared persister. Includes a
negative control that the same operations succeed through that exact
handle while the wallet is live.
Negative controls: dropping the `deactivate` call from `remove_wallet`
fails the second test on the first stale `track_asset_lock` (it
returns `Ok` against the replacement wallet); removing the
`lock_status_persist_serial().await` from `deactivate` fails the first
test at the rendezvous ("saw 0") — the bare-flag design the barrier
replaces.
Scope is asset-lock lifecycle only: no FFI handle redesign, and the
separate non-blocking ChainLocked -> InstantSendLocked monotonicity
finding is untouched.
Co-Authored-By: Claude <noreply@anthropic.com>
`advance_asset_lock_status` assigned unconditionally, and it is the single write point for asset-lock status. Serialization does not close this: `status_persist_serial` makes each writer's mutation and enqueue one indivisible unit, so the durable order matches the in-memory order — but the two proof-bearing writes come from INDEPENDENT waiters, so nothing orders them against each other. `wait_for_proof` returns whichever SPV event fires first, while the IS->CL upgrade paths (`upgrade_to_chain_lock_proof` after a Platform IS rejection, `validate_or_upgrade_proof` on a rotated quorum) can complete a `ChainLocked` write with an earlier IS waiter still parked. Released afterwards, that IS waiter's write was internally consistent and perfectly serialized — and still regressed the row. Both halves of the regression hurt. The status fell below the `>= InstantSendLocked` predicates the catch-up scanner and the ready-to-fund UI filter use. The proof swap is worse: the caller passes `Some(is_proof)`, so the ChainLock proof — the one that survives quorum rotation, and the one an IS-rejection retry had upgraded TO — was replaced by the IS proof that rejection was about. A restart then reloaded the weaker proof from the durable row and re-armed the very rejection the upgrade resolved. Guard the write under the same `status_persist_serial` the mutation and enqueue already hold, so the status it compares against is the one it is about to overwrite. A strictly-lower-rank `new_status` mutates nothing, replaces no proof, and enqueues nothing. Ordering comes from an explicit `AssetLockStatus::lifecycle_rank`, an exhaustive match on named variants rather than `as u8` on the declaration order. The two agree today, but that order is separately load-bearing for the FFI discriminants (`status_from_u8`) and the SQLite label domain, and this is a *semantic* claim about the lifecycle — tying it to declaration position would let a reordering silently redefine which writes count as downgrades. No `_` arm, so a new variant is a compile error here. Caller-return semantics are deliberately unchanged. A refusal is `Ok(empty changeset)`, not an error: every production caller returns/uses the proof it already obtained and ignores the changeset, so the delayed IS caller may still submit with its valid IS proof. Only SHARED state — the in-memory row and the durable row — is held at the stronger proof. Equal-rank writes are NOT refused, which keeps two live shapes working: attaching the first proof to a row `resolve_status_with_in_memory` marked `InstantSendLocked` with `None` (it has no IS-lock data), and re-writing `ChainLocked` with a freshly-upgraded proof at a newer height. Two regression tests, no sleep in either: - `a_late_instant_send_write_cannot_downgrade_a_chain_locked_row` parks an IS writer on a new test-only gate placed BEFORE the ordering mutex (a gate after it would invert the interleave — the parked writer would hold the lock and its IS write would land first, a legal forward advance), finalizes the same row to `ChainLocked` + chain proof and awaits that call so the stronger write is provably complete, then releases the IS writer. Asserts in-memory AND durable state both stay `ChainLocked` with the chain proof, that nothing was enqueued, and that the refusal is `Ok` with an empty changeset. The gate is one-shot so the competing finalize — same method — runs straight through instead of deadlocking against the gate it races. - `monotonicity_guard_allows_forward_and_same_status_proof_writes` pins the other direction: forward transitions, equal-rank proof attachment, and equal-rank proof refresh all still mutate and enqueue. Negative controls: disabling the guard fails the first test three times over, independently — on the empty-changeset assertion, on the in-memory status, and on the durable status. Writing the guard as `<=` instead of `<` fails the second test on the equal-rank proof attachment. Scope is monotonic asset-lock status/proof writes only. The separate manager-lifecycle fix is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
Exact-head finding confirmed and fixed —
|
| Control | Result |
|---|---|
Guard disabled (if false && …) |
FAILS at build.rs:2873 — empty-changeset assertion |
| …empty-changeset assertion also neutralized | FAILS at build.rs:2892 — in-memory left: InstantSendLocked, right: ChainLocked |
| …in-memory assertions also neutralized | FAILS at build.rs:2902 — durable left: InstantSendLocked, right: ChainLocked |
Guard written <= instead of < |
FAILS monotonicity_guard_allows_… on equal-rank proof attachment |
So the first test fails three times over, independently, on the exact state the finding describes, and the <= control shows the boundary is deliberate rather than incidental. Guard restored; tree clean.
Validation
cargo test -p platform-wallet --lib asset_lock→ 34 passed, 0 failedcargo test -p platform-wallet --lib→ 480 passed, 0 failedcargo test -p platform-wallet --lib --all-features→ 607 passed, 0 failed (covers theshieldedChainLock-upgrade caller, which default features exclude)cargo test -p platform-wallet-ffi→ 181 + 26 + 5 passed, 0 failed (4 network-gated ignored)cargo check -p platform-wallet-ffi→ cleancargo fmt -p platform-wallet -- --check→ cleancargo clippy -p platform-wallet --all-targets→ 2 warnings, bothuseless use of vec!atwallet/identity/network/withdrawal.rs:229,268— pre-existing, in a file no commit in this PR range touchescargo check -p platform-wallet --no-default-features→ fails with 4 errors inwallet/provider_key_at_index.rs. Pre-existing: reproduced at PR baseafe2611124(5 errors there), and the file is untouched by this PR.
Cumulative diff afe2611124..d59ddc4f53 is 10 files, all under packages/rs-platform-wallet/src/. No stray files; nothing touching #4227 or broadening FFI lifecycle/error-code work. Scope is monotonic asset-lock status/proof writes only; the manager-lifecycle and rendezvous fixes from prior commits are unchanged.
Commit
d59ddc4f530c8084b6a806c96e9e38ba96a6e1bb — fast-forward descendant of f93f011925, 4 files, +606/−5:
packages/rs-platform-wallet/src/wallet/asset_lock/build.rs +378
packages/rs-platform-wallet/src/wallet/asset_lock/manager.rs +39
packages/rs-platform-wallet/src/wallet/asset_lock/sync/tracking.rs +104/−5
packages/rs-platform-wallet/src/wallet/asset_lock/tracked.rs +90
Residual risk
The guard protects the row against regression, not against two callers legitimately writing at equal rank — by design, since that is how a None-proof InstantSendLocked row ever gets a proof. Last-writer-wins still applies among equal-rank proof writes; both are valid proofs for the same status, so this is not believed to be exploitable, but it is the remaining unordered case.
PR left in draft.
thepastaclaw
left a comment
There was a problem hiding this comment.
Preliminary review — Codex only
The lifecycle-rank guard correctly prevents delayed proof downgrades while preserving every forward transition and equal-rank proof attachment or refresh. Two blocking lifecycle gaps remain: concurrent registration can recreate an active stale manager generation, and retired managers can still mutate replacement-wallet state through transaction-building paths. The ordering tests now use a real mutex-arrival signal, but one rendezvous can miss an early enqueue and hang indefinitely.
Validated blockers were found in the Codex precheck. Sonnet is deferred until a fresh Codex revalidation clears the blocker gate.
Review provenance
- Codex reviewers:
gpt-5.6-sol— general (completed),gpt-5.6-sol— security-auditor (completed),gpt-5.6-sol— rust-quality (completed) - Verifier:
gpt-5.6-sol— verifier - Sonnet: not run (deferred by blocker gate)
🔴 2 blocking | 🟡 1 suggestion(s)
2 additional finding(s) omitted (not in diff).
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-platform-wallet/src/manager/wallet_lifecycle.rs`:
- [BLOCKING] packages/rs-platform-wallet/src/manager/wallet_lifecycle.rs:653-657: Per-manager persistence mutex does not serialize stale handles for the same wallet
The new deactivation barrier closes the sequential remove/re-import path, but removal still updates the inner `WalletManager` and `self.wallets` in separate critical sections. After the old inner entry is removed at lines 643-651, a concurrent same-seed registration can insert its replacement into the inner manager and then install its new `PlatformWallet` in `self.wallets` at registration lines 514-518. This unqualified `remove(wallet_id)` can then remove and return that replacement rather than the old, already-deactivated generation. The replacement remains active and present in the inner manager but absent from the public map. A later idempotent removal skips deactivation because the outer entry is missing, removes the inner entry, and reports success; another same-ID import then lets the retained active generation address the new replacement by `wallet_id`, recreating two managers with independent persistence mutexes and the original stale snapshot/delete race. Serialize registration and removal per wallet ID, or make both removals generation-aware so they can only detach the exact wallet generation that was captured and deactivated.
In `packages/rs-platform-wallet/src/wallet/asset_lock/build.rs`:
- [BLOCKING] packages/rs-platform-wallet/src/wallet/asset_lock/build.rs:67-70: Retired managers can still build against a replacement wallet
`build_asset_lock_transaction` resolves mutable wallet state solely by the deterministic `wallet_id`, without an activity or generation check synchronized with `deactivate`. The public FFI function `asset_lock_manager_build_transaction` calls this method directly, so a retained handle can build against a same-mnemonic replacement after retirement. The builder advances the replacement's funding address index and reserves its UTXOs, and with the same seed can return a signed transaction spending those inputs. The funded path has the same race: it checks `active` before awaiting `build_persist_serial`, so an operation can pass the check, remain parked while removal and re-import complete, then build against and persist an account-pool snapshot from the replacement before `track_asset_lock` finally rejects the inactive manager. Because no lifecycle row is tracked, those reservations are orphaned. Put build mutation and account-pool persistence inside a generation-safe retirement barrier, including an authoritative check after waiting for `build_persist_serial` and before any wallet lookup.
- [SUGGESTION] packages/rs-platform-wallet/src/wallet/asset_lock/build.rs:2269-2289: Finalizer rendezvous can miss an early enqueue and wait forever
The persistence baseline is captured after the finalizer task is spawned. Under the intended negative control where the status mutex is removed, the finalizer can enqueue before `queued_before_finalize` is read; that enqueue is then included in the baseline, the waiter gauge remains zero, and this unbounded loop can never observe either exit condition. A finalizer panic or early return causes the same hang. Capture the persistence count before spawning the finalizer, bound the rendezvous with `tokio::time::timeout`, and surface a finalizer task that has already completed without producing either signal.
Note: GitHub does not allow PastaClaw to submit an approval or request changes on their own PR, so the canonical verifier result is transported as a COMMENT review.
| let queued_before_finalize = persistence | ||
| .stored | ||
| .lock() | ||
| .expect("capturing persistence mutex") | ||
| .len(); | ||
| loop { | ||
| let finalizer_enqueued = persistence | ||
| .stored | ||
| .lock() | ||
| .expect("capturing persistence mutex") | ||
| .len() | ||
| > queued_before_finalize; | ||
| let finalizer_blocked = manager | ||
| .status_serial_waiters | ||
| .load(std::sync::atomic::Ordering::SeqCst) | ||
| >= 1; | ||
| if finalizer_enqueued || finalizer_blocked { | ||
| break; | ||
| } | ||
| tokio::time::sleep(Duration::from_millis(5)).await; | ||
| } |
There was a problem hiding this comment.
🟡 Suggestion: Finalizer rendezvous can miss an early enqueue and wait forever
The persistence baseline is captured after the finalizer task is spawned. Under the intended negative control where the status mutex is removed, the finalizer can enqueue before queued_before_finalize is read; that enqueue is then included in the baseline, the waiter gauge remains zero, and this unbounded loop can never observe either exit condition. A finalizer panic or early return causes the same hang. Capture the persistence count before spawning the finalizer, bound the rendezvous with tokio::time::timeout, and surface a finalizer task that has already completed without producing either signal.
source: ['codex']
Follow-up to merged #3985 / tracker thepastaclaw/tracker#1742.
The later automated review on #3985 found a real read-before-broadcast race:
resume_asset_lockcould snapshot a Built row, then a concurrent create-path Rejected cleanup could delete the row and release the reservation before resume rebroadcasted the same tx.This PR:
Validation:
cargo test -p platform-wallet --lib asset_lock::cargo clippy -p platform-wallet --lib --testscargo build -p platform-wallet --testsgit show --check HEADNote: the race regression test was also verified to fail against the pre-fix implementation.