feat(evm): share browser payment APIs and durable payment recovery - #17
Conversation
Measured on DEV-03 run 589 (2026-09-16) against sepolia-rollup.arbitrum.io: 26 of 938 native uploads failed at the payment step, none of them for a network reason, all in the journal path introduced by a5f3d61: - `observe_payment` reported `Finalizing` when a `latest` nonce read was higher than the nonce just signed from a `pending` read. The two reads are separate HTTP requests and the endpoint's replicas lag each other (reproduced: `pending=98` then `latest=99`, 1 pair in 150), and finality on Arbitrum Sepolia trails by ~28 nonces, so every stale read became "awaiting chain finality" on bytes that were never broadcast. - Every read in `prepare_payment`/`observe_payment` and the broadcast itself were single-shot, where the legacy `send_transaction_with_retries` path retried three times. One `429` or `-32000 context deadline exceeded` on any of ~9 calls per wave failed the upload, after earlier waves had paid. Now: - `prepare_payment` takes the higher of a `pending` and a `latest` read, so one lagging replica cannot sign a consumed nonce, and sets the chain id explicitly so `fill` makes no RPC call. - Reads go through `retry()`. Reads made while a caller polls use a 500ms base (the native client observes inside a 30s window). - A consumed-but-unfinalised nonce with no receipt reads as `Pending`, not `Finalizing`: it is indistinguishable from a stale read, re-sending the journaled bytes is refused by the node rather than paid twice, and finality still resolves a real replacement into `Replaced`. `Finalizing` is kept for a mined revert that is not yet final. - `broadcast_payment` retries the same bytes on transport-level failures (HTTP status, connection loss, timeouts, backend-deadline responses) and treats "already known" as success. Definitive rejections return at once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grumbach
left a comment
grumbach
left a comment
There was a problem hiding this comment.
Review at ccd65f18
Verdict: changes requested. Two small fixes, plus some hardening. The journal design holds up well, and the fix for the "Finalizing on a payment that was never sent" failure from staging run 589 is right: a nonce that looks consumed with no finalized evidence now returns Pending instead of failing the upload.
Also verified: the journal binds the signed transaction to the wallet (recovered signer), chain id, vault, zero value and the exact calldata, and the quote-hash and Merkle-calldata refactors produce the same bytes as main.
Must fix
- The gas-fee read is still single-shot.
prepare_paymentcallscrate::retry::get_eip1559_feesdirectly (src/wallet/journal.rs:214), and that function has no retry (src/retry.rs:384). The chain-id, gas-estimate and nonce reads around it now go through the newrpc(...)wrapper, but this one does not. Staging run 591 (at this exact revision) lost 2 of 3,087 native uploads toCould not get current gas price(one HTTP 429, one-32000 context deadline exceeded), both before payment. Main's legacy path retried fee fetches insidesend_transaction_with_retries, so this is a regression for CLI users. Fix: wrap the call inrpc("fee estimate", None, ...). - Clippy job is red on
clippy::result_large_erratsrc/wallet.rs:151and:391. Those signatures and error types are unchanged from main (the lines only moved), so this looks like a newer-toolchain lint that main would also hit. It still blocks the merge check. Fix: box the error or allow the lint on those two functions.
Hardening (low likelihood, but worth doing before claiming a payment can never be made twice)
observe_paymentreturnsReplacedwhenever a finalized transaction with different calldata consumed the nonce (src/wallet/journal.rs:416-425), without checking whether that transaction paid some of the same quote hashes. Only reachable if the same wallet pays overlapping quote sets in two different transactions, but decoding the replacement's quote hashes first would close it.- A failed receipt is classified
Revertedby comparing its block number with the finalized height (plus a canonical lookup at that number), not by checking that the receipt's block is an ancestor of the exact finalized block hash. A load-balanced RPC mixing replicas could in principle produce a falseReverted. Unlikely on Arbitrum. - Restoring a Merkle checkpoint recomputes the root from the stored leaves without comparing it to the root that was paid for (
src/merkle_payments/merkle_tree.rs:84-106). Storing and checking the expected root would catch corrupted leaves early; the salts are not part of the root, so they would need their own check.
Tested locally
cargo test --test wallet journaled(with Anvil): 7 passed.cargo test --no-default-features: 10 passed.cargo check --lib --no-default-features --features rpc,external-signer --target wasm32-unknown-unknown: passes.cargo clippy --all-targets --all-features -- -D warnings: fails exactly as CI does.
The consumer side of the journal (ant-client's stale-nonce retry) has a related issue; details on ant-client #186.
Testnet evidence — DEV-03 run 591 (2026-09-17),
|
| 30-min slot from T0 (UTC) | native uploads | WASM uploads | native downloads | WASM downloads |
|---|---|---|---|---|
| T0 → +30m (19:16) | 356/356 (29.8 GiB) | 12/13 (1.4 GiB) | 42/42 (9.8 GiB) | 13/13 (0.5 GiB) |
| +30m → +60m | 371/371 (30.9 GiB) | 16/16 (2.6 GiB) | 48/48 (8.9 GiB) | 15/15 (0.5 GiB) |
| +60m → +90m window | 385/385 (32.3 GiB) | 16/16 (2.0 GiB) | 42/42 (12.8 GiB) | 7/7 (0.3 GiB) |
| +90m → +120m window | 392/392 (33.4 GiB) | 16/16 (2.6 GiB) | 50/50 (10.3 GiB) | 11/11 (1.1 GiB) |
| +120m → +150m window | 391/391 (32.4 GiB) | 16/16 (2.6 GiB) | 48/48 (10.7 GiB) | 14/14 (0.5 GiB) |
| +150m → +180m window | 388/388 (32.3 GiB) | 18/18 (2.6 GiB) | 49/49 (11.2 GiB) | 11/11 (0.6 GiB) |
| +180m → +210m window | 388/389 (32.7 GiB) | 16/16 (1.7 GiB) | 53/53 (9.3 GiB) | 11/11 (0.6 GiB) |
| +210m → +240m window | 392/393 (33.8 GiB) | 17/17 (2.6 GiB) | 50/50 (9.9 GiB) | 15/15 (0.6 GiB) |
No decay in rate or success on either transport from the first half-hour to the last. Measurement-window totals: native uploads 2336/2338 (99.91%), native downloads 292/292, WASM uploads 99/99, WASM downloads 69/69. Whole run (incl. warm-up): native 3085/3087 up / 386/386 down; WASM 127/128 up / 98/98 down. WASM covered 20 MB, 300 MB (single payment) and 900 MB (merkle) uploads, and downloads up to 500 MB, all paid through the JS callback (approve + payForQuotes) with the payment network taken from the node's HELLO.
Durations (ok ops, window): native upload p50 24.4 s / p95 126.5 s, WASM upload p50 146.1 s / p95 1021.8 s; native download p50 17.2 s / p95 72.2 s, WASM download p50 110.8 s / p95 236.3 s. Size-matched 20 MB: upload 16.8 s native vs 137.5 s WASM, download 15.5 s vs 96.6 s — the browser path is slower, as expected, but consistently so.
Failures, all three of them (out of 3,699 transfers)
- WASM upload fix: use random port for Anvil instead of hardcoded 61611 #1 at T0 on the 20 MB uploader:
Witnessed close group inconclusive before payment: got 0/1 quorum-recognised peers— first-op-after-start transient, pre-payment, nothing spent; ci: add PR checks and release workflows #2 succeeded three minutes later and nothing recurred in 127 further uploads. - Native sgp/50 MB #336 (22:19Z):
witnessed close group initial lookup found 6 peers, need 7— DHT transient, pre-payment. - Native sjc/100 MB #418 (23:06Z):
payment error: Could not get current gas price: HTTP error 429from the public Arbitrum Sepolia RPC, pre-payment, nothing spent, next upload fine.
That last one (plus one identical -32000 context deadline exceeded on the same call five minutes after the window) technically fails the run's zero-tolerance "no RPC-transport payment failure" criterion, so the issue is scored 10/11. It is not a significant finding: 2 in 3,087 native uploads (0.065%), never post-payment, never money lost, and the cause is narrow — get_gas_price is the one read in the payment prepare path that evmlib ccd65f18 did not put behind retry() (estimate_gas, get_transaction_count, get_chain_id, get_transaction_receipt, get_block_by_number are). Zero evmlib::retry warnings appeared anywhere in the run, i.e. the public RPC was otherwise quiet. A one-line retry wrap closes it; it says nothing about the transport, the node, or the WASM path.
Also clean: 0 clock-skew signatures on either tier against the +3 h node (unsubmitted payment quotes expired / paid proof expired), 0 awaiting chain finality, 0 partial uploads, 0 not within its local closest peers, 0 invalid public file size (the WASM downloader skipped 111 over-cap entries by design), 0 OOM (WASM downloader peak 2.45 GB on the 500 MB pick), and a fleet-wide sweep of 1,009 ant-* units on 85 hosts found no panic, restart or replaced binary.
Node tier under this load: CPU mean 23.5% per host, per-service RSS p50 262 MB / p95 283 MB; steady-state 35.4 GB egress per GB uploaded, 8.7 GB disk per GB stored — in line with the previous (contaminated) run and not split by transport, since one fleet served both.
The "Enforce Clippy constraints" job started failing on PR #17 with `clippy::result_large_err` for both `pay_for_quotes` functions in `wallet.rs`. Nothing in the PR touched the error type: CI tracks `stable`, which is now Rust 1.98, and its clippy flags `PayForQuotesError` where 1.97 did not. `wallet::Error` is exactly 128 bytes, right at the lint threshold, so it passes on its own. Wrapping it together with the `BTreeMap` of succeeded payments pushes `PayForQuotesError` to 160 bytes. Boxing the inner error brings the struct down to 32 bytes. This is preferred over an `#[allow]` because every downstream consumer in `ant-protocol` and `ant-client` destructures the error as `PayForQuotesError(err, _)` and only formats `err` via `Display`, which works unchanged through `Box`. Verified on Rust 1.98.1: - `RUSTFLAGS="-D warnings" cargo clippy --all-targets --all-features`: clean - `cargo check --lib --no-default-features --features rpc,external-signer --target wasm32-unknown-unknown`: clean - `cargo fmt --check`: clean - `cargo test --lib --all-features`: 21 passed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 participants
Summary
Allows browser clients to use the same EVM payment types, quote hashing, pricing, Merkle algorithms, and vault encodings as native clients. The portable layer excludes native process and environment facilities; optional RPC uses browser Fetch and timers, while the default native feature retains Anvil and native wallet support.
Merkle checkpoints preserve the original salts and random padding so recovery reconstructs the same payment intent. Native wallets can prepare and sign a payment without broadcasting, persist its exact transaction bytes, then submit or observe that same transaction after interruption. Journal validation binds the transaction to its wallet, chain, vault, and intended calldata.
Recovery distinguishes pending transactions, unfinalized failures, finalized reverts, and finalized nonce replacements. A same-intent fee replacement returns its successful receipt and settlement hash; a different finalized nonce consumer permits a fresh attempt. Missing history, RPC errors, and unexplained nonce advances retain the journal. Failed-payment retry decisions use canonical receipts and the RPC
finalizedtag; successful payments keep the existing optimistic canonical-inclusion semantics. See PAYMENT_RECOVERY.md for finality and historical-RPC requirements.Native-only tests are feature-gated, and CI covers both portable and RPC-only test configurations.
Linear issue
Closes V2-803 — Linear issue
Risk tier
Proposed for human review: changes payment submission and recovery behavior.
Compatibility
Consumers of the draft journal API must handle
PaymentStatus::FinalizingandPaymentStatus::Replaced. The storedSignedPaymentrepresentation is unchanged.Semver impact
Test evidence
Validated on the current revision during this refresh:
cargo test --test wallet journaled: 7 passed, covering canonical success, Merkle winner recovery, reverted transactions across a reorg, finality before retry, finalized nonce replacement, successful fee replacement, and unexplained nonce consumption.cargo test --no-default-features: 10 tests passed; native-only integration targets are gated out.cargo test --lib --test cryptography --no-default-features --features rpc,external-signer: 19 passed.cargo clippy --all-targets --all-features -- -D warnings: passed.cargo check --lib --no-default-features --features rpc,external-signer --target wasm32-unknown-unknown: passed.The coordinated ant-client recovery suite also exercises the native caller's handling of finalized replacements. Local results do not replace CI or production RPC validation.
New dependency
Adds
web-timefor portable clocks and WASM-targetgloo-timers,futures-util, and JavaScript-enabledgetrandom. Existing Alloy and Tokio dependencies use explicit feature gates. No new standalone crate or parallel browser payment implementation is introduced.ADR
ant-client ADR-0004: Direct browser immutable-data client documents shared payment preparation, journals, and recovery across native/browser adapters. Saorsa Core ADR-019 covers the shared portable dependency architecture.
Mitigation / rollback
Keep consumers on the default native feature or revert the coordinated client/EVM revisions together. Preserve unresolved transaction journals; a missing receipt is not proof that another payment is safe.
Coordinated consumers
Current stack revisions
All PRs link to V2-803. Immutable Git revisions used by the validated stack:
29a2b272cf424c046f0b1f620df7853ecdae7d1926f5fb79f1303e1aEVM recovery now retains ambiguous journals until finalized failure/replacement evidence is available. The protocol and native client include the matching consumer changes. Existing browser wire and payment formats are retained.