Skip to content

wollet: handle esplora address history paging - #169

Open
dadafros wants to merge 2 commits into
Blockstream:masterfrom
dadafros:faster-esplora-address-history
Open

wollet: handle esplora address history paging#169
dadafros wants to merge 2 commits into
Blockstream:masterfrom
dadafros:faster-esplora-address-history

Conversation

@dadafros

Copy link
Copy Markdown

Two changes to the async esplora client's vanilla (non-waterfalls) scan path, in two commits:

1. wollet: handle esplora address history paging — resolves the TODO must handle paging in get_scripts_history_esplora. The client read only the first page of GET /address/{addr}/txs, so an address with more confirmed transactions than one page (25 on Blockstream esplora) synced a silently truncated history — and since the scan treats cache txids absent from the fetch as deletions (txid_height_delete), the missing transactions were also evicted from the cache, churning delete/re-add on every scan.

The loop now follows /txs/chain/{last_seen_txid} from the last confirmed txid until a page with no confirmed transactions arrives — the only termination condition every implementation guarantees: page sizes are configurable in the mempool/electrs fork, and its first page shares a single budget between mempool and confirmed entries (observed live on liquid.network: 50-entry shared first page vs blockstream.info's 25-confirmed + mempool), which can crowd confirmed entries out of the first page entirely — that case is handled with an explicit cursor-less /txs/chain request. A repeated cursor errors out so a server that ignores it cannot loop the client forever. This also brings the vanilla path in line with the waterfalls path, which already pages via has_more.

2. wollet: fetch address histories concurrently — the address walk awaited one request per address sequentially, so scan wall-clock grew linearly with address count even though EsploraClientBuilder::concurrency already parallelizes transaction and header downloads. The batch now runs through .buffered(self.concurrency) (ordered — callers map results back to derivation indices positionally) with try_collect keeping the sequential loop's fail-fast semantics. Default concurrency remains 1: no behavior change unless opted in.

Notes

  • Request-count trade-off: an address with confirmed history now costs at least one extra (final, empty) page request — the price of a termination rule that works across implementation-dependent page sizes (an early-exit-on-short-page heuristic would silently truncate on servers with page sizes other than 25).
  • No total page cap: an honest large history takes the requests it takes; the repeated-cursor guard bounds cursor-ignoring/cycling servers. A hostile server fabricating endless distinct txids could still stream pages — a total cap felt arbitrary, but happy to add one if you prefer.
  • Reorg mid-pagination on PoW chains: an evicted cursor txid makes esplora return an empty page (observed on Blockstream esplora: /txs/chain/{unknown-txid} answers [] with HTTP 200), ending the walk early for that scan; the next scan self-heals. On Liquid reorgs are effectively nonexistent.
  • Mempool transactions beyond esplora's first-page window are still not fetched (the API offers no mempool paging) — pre-existing limitation, now documented on get_address_history.
  • Tests: a unit test for the cursor helper and a regtest e2e (test_esplora_address_history_paging) funding one address with 30 confirmed transactions and asserting the full history syncs, at concurrency(4) to also exercise the ordered walk. The e2e runs in the GitLab docker environment (test_wollet job), not on GitHub Actions.

Context: we hit the truncation in production at DePix — merchant wallets reuse deposit addresses well past 25 transactions, and during a waterfalls outage the fallback to the vanilla esplora path synced wrong balances.

dadafros added 2 commits July 18, 2026 15:16
The esplora client read only the first page of an address's
transaction history: implementations bound it (25 confirmed
transactions on Blockstream esplora), so busier addresses got a
silently truncated history, and the scan cache then evicted the
missing transactions (txid_height_delete) — an incomplete view that
re-added and deleted the same transactions on every scan.

Follow the /txs/chain/{last_seen_txid} pages from the last confirmed
txid until a page with no confirmed transactions arrives, the only
termination condition every esplora implementation guarantees (page
sizes are configurable in the mempool/electrs fork, and its first
page shares one budget between mempool and confirmed entries, which
can crowd confirmed transactions out entirely). A repeated cursor is
rejected so a server that ignores it cannot loop the client forever.

Resolves the "TODO must handle paging" in get_scripts_history_esplora.
The address walk of the esplora full scan awaited one history request
per address sequentially, so wall-clock grew linearly with the number
of scanned addresses even though EsploraClientBuilder::concurrency
already parallelizes transaction and header downloads. Drive the same
batch through an ordered buffered stream honoring the configured
concurrency: buffered (not buffer_unordered) because callers map
results back to derivation indices positionally, and try_collect to
stop at the first error like the sequential loop did.

The per-address futures are instantiated eagerly (they stay inert
until polled; buffered still caps concurrent polling): holding an
&Address-borrowing closure inside the stream trips rustc's
higher-ranked FnOnce limitation (rust-lang/rust#89976) as soon as a
downstream async_trait consumer such as lwk_boltz must prove the
resulting future Send.

Default concurrency stays 1, so behavior is unchanged unless opted in.
@dadafros
dadafros force-pushed the faster-esplora-address-history branch from daf5d75 to b3df2c7 Compare July 20, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant