wollet: handle esplora address history paging - #169
Open
dadafros wants to merge 2 commits into
Open
Conversation
The esplora client read only the first page of an address's
transaction history: implementations bound it (25 confirmed
transactions on Blockstream esplora), so busier addresses got a
silently truncated history, and the scan cache then evicted the
missing transactions (txid_height_delete) — an incomplete view that
re-added and deleted the same transactions on every scan.
Follow the /txs/chain/{last_seen_txid} pages from the last confirmed
txid until a page with no confirmed transactions arrives, the only
termination condition every esplora implementation guarantees (page
sizes are configurable in the mempool/electrs fork, and its first
page shares one budget between mempool and confirmed entries, which
can crowd confirmed transactions out entirely). A repeated cursor is
rejected so a server that ignores it cannot loop the client forever.
Resolves the "TODO must handle paging" in get_scripts_history_esplora.
The address walk of the esplora full scan awaited one history request per address sequentially, so wall-clock grew linearly with the number of scanned addresses even though EsploraClientBuilder::concurrency already parallelizes transaction and header downloads. Drive the same batch through an ordered buffered stream honoring the configured concurrency: buffered (not buffer_unordered) because callers map results back to derivation indices positionally, and try_collect to stop at the first error like the sequential loop did. The per-address futures are instantiated eagerly (they stay inert until polled; buffered still caps concurrent polling): holding an &Address-borrowing closure inside the stream trips rustc's higher-ranked FnOnce limitation (rust-lang/rust#89976) as soon as a downstream async_trait consumer such as lwk_boltz must prove the resulting future Send. Default concurrency stays 1, so behavior is unchanged unless opted in.
dadafros
force-pushed
the
faster-esplora-address-history
branch
from
July 20, 2026 19:17
daf5d75 to
b3df2c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes to the async esplora client's vanilla (non-waterfalls) scan path, in two commits:
1.
wollet: handle esplora address history paging— resolves theTODO must handle paginginget_scripts_history_esplora. The client read only the first page ofGET /address/{addr}/txs, so an address with more confirmed transactions than one page (25 on Blockstream esplora) synced a silently truncated history — and since the scan treats cache txids absent from the fetch as deletions (txid_height_delete), the missing transactions were also evicted from the cache, churning delete/re-add on every scan.The loop now follows
/txs/chain/{last_seen_txid}from the last confirmed txid until a page with no confirmed transactions arrives — the only termination condition every implementation guarantees: page sizes are configurable in the mempool/electrs fork, and its first page shares a single budget between mempool and confirmed entries (observed live on liquid.network: 50-entry shared first page vs blockstream.info's 25-confirmed + mempool), which can crowd confirmed entries out of the first page entirely — that case is handled with an explicit cursor-less/txs/chainrequest. A repeated cursor errors out so a server that ignores it cannot loop the client forever. This also brings the vanilla path in line with the waterfalls path, which already pages viahas_more.2.
wollet: fetch address histories concurrently— the address walk awaited one request per address sequentially, so scan wall-clock grew linearly with address count even thoughEsploraClientBuilder::concurrencyalready parallelizes transaction and header downloads. The batch now runs through.buffered(self.concurrency)(ordered — callers map results back to derivation indices positionally) withtry_collectkeeping the sequential loop's fail-fast semantics. Default concurrency remains 1: no behavior change unless opted in.Notes
/txs/chain/{unknown-txid}answers[]with HTTP 200), ending the walk early for that scan; the next scan self-heals. On Liquid reorgs are effectively nonexistent.get_address_history.test_esplora_address_history_paging) funding one address with 30 confirmed transactions and asserting the full history syncs, atconcurrency(4)to also exercise the ordered walk. The e2e runs in the GitLab docker environment (test_wolletjob), not on GitHub Actions.Context: we hit the truncation in production at DePix — merchant wallets reuse deposit addresses well past 25 transactions, and during a waterfalls outage the fallback to the vanilla esplora path synced wrong balances.