Skip to content

dash-spv gets permanently stuck after a testnet reorg and blocks masternode sync #986

Description

@llbartekll

Summary

dash-spv can become permanently stuck after a shallow testnet reorg. When the persisted block-header tip is no longer part of the peer's active chain, header synchronization repeatedly requests headers using that stale tip as a single-entry locator. Peers that do not recognize it respond from genesis, every response is rejected with no segment matched, and the same request is retried indefinitely.

Because BlockHeaderSyncComplete is never emitted, masternode synchronization remains in WaitForEvents at height 0 and never requests QRInfo or masternode-list diffs.

Environment

  • Network: Testnet
  • dash-spv: 0.45.0
  • Revision: 3d13d9838c80
  • Client: DashWallet iOS through the Swift SDK, running in an iOS Simulator
  • Persisted header tip: height 1542169, hash 0000011367b7be55c100ed886d89e976495f698206b3dcdd2358db5f20743b50
  • Peer height after restart: 1542198

Observed sequence

  1. The client synchronized successfully through height 1542169, including the masternode list:

    Masternode sync complete at height 1542169
    Sync complete in 80.83s
    
  2. It then verified a ChainLock for the common ancestor at height 1542168:

    ChainLock signature verified for height 1542168
    
  3. Peers announced a replacement two-header branch anchored at that block. The pipeline could not associate it with its segment, whose active tip was the now-stale block at 1542169:

    Received 2 headers with prev_hash 0000007987f44d2020a5bc835d3d8fffc38d0035439211e3ae60ffcb96065db9 but no segment matched
    
  4. A later ChainLock at height 1542171 was successfully verified, while the header manager remained at 1542169:

    ChainLock signature verified for height 1542171
    Headers: Synced 1542169/1542169
    
  5. After restarting, the header manager restored the stale tip and created its only segment from that hash:

    dash-spv 0.45.0 (3d13d9838c80)
    BlockHeadersManager initialized at height 1542169
    Created segment 0: 1542169 -> None (start_hash: 0000011367b7be55c100ed886d89e976495f698206b3dcdd2358db5f20743b50)
    
  6. Multiple peers repeatedly returned 8,000 headers beginning after the testnet genesis hash because they could not match the stale locator. Each response was rejected and retried after 30 seconds:

    Decompressed 8000 headers ... forwarding as regular Headers
    Received 8000 headers with prev_hash 00000bafbc94add76cb75e2ec92894837288a481e5c005f6563d91623bf8bc2c but no segment matched
    Segment 0: request timed out for hash 0000011367b7be55c100ed886d89e976495f698206b3dcdd2358db5f20743b50, will retry
    Retrying item (attempt 1)
    

    The loop continued for dozens of attempts and reproduced with different peers.

  7. Masternode synchronization never started its actual requests:

    Masternodes: WaitForEvents 0/1542198 | diffs_processed: 0, qr_infos_requested: 0
    

Expected behavior

When the persisted tip is no longer on the active chain, the client should use a multi-entry block locator to find the latest common ancestor, validate the replacement branch, switch/roll back the active stored chain as required, and resume normal synchronization. Header completion should then unblock masternode synchronization.

Actual behavior

The client retries forever from the stale hash, does not recover across restarts, and leaves header-dependent managers blocked. In the iOS app this presents as the masternode list remaining at 0 / <peer height>.

Related work

PR #803 adds an exponential-backoff block locator and staged fork detection. In particular, commit 66f6967b describes and tests the stale-tip retry path seen here. However, the PR currently states that applying the detected branch to reorg the active chain is future work, so it appears to address detection and locator recovery but not the complete recovery path required by this issue.

Additional material

Full logs covering the successful sync, the reorg, and the persistent post-restart retry loop are available and will be attached separately. A simulator screenshot showing the masternode list stuck at zero will also be attached separately.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions