feat: Lock-free apply_block refactor by sergerad · Pull Request #2345 · 0xMiden/node

sergerad · 2026-07-15T03:05:41Z

Summary

Closes #1539.
Closes #1853.
Closes #2414.

Makes the store's block-write path lock-free for readers, and makes the resulting read-consistency rules type-enforced. Reads previously contended on RwLocks over the in-memory trees (and blocked during apply_block's DB-commit window); they now load an immutable snapshot via ArcSwap and are never blocked by writes. All reads flow through a request-scoped StateView, so a query combining tree and DB data at different chain heights is no longer expressible.

                                   LoadedState::start()
                                           │ spawns worker task, hands out one of each
             ┌─────────────────────┬───────┴────────────┬─────────────────────┐
             ▼                     ▼                    ▼                     ▼
      ┌────────────┐        ┌─────────────┐      ┌─────────────┐      ╔══════════════╗
      │ Arc<State> │        │ BlockWriter │      │ ProofWriter │      ║  WriteWorker ║
      │ (read-only,│        │ (write cap, │      │ (write cap, │      ║ (tokio task) ║
      │  shared)   │        │  1 holder)  │      │  1 holder)  │      ╚══════════════╝
      └────────────┘        └─────────────┘      └─────────────┘        owns MUTABLE
             │                     │                    │               trees: nullifier,
             │       apply_block() │      apply_proof() │               account, MMR,
             │                     │                    │               forest
             │         WriteRequest│                    ├─ commit proof       │
             │             mpsc(1) │                    │  to block store     │ per committed
             │                     ▼                    │                     │ block: builds
             │                     ═══════════▶ ════════╪═══════════▶         │ + publishes
             │                                          │                     ▼
             │                                          │              ┌────────────────┐
      State fields                                      │              │ StateSnapshot  │
      ┌───────────────────────────────────────┐         │              │  (immutable,   │
      │ latest_snapshot: Arc<ArcSwap<─────────── swap on commit ──────▶│  per block N)  │
      │                        StateSnapshot>>│         │              │  tree READER   │
      │ committed_tip_tx: watch ◀── fired by worker     │              │  views + MMR   │
      │ proven_tip:       watch ◀── advanced by ────────┘              │ + SnapshotGuard│
      │ db, block_store, block/proof caches   │                        └────────────────┘
      └───────────────────────────────────────┘                                ▲
             │                                                                 │ pins ONE
             │ view()  (wait-free ArcSwap load, one per request)               │ generation
             ▼                                                                 │
      ┌─────────────────────────────────────────────┐                          │
      │ StateView  { snapshot: Arc<StateSnapshot> ──┼──────────────────────────┘
      │              db:       Arc<Db>            } │
      │  • tip() = snapshot height                  │   ALL reads live here:
      │  • DB queries scoped by tip                 │   get_account, get_*_inputs,
      │    (ScopedBlockNum / ScopedBlockRange)      │   sync_*, get_block_header, …
      │  • trees only via block_in_place helpers    │
      └─────────────────────────────────────────────┘

  Writes flow LEFT→RIGHT (capability → worker → snapshot); reads flow DOWN (State → view →
  pinned snapshot + tip-scoped DB). Live tips (committed/proven) bypass snapshots via watch
  channels. Readers never block writes; a view keeps serving block N while the worker
  publishes N+1.

Why:

Read endpoints (sync, account/nullifier proofs, chain tip) no longer stall while a block is applied.
Removes the fragile cross-task lock choreography in apply_block (oneshot handshakes between the DB task and the in-memory update).
Snapshot-scoped reads were previously enforced only by convention — several paths read the tip and the data from different snapshots. StateView makes the scoping structural instead.

How:

Lock-free write path (state/writer/)

A single WriteWorker task owns the mutable nullifier tree, account tree, blockchain MMR, and account-state forest, processing blocks serially from an mpsc channel — no locks. In-flight writes always complete; shutdown is only observed between requests.
After each DB commit, the worker builds an immutable StateSnapshot (trees backed by read-only RocksDB snapshot views) and publishes it atomically via ArcSwap, so readers keep a consistent frozen view while the next block commits.
Db::apply_block is now a plain transaction — the oneshot allow_acquire/acquire_done synchronization is removed.

Write capabilities (state/lifecycle.rs)

LoadedState::start spawns the worker and returns the read-only Arc<State> plus non-cloneable BlockWriter/ProofWriter capabilities and a WriterTask handle, statically limiting each write path to one task. The capabilities expose no read access; tasks that read and write get Arc<State> alongside their capability.
BlockWriter::stop drains and joins the worker so tree storage is released deterministically before the data directory is re-opened or deleted (used by recover and stress-test seeding).

Type-enforced reads (state/view/)

All tree and DB reads live on StateView, pinned to one snapshot per request (State::view()). DB queries are scoped by the view's tip internally; callers cannot supply their own. RocksDB-backed trees are only reachable through block_in_place helpers; snapshot fields are only visible inside the view module.
Tip-scoped Db queries require view-issued proof types (ScopedBlockNum / ScopedBlockRange), constructible only by a StateView after validating the bound against its tip — extending the enforcement to the DB boundary itself.
Range-scoped sync queries validate range.end() <= tip themselves via a new RangeBeyondTip error (same InvalidArgument response as before); the RPC layer's range_bounds_check is deleted and pagination's chain_tip is now the tip the query actually ran against.
Fixes paths that previously took two snapshots per request (get_account, the block producer's get_tx_inputs); sync_chain_mmr clamps the proven tip to the view's tip.

Live tips (state/tip.rs)

State::committed_tip() / proven_tip() read the watch channels their writers publish to (mirroring subscribe_committed_tip / subscribe_proven_tip); the Finality enum is removed. The committed tip is published after the snapshot, so it never reports a block a fresh view cannot serve.

Observability: SnapshotGuard tracks live snapshot generations and lifetimes; warns when a snapshot outlives 10s or more than 4 generations are pinned (a leaked/slow reader pins a RocksDB snapshot).

Supporting changes: read-only reader() views for AccountStateForest / AccountTreeWithHistory (relaxed to BackendReader/SmtStorageReader bounds); state module restructured into view/ (read endpoints) and writer/ (worker + capabilities); new tracing field names allowlisted.

Changelog

[[entry]]
scope       = "node"
impact      = "changed"
description = "Store reads are lock-free: readers use atomically published in-memory snapshots and are no longer blocked while blocks are applied."

…ckfree-store-state

Mirko-von-Leipzig

Some comments; but the size of the PR does somewhat force our hand towards merging this and fixing things in post.

A more staggered/stacked approach could have been:

snapshot API + one or two example impls
more impls
lastly update apply block

I do think we can do better once we have sqlite added; but this is a decent stepping stone.

Mirko-von-Leipzig · 2026-08-03T14:35:26Z

+    let (current_block_height, store_inputs) = state
+        .with_view(async |view| {


Why do we use with_state as a closure?

Do you mean with_view? Mainly so that a set of view calls are ensured to use the same snapshot (because state.view() could return a different snapshot different calls).

Mirko-von-Leipzig · 2026-08-03T14:44:27Z

+    /// Runs a read operation over a view pinned at the current chain tip, dropping the view — and
+    /// releasing its snapshot generation — as soon as the operation completes.
+    ///
+    /// This is the required form whenever multiple reads must observe the *same* snapshot: the
+    /// closure's view is one consistent generation, whereas consecutive [`Self::view`] calls may
+    /// straddle a commit. The typical case is pairing a query with [`StateView::tip`] so a
+    /// response reports exactly the height it was served at.


tbh I don't quite understand the need for view and with_view. Why can't the caller hold the view for as many calls as they want?

Both are intended to be used so that the lifetime of the StateView (and underlying Arc<StateSnapshot>) are as short as possible.

So for example

// Bad let view = state.view(); let tip = view.tip(); // more unrelated instructions... // scope finally ends } // Good let tip = state.view().tip();

And

// Bad let view = state.view(); let x = view.get_accounts(); let y = view.compute_something(x); // more unrelated instructions... // scope finally ends } // Good // Snapshot tightly scoped state.with_view(...); // unrelated instructions don't affect lifetime of snapshot

I understand in general, but because we offer both APIs there isn't really any safety in that; its an improvement in cases where we should otherwise manually drop.

But I don't know of any such cases? I would expect a view to live for 99% of an RPC request's lifespan.

Frankly I think we should just keep view() and use an explicit block on the client side if early drops are desired.

Personally I like having view()/with_view() API because it is very explicit about the importance of limiting StateView/StateSnapshot lifetime. Up to you guys.

I'm fine with having both, too!

…ckfree-store-state

igamigo

Not a full review (hopefully by tomorrow morning), but so far it's looking good. Left just one comment for now

igamigo · 2026-08-04T03:57:05Z

+                .reader()
+                .expect("nullifier tree snapshot creation should not fail"),
+            account_tree: self.account_tree.reader(),
+            blockchain: self.blockchain.clone(),


Is this fine to do? I haven't done the math but wouldn't this self.blockchain grow into the hundreds of MBs fairly quickly (weeks/months)? Not sure there is an easy way to work around this though. This might have been accounted for already

kkovaacs · 2026-08-04T10:09:44Z

+    /// Runs a read operation over a view pinned at the current chain tip, dropping the view — and
+    /// releasing its snapshot generation — as soon as the operation completes.
+    ///
+    /// This is the required form whenever multiple reads must observe the *same* snapshot: the
+    /// closure's view is one consistent generation, whereas consecutive [`Self::view`] calls may
+    /// straddle a commit. The typical case is pairing a query with [`StateView::tip`] so a
+    /// response reports exactly the height it was served at.


Frankly I think we should just keep view() and use an explicit block on the client side if early drops are desired.

kkovaacs · 2026-08-04T11:39:42Z

+            self.nullifier_tree
+                .reader()
+                .expect("nullifier tree snapshot creation should not fail"),
+            self.blockchain.clone(),


As @igamigo has pointed out this might be problematic if the underlying MMR grows. I don't think we can avoid this clone now, but we should probably check if it's possible to implement the underlying MMR as an immutable data structure.

kkovaacs · 2026-08-04T11:46:07Z

+        // commits; queries that combine DB and in-memory data are scoped by block number.
+        let resolved_note_ids = self
+            .db
+            .apply_block(signed_block, notes, precomputed_public_states, unresolved_note_nullifiers)


This operation also does pruning, possibly removing data that is still reachable from a StateView.

At tip N, the account-tree snapshot can still serve block N-50. If N+1 commits while that view is alive, the cutoff advances to N-49 and may delete non-latest rows from N-50. The view can then return a valid N-50 witness/header but missing or incorrect historical code, vault, or storage
details. An upper-bound ScopedBlockNum cannot prevent data below that bound from disappearing.

To fix this we should either open and use an SQLite transaction for the view (I don't think this is currently possible, maybe it will be once we migrate off Diesel) or make pruning honor the oldest live snapshot height and retain everything needed by that generation.

I have gone with the latter. Although if a StateSnapshot is too far behind, it will be ignored by pruning. That is a failure state that our traces / alerts would let us know about.

For other potential problems, followup #2437

…ckfree-store-state

sergerad added 18 commits July 15, 2026 15:04

Initial apply_block lock-free refactor

7195581

Add snapshot count observability

5c7dbeb

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

b7699fc

…ckfree-store-state

Fix ui tests

46b8df6

Lint

bb3cf85

Warn about long lived snapshots

891d702

Warn on snapshot count

8c2bc02

Scape all requests

8373d9b

Lint

3a27431

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

8fa1502

…ckfree-store-state

Replace result with control flow

c21a950

Compartmentalize apply_block

0b562a8

snapshots_live fn

ddf4f15

Shutdown writer

6556067

Lint

2ad2bbf

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

09b11f8

…ckfree-store-state

Fix comments

dc1e08a

Lint

d7e6c52

sergerad requested review from Mirko-von-Leipzig and kkovaacs July 23, 2026 01:18

sergerad marked this pull request as ready for review July 23, 2026 01:18

sergerad added 8 commits July 23, 2026 09:31

Add branch comment

d14362e

Fix shutdown

d6bf7bd

Shutdown integration without token

220f911

integrate cancellation token

8a531c2

Refactor writer task management

3b648ff

Split module

aea62ce

WriterTask

7f8ebc5

Add benchmarks

c528f19

sergerad mentioned this pull request Jul 23, 2026

test: Benchmarks for lock free apply_block #2370

Merged

sergerad added 3 commits August 3, 2026 13:28

RM unnecessary db()

bd36c48

Fix scope_block

d990f4c

Fix committed tip w/o subs

adab95a

sergerad requested a review from Mirko-von-Leipzig August 3, 2026 02:21

sergerad added 2 commits August 3, 2026 19:20

Update with_view comment

862f5d9

Fmt comment

dddafb3

Mirko-von-Leipzig approved these changes Aug 3, 2026

View reviewed changes

bobbinth requested a review from igamigo August 3, 2026 17:30

sergerad mentioned this pull request Aug 3, 2026

Re-design the recovery command in the context of multi-validators #2427

Open

sergerad added 3 commits August 4, 2026 08:40

Update tip api comment

8c13ffd

Move use and mod statements

f593daf

Move cancellation token

5c8b840

sergerad mentioned this pull request Aug 3, 2026

Integrate Rpc::sync_writers into RpcMode::FullNode #2428

Open

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

1b8a079

…ckfree-store-state

igamigo reviewed Aug 4, 2026

View reviewed changes

kkovaacs reviewed Aug 4, 2026

View reviewed changes

sergerad added 6 commits August 5, 2026 07:49

Scope select_note_inclusion_proofs

96798be

RM clone block bytes

e79ea50

&mut self proof writer

7c8d92c

&mut self block writer

4e5f479

Fix rollback

767cab7

Bound pruning by snapshot generation

1dddb62

sergerad mentioned this pull request Aug 4, 2026

Reduce / remove MMR clone in apply block #2436

Open

sergerad added 3 commits August 5, 2026 11:40

Panic instead of abort

e7bae73

Merge branch 'next' of github.com:0xMiden/miden-node into sergerad-lo…

7f07029

…ckfree-store-state

Drain all tasks on cancel / failure

21c9cef

sergerad requested review from igamigo and kkovaacs August 5, 2026 00:20

sergerad mentioned this pull request Aug 5, 2026

Fix pruning related snapshot / sqlite inconsistency #2437

Open

Warn only when snapshot has been superseded

eb14eb6

		let (current_block_height, store_inputs) = state
		.with_view(async \|view\| {

Uh oh!

Conversation

sergerad commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changelog

Uh oh!

Mirko-von-Leipzig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igamigo left a comment

Choose a reason for hiding this comment

Uh oh!

igamigo Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sergerad commented Jul 15, 2026 •

edited

Loading

igamigo Aug 4, 2026 •

edited

Loading