Skip to content

feat: add fetch/unfetch to Vfs trait (xFetch/xUnfetch, iVersion 3)#81

Open
russellromney wants to merge 4 commits intoorbitinghail:mainfrom
russellromney:feat/xfetch-xunfetch
Open

feat: add fetch/unfetch to Vfs trait (xFetch/xUnfetch, iVersion 3)#81
russellromney wants to merge 4 commits intoorbitinghail:mainfrom
russellromney:feat/xfetch-xunfetch

Conversation

@russellromney
Copy link
Copy Markdown

Context

Follows from #79 and #80. PR #80 fixes the SEGFAULT by setting iVersion=2. This PR is the alternative: properly implement iVersion=3 by adding fetch/unfetch to the Vfs trait.

Changes

Add fetch() and unfetch() methods to the Vfs trait:

fn fetch(&self, handle: &mut Self::Handle, offset: i64, amt: usize)
    -> VfsResult<Option<NonNull<u8>>> { Ok(None) }

fn unfetch(&self, handle: &mut Self::Handle, offset: i64, ptr: *mut u8)
    -> VfsResult<()> { Ok(()) }

Default implementations decline mmap (fetch returns None, unfetch is a no-op). SQLite gracefully falls back to xRead. VFS implementations that want memory-mapped page reads can override fetch() to return NonNull pointers.

C shims x_fetch and x_unfetch wired into io_methods. iVersion stays at 3, now legitimately.

Use case

We're building turbolite, a SQLite VFS that serves queries from S3 with a local NVMe cache. With fetch(), we can mmap the cache file and return direct pointers to SQLite, eliminating pread syscalls for warm-cache reads (~5us to <1us per page).

Tests

4 new tests in tests/fetch_test.rs:

  • Basic write/read roundtrip with default fetch (decline mmap)
  • Concurrent WAL (1 writer + 4 readers, 3 seconds) with default fetch, 30/30 pass
  • Checkpoint under load (2500 rows, explicit TRUNCATE checkpoint)
  • iVersion=3 end-to-end validation

Relationship to #80

Either PR fixes the SEGFAULT. #80 is the minimal fix (iVersion 3 -> 2). This PR is the proper fix (implement the methods that iVersion 3 declares). They're alternatives; merging either one works. If you prefer the minimal approach, merge #80 and this becomes a follow-up feature.

Add fetch() and unfetch() methods to the Vfs trait with safe defaults:
- fetch() returns Ok(None), telling SQLite to fall back to xRead
- unfetch() is a no-op

Wire x_fetch/x_unfetch C shims into io_methods, making iVersion=3
legitimate. VFS implementations that want mmap can override fetch()
to return NonNull pointers to memory-mapped regions.

4 tests:
- Basic write/read roundtrip with default fetch
- Concurrent WAL (1W + 4R, 3 seconds) with default fetch (30/30)
- Checkpoint under load exercises xFetch code path
- Verify iVersion=3 works end-to-end
@russellromney
Copy link
Copy Markdown
Author

simplifying tests.

@russellromney
Copy link
Copy Markdown
Author

Ready for review.

Copy link
Copy Markdown
Contributor

@carlsverre carlsverre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation looks good. Can you add fetch/unfetch support to your minimal vfs to exercise it? Make sure that something asserts they are actually reached in testing. I'm not convinced from reading the docs that they are. It sounds like mmap_size has to be set?

@russellromney
Copy link
Copy Markdown
Author

Good point. Adding

Implement actual mmap-based fetch() in test VFS. Assert that SQLite
calls fetch() via atomic counter (PRAGMA mmap_size=1048576 required).

Two tests:
- test_fetch_mmap_reads: 200 rows, read back through mmap, assert fetch called
- test_fetch_survives_checkpoint: 2500 rows triggers auto-checkpoint under mmap
@russellromney
Copy link
Copy Markdown
Author

russellromney commented Apr 13, 2026

updated. test VFS now implements real mmap fetch/unfetch, and an atomic counter asserts that SQLite actually calls fetch() (requires PRAGMA mmap_size). ready for review.

Copy link
Copy Markdown
Contributor

@carlsverre carlsverre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just some small tweaks then we can land this! thank you

Comment thread tests/fetch_test.rs Outdated
Comment thread tests/fetch_test.rs
Comment thread tests/fetch_test.rs Outdated
Move fetch/unfetch counters from global statics into per-VFS
Arc<FetchCounters>. Each test gets its own counters via setup(),
safe for parallel test execution.

Add unfetch assertion to test_fetch_mmap_reads (carlsverre review).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@russellromney
Copy link
Copy Markdown
Author

All addressed, great feedback thx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants