libext: ext4 sequential-read performance (read-ahead + async prefetch) - #1447
libext: ext4 sequential-read performance (read-ahead + async prefetch)#1447gburd wants to merge 9 commits into
Conversation
103c0ed to
3e8cfd5
Compare
|
I see it is a draft that depends on #1431, which I hope to merge soon. I am really surprised by the performance results you list here - look very, very good compared to Linux. I am interested in what exactly your setup is; you mention nmve - emulated one? Do you compare to a Linux guest? On my machine, when I run the fio read test with ext and compare it to zfs on OSv, I get pretty terrible results:
1.5 K slower When I increase the test block size to 128K from the default 4K, it gets a little better:
-ext 150-200 times slower. You mention the baseline. Is it what you see before your changes or after? |
|
Thanks for digging in, and the reconciliation is worth doing carefully because I think we are measuring two different things. The setupThe numbers in the PR body are an A/B of ext-before vs ext-after this branch's four perf commits, single-stream, on the same disk:
So "baseline" in the table means ext on this branch's base (#1431), before the read-ahead/prefetch commits vs ext with them. It is not ext-vs-zfs. Why your fio run shows the oppositeI think there are three separate reasons your
Suggested apples-to-applesIf you build this branch's tip and run I do not want to overclaim: this closes most of the single-stream sequential gap for lwext4, it exceeds the 128K O_DIRECT single-stream ceiling on this host, and it is still ~within 2x of Linux buffered read and behind zfs. Multi-stream and the ext-vs-zfs gap are follow-ups. |
Two gaps in the ext (lwext4) filesystem module for durable, real-world use. 1. fsync durability. ext mounts with lwext4 block-cache write-back enabled, so a write only reaches the disk when the block cache is flushed -- but vop_fsync was vop_nullop, so fsync(2)/fdatasync(2) on an ext file persisted nothing. Implement ext_fsync() to flush the device's block cache (like ext_sync does at unmount), making written data durable. The cache is shared per device, so this persists the file along with any other dirty buffers, which is correct if slightly more than the theoretical per-inode minimum. 2. Page-cache bridge (vop_cache). The vop_cache slot was null, so mmap faults on ext files went through the block layer on every fault, unlike ROFS and ZFS which populate the shared page cache. Add ext_map_cached_page(): on a VOP_CACHE call it reads one page-aligned page of file data into a freshly allocated page and hands it to pagecache::map_read_cached_page(), warming the read cache so subsequent faults and readahead are served from it. This is an allocate-and-copy bridge (lwext4's block-cache buffers are not page-aligned/shareable the way ROFS's read-around cache is); a zero-copy borrow-and-pin bridge like the ZFS ARC one can follow if it shows up hot. Because the module is built with -fno-rtti and <osv/pagecache.hh> pulls in <osv/trace.hh> (which uses typeid), the two page-cache symbols we need are declared minimally instead of including the heavy header (the uio carries the hashkey opaquely, so its layout is never needed). Add tests/tst-ext4-rw.cc: mmap a pre-populated ext4 file and verify the pattern survives the vop_cache bridge (first fault + cached re-read), then write a file, fsync it, and read it back (plus fdatasync and a fresh re-open). Verified on OSv under KVM with an ext4 second disk (created with mkfs.ext4 -b 4096 -O ^64bit,^metadata_csum, which lwext4 supports). Known pre-existing limitation (not introduced here, out of scope for this PR): libext's inode-delete path does not set the inode dtime, so Linux e2fsck flags "deleted inode has zero dtime" on a disk after OSv deletes a file. A fresh disk that OSv only reads/writes+fsyncs (no delete) fscks clean. Tracked as a follow-up in the ext write-path correctness work.
Follow-up to the fsync/page-cache work: libext's inode-delete path freed the inode from the bitmap but never set its on-disk deletion time (dtime), so after OSv created and deleted a file, Linux e2fsck flagged "Deleted inode NN has zero dtime" and reported the filesystem as still having errors. lwext4's own delete path marks the inode with ext4_inode_set_del_time(inode, -1L) before ext4_fs_free_inode(), but that symbol is not exported from liblwext4.so. Add a small ext_mark_inode_deleted() helper that sets inode->deletion_time = 0xffffffff directly (byte-order invariant, so no to_le32() needed) and call it before each ext4_fs_free_inode() in the module (unlink, rmdir, delete-on-last-close, delete-outstanding-on-unmount, and the dir_link allocation-rollback path). Verified: after OSv boots an ext4 second disk, mmaps/reads a file, writes and fsyncs another, then deletes it, `e2fsck -n -f` on the disk now exits 0 (clean), where before it reported the zero-dtime error. tst-ext4-rw still passes.
A filesystem-agnostic micro-benchmark (tst-ext4-bench) measuring sequential write+fsync, sequential read, and 4K random read throughput in MB/s, so the ext4/libext path can be compared A/B against raw virtio-blk and against Linux ext4 on the same storage. Point its argument at any mount (ext, zfs, rofs) for a cross-filesystem comparison.
ext_read allocated a full-size aligned bounce buffer, had lwext4 read into it, then uiomove()'d it to the caller - a per-read malloc plus a second full-size memcpy on every read; ext_write mirrored this in reverse. On the common path (a single contiguous iovec) hand lwext4 the caller's buffer directly. This removes the allocation and one memcpy per read/write and fixes a latent free()/free_contiguous_aligned() mismatch in both error paths. It is a correctness/cleanup change: an A/B on local NVMe showed throughput unchanged (the memcpy is not the bottleneck), which confirms the sequential-read gap vs Linux is the absence of async read-ahead, not copy overhead - that is a separate, larger change (see the ext4 perf notes).
The local-NVMe A/B showed ext4 sequential reads at ~33% of Linux while writes were at parity and random reads ~86%. The gap is that ext_read issues one synchronous bio per read() with no prefetch (ext4_blocks_get_direct bypasses lwext4's block cache), so a stream of small sequential reads cannot keep the device pipeline busy - larger reads were measurably faster purely from amortizing the per-read() round-trip. Add a per-vnode sequential read-ahead cache (in ext_vdata): on a detected sequential single-iovec read smaller than the window, fill a 1 MiB window from disk once and serve that read and subsequent in-window reads from it (a memcpy, no I/O). This turns N small synchronous reads into 1 large read + N copies. The window is invalidated on any write to the file and freed when the vnode goes inactive; access is serialized by a per-vnode mutex. tst-ext4-bench now writes an absolute-offset pattern and verifies every byte on the sequential read, so the benchmark doubles as a read-ahead correctness test (no VERIFY FAIL = the cache returns correct data across window boundaries).
The synchronous 1 MiB read-ahead (6a7990d) reads a window, lets the app consume it, then reads the next window - so the NVMe queue goes idle during the app's compute. Linux keeps the device busy via async prefetch. Add double buffering: two 1 MiB windows (cur + next) and a per-vnode worker pthread. While the app consumes cur (served by memcpy), the worker prefetches next in the background. When a read crosses out of cur, next is promoted to cur and the following window is kicked off. On a sequential pattern the next window is already in memory, keeping the device queue full and hiding read latency behind compute. Correctness: writes and truncation cancel any in-flight prefetch and drop both windows (ext_ra_invalidate); the worker uses its own inode_ref (lwext4 locks its block cache internally); ~ext_vdata joins the worker and frees both buffers on vnode inactive. The tst-ext4-bench byte-for-byte verify passes (no VERIFY FAIL) for 64K/128K/256K reads.
The async double-buffered read-ahead only kept ~1-2 windows ahead of the consumer, so a single sequential stream drove the NVMe queue to depth ~1-2 and topped out ~1.25 GB/s vs Linux's ~2.3 GB/s (Linux issues many concurrent readahead requests, keeping the device queue deep). Replace the two-window (cur/next) scheme with a ring of RA_WINDOWS windows (256 KiB x 8 = 2 MiB per open file, same memory as before) filled by a pool of RA_WORKERS worker pthreads. Each worker calls the existing synchronous ext_internal_read() into a ring slot, so RA_WORKERS fills are in flight at once -> device queue depth ~RA_WORKERS. As soon as the consumer drains a window the ring slides forward and re-arms that slot for the window RA_WINDOWS ahead, keeping the pipeline full. Correctness: per-slot state (EMPTY/FILLING/READY) + a ring generation. All worker-shared fields are under ra_cvmtx; the read path holds ra_lock (outer) and takes ra_cvmtx to inspect slots. Workers never take ra_lock, so no deadlock. Seek/write/truncate bump the generation and drain in-flight fills before reusing buffers (no use-after-free); stale worker results whose generation no longer matches are discarded. Reads larger than a window or spanning two windows fall through to the direct read path. ~ext_vdata broadcasts shutdown, joins all workers, then frees the buffers. Reuses lwext4's existing bio path (ext_internal_read -> ext4_blocks_get_direct -> one bio + bio_wait) rather than issuing raw bios, which would mean re-implementing lwext4's extent->block mapping; a worker pool blocked in bio_wait maps directly to device queue depth and keeps all that code correct. Verified with tests/tst-ext4-bench.cc (absolute-offset byte verification) at bs 4K/64K/128K/256K/262143/1M and files smaller than the ring: no VERIFY FAIL.
Sweep on a local-NVMe host, ext4 4K/no-journal, O_DIRECT so reads hit the device, 512 MiB, median of 3): config 64K 128K 256K prefetch mem/file N=1 M=1 ~0.69 ~0.85 ~0.85 256 KiB (~QD1) N=4 M=2 ~1.20 - ~1.28 1 MiB N=4 M=4 ~1.26 ~1.23 ~1.29 1 MiB <- knee N=8 M=4 ~1.28 ~1.28 ~1.31 2 MiB N=8 M=8 ~1.28 ~1.28 ~1.29 2 MiB N=16 M=16 ~1.25 - ~1.21 4 MiB Throughput climbs steeply from N=1/M=1 to the knee at N=4/M=4 (~1.28 GB/s) and then plateaus: deeper rings or more workers give no further gain. The cap is downstream of this code -- OSv's virtio-blk make_request() serialises submission under a single _lock and a single virtqueue, so effective device queue depth tops out at ~2-4 regardless of how many prefetch windows are in flight. (Linux fio O_DIRECT on the same raw NVMe: ~1.32 GB/s @128k QD1, ~1.68 GB/s @128k QD>=2, i.e. the device itself saturates at QD2; the remaining OSv gap is the guest virtio-blk path, not prefetch depth.) So N=4/M=4 is the sweet spot: same throughput as the deeper configs at half the memory (1 MiB vs 2 MiB per open file). It also matches the prior 2-window async design (~1.25 GB/s here) while using a general N-window ring. Make N and M build-time tunables (-DRA_WINDOWS_N / -DRA_WORKERS_M via $(RA_FLAGS)) so the sweep is reproducible when the virtio-blk submission path is later parallelised. No VERIFY FAIL at any config or block size (4K/64K/128K/256K/262143/1M, files smaller than the ring).
gburd
changed the title
|
Rebased onto current master (0e34e4d) and the body rewritten. Summary of what changed and why one number is gone. Standalone now. #1431 is merged upstream as af27ba4, so the two fsync/dtime commits that used to sit at the base of this branch are dropped and this reduces to the six performance commits. Title prefix removed. Rebased branch. Force-push is not available to me on this PR's head branch, so the rebase is on a new branch rather than an update in place:
Retarget this PR to that branch, or say the word and I will open a fresh one. Three things conflicted against current master and are resolved:
One number withdrawn. The earlier revision cited roughly 620 MB/s for sequential write with fsync and used it to claim write parity with Linux. That was measured when Read numbers retained, explicitly pending re-validation. They were taken on a pre-af27ba4b7 base. I checked whether af27ba4 could have flattered them and believe it cannot, for reasons a reviewer can verify: Staying in draft until both are re-run on a 2-socket bare-metal host with local NVMe on a base containing af27ba4: the 64K/128K/256K read sweep, and a fresh write-with-real-fsync number. I would rather leave this draft with an honest gap than mark it ready on a number I know was measured against a no-op fsync. |
2 participants
Important
Draft: one previously cited number has been withdrawn as invalid. See "Withdrawn" below. The read results are retained but were measured on a base predating af27ba4, so they are pending re-validation. Not ready to merge until re-measured.
#1431 is now merged upstream as af27ba4, so this no longer depends on it and stands alone. Rebased onto current master (0e34e4d); the two fsync/dtime commits that used to sit at the base of this branch are gone, since upstream now carries them as af27ba4 and bf16817.
Rebased branch, since force-push is not available to me on this PR's head branch:
pr/ext4-perf-v28f95be01e2ecf9192b279aacf0370fd2e1157379git merge-tree --write-tree --messages upstream/master pr/ext4-perf-v2exits 0 with no conflictsThe gap this addresses
ext_readissued one synchronous bio perread()with no prefetch (ext4_blocks_get_directbypasses lwext4's block cache), so a stream of sequential reads left the NVMe queue idle during application compute. Linux keeps the device busy with multi-window async read-ahead. Baseline ext4 sequential read was roughly 560 MB/s against Linux's ~2300 MB/s on the same device, while 4K random read was already at or above Linux.Changes (6 commits)
tst-ext4-bench): sequential write+fsync / sequential read / 4K random, reported in MB/s, filesystem-agnostic. Writes an absolute-offset pattern and verifies every byte on read, so it doubles as a read-path correctness test.uiomovecopy on the single-iovec fast path; also fixes a latentfree()/free_contiguous_aligned()mismatch. Correctness cleanup, not the bottleneck.nextwhile the application consumescur. A worker thread (not rawbio_done) reusesext_internal_readverbatim; writes and truncation invalidate,~ext_vdatajoins the worker and frees.RA_WINDOWS_N/RA_WORKERS_Moverridable at build time.Read results (pre-af27ba4b7 base, retained but pending re-validation)
Measured A/B on a 2-socket x86-64 bare-metal host booting OSv under KVM against local NVMe, ext4 4K/no-journal, median of 3, no VERIFY FAIL:
Reference on the same NVMe: Linux ext4 buffered read ~2300 MB/s; NVMe single-stream O_DIRECT ceiling 1.5 GB/s at 128K, 1.9 GB/s at 1M. The remaining gap to Linux is queue depth: Linux prefetches many windows, this prefetches a bounded ring, and virtio-blk's
make_request()serialises submission under a single lock, which caps effective device queue depth at roughly 2 to 4.Why these read numbers are probably still valid on the current base, stated as an argument a reviewer can check rather than something to take on trust:
ext_fsyncfromvop_nullopto a flush of lwext4's block cache, and addedvop_cachefor mmap page-cache warming.ext_internal_readreaches the device viaext4_blocks_get_direct->ext4_bdif_bread, which addresses the block device directly and never consults the cacheext_fsyncflushes.read()/pread(), not mmap, sovop_cacheis not on their path either.fsync()is called only insidebench_write's timing window (tst-ext4-bench.cc:56); neither read phase calls it.Caveat that keeps this "probably" rather than "confirmed": the benchmark writes a file and then reads that same file. With a real fsync the dirty state at read time is different, so the read number could shift second-order even though the mechanism above does not touch the read path. Likely intact is not measured.
Withdrawn
The earlier revision of this PR cited roughly 620 MB/s for sequential write with fsync, and used it to claim write throughput was at parity with Linux. Both the number and the parity claim are withdrawn.
That measurement was taken on a base where
ext_fsyncwas#define ext_fsync ((vnop_fsync_t)vop_nullop), sofsync()returned without flushing anything and the benchmark's timer stopped before the data was durable. It was not measuring durable write throughput. Sinceext_fsyncis now a real block-cache flush (af27ba4), the figure is invalid on the current base and I am not restating it with a caveat. There is currently no supported claim about write throughput in this PR.Nothing in this series targets the write path, so this is a withdrawn measurement rather than a regression, but the honest position is that write performance on the current base is unmeasured.
To re-validate
On a 2-socket x86-64 bare-metal host (KVM, local NVMe, ext4 4K/no-journal), with a base containing af27ba4:
Staying in draft until both are done.