fix(io): publish object store metrics for local reads and writes#7994
Open
wjones127 wants to merge 1 commit into
Open
fix(io): publish object store metrics for local reads and writes#7994wjones127 wants to merge 1 commit into
wjones127 wants to merge 1 commit into
Conversation
github-actions
Bot
added
A-encoding
Contributor
📝 WalkthroughWalkthroughLocal filesystem readers, writers, io_uring paths, and copy/delete shortcuts now record object-store metrics through ChangesLocal I/O metrics
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ObjectStore
participant LocalObjectReader
participant LocalFilesystem
participant IOTracker
ObjectStore->>LocalObjectReader: open with local_io_tracker()
LocalObjectReader->>IOTracker: begin_io("get")
LocalObjectReader->>LocalFilesystem: perform blocking read
LocalFilesystem-->>LocalObjectReader: return Result and bytes
LocalObjectReader->>IOTracker: record outcome and byte count
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The optimized local paths — `LocalWriter`, `LocalObjectReader`, the io_uring readers, and the local `copy` / recursive-delete shortcuts — go straight to the filesystem, so they never reach `MeteredObjectStore` and published no metrics at all. Writing a dataset to a local path produced zero object store metrics even though the IO tracker recorded it. `IOTracker` now carries the metrics `base` label of the store it belongs to and hands out an `IoMetricsGuard` for IO that bypasses the `object_store` layer. Every such path records requests, bytes, latency, errors and the in-flight gauge under the same labels the metered store uses, so local IO aggregates with cloud IO. A caller-supplied `ObjectStore` (the deprecated `ObjectStoreParams::object_store`) is now metered too. `IoStats` is unchanged: the local `copy`, recursive delete and size lookup were never counted there and still aren't, so existing IO assertions are unaffected. Fixes lance-format#7993 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wjones127
force-pushed
the
fix-local-writer-metrics
branch
from
July 24, 2026 21:53
abda9ee to
4b64b21
Compare
wjones127
marked this pull request as ready for review
July 24, 2026 21:54
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
🟡 Other comments (1)
rust/lance-io/src/object_store/metrics.rs-1517-1536 (1)
1517-1536: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAssert on the error variant/message, not just
is_err().
test_local_read_error_is_countedonly checksreader.get_range(0..100).await.is_err()). As per coding guidelines: "Assert on both the error variant and the message content in tests; do not check onlyis_err()."✅ Proposed fix
let reader = store.open(&path).await.unwrap(); // Reading past the end of the file fails. - assert!(reader.get_range(0..100).await.is_err()); + let err = reader.get_range(0..100).await.unwrap_err(); + assert!( + matches!(err, object_store::Error::Generic { .. }), + "unexpected error variant: {err:?}" + );🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-io/src/object_store/metrics.rs` around lines 1517 - 1536, Update test_local_read_error_is_counted to capture the get_range error and assert both its expected error variant and message content instead of checking only is_err(). Preserve the existing metrics assertions and the out-of-bounds read scenario.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/lance-io/src/utils/tracking_store.rs`:
- Around line 104-144: The local-bypass metrics label must honor BaseLabelMode
scoping and avoid per-operation allocation. In
rust/lance-io/src/utils/tracking_store.rs lines 104-144, update
with_metrics_base to materialize the same scoped label used by
ObjectStoreMetricsExt::metered(), including the configured off/full/scheme
behavior. In rust/lance-io/src/object_store.rs lines 653-661, precompute and
cache the scoped local IOTracker alongside the existing io_tracker/store_prefix
state, then reuse it across open, open_with_size, create, copy_impl, and
remove_dir_all instead of calling local_io_tracker() for each operation.
---
Other comments:
In `@rust/lance-io/src/object_store/metrics.rs`:
- Around line 1517-1536: Update test_local_read_error_is_counted to capture the
get_range error and assert both its expected error variant and message content
instead of checking only is_err(). Preserve the existing metrics assertions and
the out-of-bounds read scenario.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: 7e76efbb-517b-4558-9642-4ee3f9999754
📒 Files selected for processing (8)
rust/lance-io/src/local.rsrust/lance-io/src/object_reader.rsrust/lance-io/src/object_store.rsrust/lance-io/src/object_store/metrics.rsrust/lance-io/src/object_writer.rsrust/lance-io/src/uring/current_thread.rsrust/lance-io/src/uring/reader.rsrust/lance-io/src/utils/tracking_store.rs
Comment on lines
+104
to
+144
|
|
||
| /// Label the metrics published through [`Self::begin_io`] with the prefix of | ||
| /// the store this tracker belongs to, so IO that bypasses the `object_store` | ||
| /// layer carries the same `base` label as the store's metered operations. | ||
| #[cfg(feature = "metrics")] | ||
| pub fn with_metrics_base(mut self, base: &str) -> Self { | ||
| self.metrics_base = Some(base.into()); | ||
| self | ||
| } | ||
|
|
||
| /// Without the `metrics` feature there is nothing to label. | ||
| #[cfg(not(feature = "metrics"))] | ||
| pub fn with_metrics_base(self, _base: &str) -> Self { | ||
| self | ||
| } | ||
|
|
||
| /// Begin an operation that talks to storage without going through the | ||
| /// `object_store` layer, and so is invisible to the `MeteredObjectStore` | ||
| /// wrapper: the optimized local reads and writes go straight to the | ||
| /// filesystem. `operation` must be one of the labels that wrapper uses | ||
| /// (`get`, `put`, `head`, ...) so this IO aggregates with the rest. | ||
| /// | ||
| /// The returned guard keeps the in-flight gauge raised until it is dropped. | ||
| #[cfg(feature = "metrics")] | ||
| pub fn begin_io(&self, operation: &'static str) -> IoMetricsGuard { | ||
| IoMetricsGuard { | ||
| state: self.metrics_base.as_ref().map(|base| IoMetricsState { | ||
| _in_flight: InFlightGuard::new(base, operation), | ||
| base: base.clone(), | ||
| operation, | ||
| start: Instant::now(), | ||
| }), | ||
| } | ||
| } | ||
|
|
||
| /// Without the `metrics` feature there is nothing to publish. | ||
| #[cfg(not(feature = "metrics"))] | ||
| pub fn begin_io(&self, _operation: &'static str) -> IoMetricsGuard { | ||
| IoMetricsGuard {} | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Local-bypass metrics label ignores BaseLabelMode scoping and is re-allocated on every local I/O call.
with_metrics_base stores the raw, unscoped store_prefix as-is, unlike ObjectStoreMetricsExt::metered() which applies scoped_base(parse_base_label_mode(...), prefix) (see metrics.rs test_scoped_base/test_base_label_defaults_to_scheme). Because local_io_tracker() calls with_metrics_base afresh on every local read/write/copy/delete, this also means: (1) the runtime LANCE_METRICS_BASE_LABEL_MODE (off/full/scheme) opt-out isn't honored for local bypass paths the way it is for cloud/MeteredObjectStore paths, and (2) a new Arc<str> is allocated on every single local I/O call instead of once per ObjectStore.
rust/lance-io/src/utils/tracking_store.rs#L104-L144: apply the sameBaseLabelMode/scoped_baselogic used by.metered()when materializingmetrics_base(or accept an already-scoped label from the caller) instead of storing the raw prefix.rust/lance-io/src/object_store.rs#L653-L661: precompute and cache the scoped localIOTrackeronce (e.g., alongside theio_tracker/store_prefixfields at construction) instead of callinglocal_io_tracker()— and re-allocating — on everyopen/open_with_size/create/copy_impl/remove_dir_allcall.
📍 Affects 2 files
rust/lance-io/src/utils/tracking_store.rs#L104-L144(this comment)rust/lance-io/src/object_store.rs#L653-L661
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance-io/src/utils/tracking_store.rs` around lines 104 - 144, The
local-bypass metrics label must honor BaseLabelMode scoping and avoid
per-operation allocation. In rust/lance-io/src/utils/tracking_store.rs lines
104-144, update with_metrics_base to materialize the same scoped label used by
ObjectStoreMetricsExt::metered(), including the configured off/full/scheme
behavior. In rust/lance-io/src/object_store.rs lines 653-661, precompute and
cache the scoped local IOTracker alongside the existing io_tracker/store_prefix
state, then reuse it across open, open_with_size, create, copy_impl, and
remove_dir_all instead of calling local_io_tracker() for each operation.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The optimized local paths —
LocalWriter,LocalObjectReader, the io_uring readers, and the localcopy/ recursive-delete shortcuts — go straight to the filesystem, so they never reachMeteredObjectStoreand published no metrics at all. Writing a dataset to a local path produced zero object store metrics even though the IO tracker recorded it.IOTrackernow carries the metricsbaselabel of the store it belongs to (its store prefix, the same valueMeteredObjectStoreis given) and hands out anIoMetricsGuardfor IO that bypasses theobject_storelayer. Each bypassing path now records requests, bytes, latency, errors and the in-flight gauge under the same labels, so local IO aggregates with cloud IO:LocalWriter(oneputper file, from open to durable at its final path)putLocalObjectReader::get_range/get_all/ streamed chunksgetLocalObjectReader::size(the local equivalent of a HEAD)headUringReader/UringCurrentThreadReaderget_range/get_allgetObjectStore::copycopyObjectStore::remove_dir_all(one request, likedelete_stream)deleteTwo things worth a second opinion:
ObjectStoreParams::object_store, deprecated) got no metered wrapper at all, so it published nothing either. It is now wrapped like a registry-built store. This would double-count if a caller passed in a store that Lance had already metered.IoStatsis left alone. The localcopy, recursive delete and size lookup were never counted there and still aren't, so existing IO assertions are unaffected; adding them would shift IO counts across the test suite.The io_uring readers have no coverage here — they need Linux plus a working ring, so the new tests exercise the non-uring local paths only.
Fixes #7993