feat(namespace): idempotent create_table_version with strict CAS and dotfile-safe reserved marker#7940
Conversation
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe directory namespace now publishes reserved markers through staging and conditional creation, and creates table versions with strict sequential CAS, idempotent identical-content retries, conflict detection, and create-only publication semantics. ChangesNamespace publication
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant create_table_version
participant ObjectStore
participant ManifestResolver
Client->>create_table_version: submit version and staging manifest
create_table_version->>ObjectStore: head final manifest
ObjectStore-->>create_table_version: existing or missing manifest
create_table_version->>ManifestResolver: compare staged and final bytes
ManifestResolver-->>create_table_version: success or ConcurrentModification
create_table_version->>ObjectStore: create-only publish
ObjectStore-->>Client: published manifest response
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
rust/lance-namespace-impls/src/dir.rs (1)
11308-11312: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert on the error variant, not just the message.
test_create_table_version_conflictandtest_create_table_version_cas_rejects_gaponly check the error string. This file already has anamespace_code(&err)helper; assertingErrorCode::ConcurrentModificationin addition to the message makes these tests robust against message wording changes.As per coding guidelines: "Assert on both the error variant and the message content in tests; do not check only
is_err()."Also applies to: 11413-11417
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-namespace-impls/src/dir.rs` around lines 11308 - 11312, Update test_create_table_version_conflict and test_create_table_version_cas_rejects_gap to assert that namespace_code(&err) equals ErrorCode::ConcurrentModification, while retaining the existing message-content checks for “already exists” or “ConcurrentModification”.Source: Coding guidelines
rust/lance-namespace-impls/src/dir/manifest.rs (1)
3493-3576: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDeduplicate the dotfile-safe marker publication logic. Both sites implement the identical stage-non-dot-sibling →
rename_if_not_exists→PutMode::Createfallback → best-effort cleanup flow; keeping two copies risks divergent fixes to this security/correctness-sensitive path.
rust/lance-namespace-impls/src/dir/manifest.rs#L3493-L3576: replace the inline staging/publish block indeclare_tablewith a call to a shared helper (e.g. a free functionput_marker_file_atomic(object_store, path, description)in the crate).rust/lance-namespace-impls/src/dir.rs#L2685-L2751: extract the body ofDirectoryNamespace::put_marker_file_atomicinto that shared helper so both callers stay in sync.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-namespace-impls/src/dir/manifest.rs` around lines 3493 - 3576, Deduplicate marker publication by extracting the existing DirectoryNamespace::put_marker_file_atomic implementation into a shared crate-level helper, preserving staging, conditional rename, PutMode::Create fallback, and best-effort cleanup. In rust/lance-namespace-impls/src/dir.rs#2685-2751, move the method body into the helper and update the method to use it if needed; in rust/lance-namespace-impls/src/dir/manifest.rs#3493-3576, replace the inline declare_table flow with the shared helper, passing the object store, marker path, and description while preserving current error context.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/lance-namespace-impls/src/dir/manifest.rs`:
- Around line 3531-3546: Move the inline object_store import from the
declare_table fallback arm to the file-level imports, adding PutMode and
PutOptions alongside the existing ObjectStoreError and path::Path imports.
Remove the function-body use declaration while leaving the put_opts logic
unchanged.
---
Nitpick comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 11308-11312: Update test_create_table_version_conflict and
test_create_table_version_cas_rejects_gap to assert that namespace_code(&err)
equals ErrorCode::ConcurrentModification, while retaining the existing
message-content checks for “already exists” or “ConcurrentModification”.
In `@rust/lance-namespace-impls/src/dir/manifest.rs`:
- Around line 3493-3576: Deduplicate marker publication by extracting the
existing DirectoryNamespace::put_marker_file_atomic implementation into a shared
crate-level helper, preserving staging, conditional rename, PutMode::Create
fallback, and best-effort cleanup. In
rust/lance-namespace-impls/src/dir.rs#2685-2751, move the method body into the
helper and update the method to use it if needed; in
rust/lance-namespace-impls/src/dir/manifest.rs#3493-3576, replace the inline
declare_table flow with the shared helper, passing the object store, marker
path, and description while preserving current error context.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: 5a061bb4-d017-42be-8f71-3666f88028b9
📒 Files selected for processing (2)
rust/lance-namespace-impls/src/dir.rsrust/lance-namespace-impls/src/dir/manifest.rs
| Err(ObjectStoreError::NotImplemented { .. }) | ||
| | Err(ObjectStoreError::NotSupported { .. }) => { | ||
| use object_store::{PutMode, PutOptions}; | ||
| let result = self | ||
| .object_store | ||
| .inner | ||
| .put_opts( | ||
| &reserved_file_path, | ||
| Bytes::from_static(b"reserved").into(), | ||
| PutOptions { | ||
| mode: PutMode::Create, | ||
| ..Default::default() | ||
| }, | ||
| ) | ||
| .await | ||
| .map(|_| ()); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Move use object_store::{PutMode, PutOptions}; to the top of the file.
This use is declared inside the declare_table fallback arm. The top-level import (Line 63) already brings in ObjectStoreError and path::Path; add PutMode/PutOptions there.
As per coding guidelines: "Place use imports at the top of the file, not inline within function bodies."
♻️ Proposed change
-use object_store::{Error as ObjectStoreError, ObjectStoreExt, path::Path};
+use object_store::{Error as ObjectStoreError, ObjectStoreExt, PutMode, PutOptions, path::Path}; Err(ObjectStoreError::NotImplemented { .. })
| Err(ObjectStoreError::NotSupported { .. }) => {
- use object_store::{PutMode, PutOptions};
let result = self📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Err(ObjectStoreError::NotImplemented { .. }) | |
| | Err(ObjectStoreError::NotSupported { .. }) => { | |
| use object_store::{PutMode, PutOptions}; | |
| let result = self | |
| .object_store | |
| .inner | |
| .put_opts( | |
| &reserved_file_path, | |
| Bytes::from_static(b"reserved").into(), | |
| PutOptions { | |
| mode: PutMode::Create, | |
| ..Default::default() | |
| }, | |
| ) | |
| .await | |
| .map(|_| ()); | |
| Err(ObjectStoreError::NotImplemented { .. }) | |
| | Err(ObjectStoreError::NotSupported { .. }) => { | |
| let result = self | |
| .object_store | |
| .inner | |
| .put_opts( | |
| &reserved_file_path, | |
| Bytes::from_static(b"reserved").into(), | |
| PutOptions { | |
| mode: PutMode::Create, | |
| ..Default::default() | |
| }, | |
| ) | |
| .await | |
| .map(|_| ()); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@rust/lance-namespace-impls/src/dir/manifest.rs` around lines 3531 - 3546,
Move the inline object_store import from the declare_table fallback arm to the
file-level imports, adding PutMode and PutOptions alongside the existing
ObjectStoreError and path::Path imports. Remove the function-body use
declaration while leaving the put_opts logic unchanged.
Source: Coding guidelines
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
rust/lance-namespace-impls/src/dir.rs (2)
11304-11313: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAssert the namespace error code as well as message text. These tests can pass if the implementation returns the wrong error variant with a similar message.
rust/lance-namespace-impls/src/dir.rs#L11304-L11313: assertErrorCode::ConcurrentModificationbefore checking the message.rust/lance-namespace-impls/src/dir.rs#L11412-L11420: assertErrorCode::ConcurrentModificationbefore checking the message.As per coding guidelines, “Assert on both the error variant and the message content in tests; do not check only
is_err().”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-namespace-impls/src/dir.rs` around lines 11304 - 11313, Update both error assertions in rust/lance-namespace-impls/src/dir.rs at lines 11304-11313 and 11412-11420 to verify the returned error has ErrorCode::ConcurrentModification before checking its message content. Replace the broad is_err/message-only checks while preserving the existing message validation.Source: Coding guidelines
1650-1654: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winDo not accept a caller-supplied ETag as manifest identity.
The REST request forwards
e_tagdirectly, but this shortcut never verifies it against the staging object. A request can supply the published manifest’s ETag with different staging bytes, receive a false idempotent success, and have its staging manifest deleted. Always compare content (or a verified digest of both objects); retain size only as a rejection fast path.Proposed fix
- if let (Some(req_tag), Some(exist_tag)) = (request_e_tag, final_meta.e_tag.as_deref()) - && req_tag == exist_tag - { - return Ok(true); - }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-namespace-impls/src/dir.rs` around lines 1650 - 1654, Remove the request-Etag equality shortcut in the manifest identity check near final_meta.e_tag. Determine idempotent identity by comparing the staging and published manifest contents, or verified digests of both objects; retain size comparison only as an early rejection fast path, and do not delete staging or return success unless content matches.
🧹 Nitpick comments (1)
rust/lance-namespace-impls/src/dir.rs (1)
11069-11071: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMove new test imports to the module import block.
rust/lance-namespace-impls/src/dir.rs#L11069-L11071: move these imports to the test module imports.rust/lance-namespace-impls/src/dir.rs#L11184-L11186: reuse the module-level imports.rust/lance-namespace-impls/src/dir.rs#L11333-L11335: reuse the module-level imports.As per coding guidelines, “Place
useimports at the top of the file, not inline within function bodies.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-namespace-impls/src/dir.rs` around lines 11069 - 11071, Move the inline imports for TryStreamExt, DatasetBuilder, and CreateTableVersionRequest into the test module’s top-level import block in rust/lance-namespace-impls/src/dir.rs at lines 11069-11071, then remove the duplicate local imports at lines 11184-11186 and 11333-11335 so those tests reuse the module-level imports.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 1748-1764: Update enforce_create_table_version_cas so an empty
chain accepts a non-1 initial version only for the branch-bootstrap path, while
the main path requires version 1. Replace saturating_add(1) with checked_add(1),
returning a contextual error when the latest version overflows.
---
Outside diff comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 11304-11313: Update both error assertions in
rust/lance-namespace-impls/src/dir.rs at lines 11304-11313 and 11412-11420 to
verify the returned error has ErrorCode::ConcurrentModification before checking
its message content. Replace the broad is_err/message-only checks while
preserving the existing message validation.
- Around line 1650-1654: Remove the request-Etag equality shortcut in the
manifest identity check near final_meta.e_tag. Determine idempotent identity by
comparing the staging and published manifest contents, or verified digests of
both objects; retain size comparison only as an early rejection fast path, and
do not delete staging or return success unless content matches.
---
Nitpick comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 11069-11071: Move the inline imports for TryStreamExt,
DatasetBuilder, and CreateTableVersionRequest into the test module’s top-level
import block in rust/lance-namespace-impls/src/dir.rs at lines 11069-11071, then
remove the duplicate local imports at lines 11184-11186 and 11333-11335 so those
tests reuse the module-level imports.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: dbad1e3b-0fc0-4049-bc07-62a7f8143400
📒 Files selected for processing (1)
rust/lance-namespace-impls/src/dir.rs
28fee9e to
6a2fd05
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
rust/lance-namespace-impls/src/dir.rs (1)
1650-1654: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winDo not treat a request ETag as proof of staging-content equality.
A caller can reuse the published final manifest’s ETag in
request.e_tagwhile supplying different staging bytes. This returns idempotent success without reading staging content, violating the required differing-content conflict behavior. Remove this success shortcut and always compare bytes (or a server-verified content digest); add a regression test with a matching request ETag and different staging bytes.Proposed fix
- if let (Some(req_tag), Some(exist_tag)) = (request_e_tag, final_meta.e_tag.as_deref()) - && req_tag == exist_tag - { - return Ok(true); - } + // Request ETags are not a verified digest of `staging_path`; use them + // only as metadata hints, never as proof of content equality.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-namespace-impls/src/dir.rs` around lines 1650 - 1654, Remove the request_e_tag/final_meta.e_tag equality early return in the surrounding manifest validation flow so a matching request ETag cannot establish staging-content equality. Always compare the staging bytes or a server-verified content digest and return the differing-content conflict when they do not match; add a regression test covering a matching request ETag with different staging bytes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@rust/lance-namespace-impls/src/dir.rs`:
- Around line 1650-1654: Remove the request_e_tag/final_meta.e_tag equality
early return in the surrounding manifest validation flow so a matching request
ETag cannot establish staging-content equality. Always compare the staging bytes
or a server-verified content digest and return the differing-content conflict
when they do not match; add a regression test covering a matching request ETag
with different staging bytes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: 3f5dec13-b059-4c6a-be2d-8e105504b8ba
📒 Files selected for processing (1)
rust/lance-namespace-impls/src/dir.rs
6a2fd05 to
0e421e5
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…dotfile-safe reserved marker - Enforce strict version CAS (latest+1) in create_table_version and treat an already-published identical-content version as idempotent success instead of ConcurrentModification. - Resolve create conflicts by comparing staging vs final manifest content (e_tag/size/byte equality) so retries succeed after a conditional rename wins the publish race. - Stage .lance-reserved via a non-dot sibling and rename_if_not_exists to avoid object stores whose Create uses temp+rename producing '..' temp names rejected by the backing store; fall back to PutMode::Create when rename is unsupported. - Add tests for idempotent retry, content conflict, and CAS gap rejection.
0e421e5 to
f184470
Compare
There was a problem hiding this comment.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
🟡 Other comments (1)
rust/lance-namespace-impls/src/dir/manifest.rs-3568-3575 (1)
3568-3575: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMap atomic marker publish failures to
TableAlreadyExists.
declare_tablekeeps the manifest-lookup guard but then writes the.lance-reservedmarker directly. If two concurrent calls both miss the manifest check, the loser loses onrename_if_not_existsor thePutMode::Createfallback. MapAlreadyExistsandPreconditionfailures from this publish toNamespaceError::TableAlreadyExists { table_name }; all other publish failures may remainInternal.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-namespace-impls/src/dir/manifest.rs` around lines 3568 - 3575, Update the error mapping around the atomic marker publish in declare_table so AlreadyExists and Precondition failures from rename_if_not_exists or the PutMode::Create fallback become NamespaceError::TableAlreadyExists { table_name }. Preserve the existing Internal mapping for all other publish failures and keep the manifest-lookup guard unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Other comments:
In `@rust/lance-namespace-impls/src/dir/manifest.rs`:
- Around line 3568-3575: Update the error mapping around the atomic marker
publish in declare_table so AlreadyExists and Precondition failures from
rename_if_not_exists or the PutMode::Create fallback become
NamespaceError::TableAlreadyExists { table_name }. Preserve the existing
Internal mapping for all other publish failures and keep the manifest-lookup
guard unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: e90838e0-347c-46c8-8d8e-726db082f285
📒 Files selected for processing (2)
rust/lance-namespace-impls/src/dir.rsrust/lance-namespace-impls/src/dir/manifest.rs
A branch's first commit carries its fork version, which can be greater than 1 (e.g. forking main at v2). enforce_create_table_version_cas previously hardcoded the expected version to 1 for an empty chain, so creating a branch forked from a non-v1 version failed with DatasetAlreadyExists. Relax the CAS so an empty branch chain accepts the requested fork version as its bootstrap, while an empty main chain still must start at v1.
Replace saturating_add(1) with checked_add(1) when computing the expected next version in enforce_create_table_version_cas, so a u64 version overflow returns a descriptive ConcurrentModification error instead of silently wrapping around.
| // Empty chain: main starts at v1, but a branch bootstraps at its fork | ||
| // version which can be > 1. | ||
| None => { | ||
| if is_branch { | ||
| version | ||
| } else { | ||
| 1 | ||
| } | ||
| } |
There was a problem hiding this comment.
enforce_create_table_version_cas only retrieves is_branch: bool, failing to obtain
BranchContents.parent_version. The documentation comment states that "branch bootstraps at its fork version", yet the actual implementation permits any arbitrary version (such as 1) to be set as the first
manifest of a branch forked from v5, breaking the semantics of "branch inherits main up to fork version".
resolve_branch_for_commit has already fetched BranchContents (the Ok(_) branch discards parent_version), and passing it downstream to perform the version == parent_version + 1 validation incurs negligible overhead.
| publish.map_err(|e| { | ||
| lance_core::Error::from(NamespaceError::Internal { | ||
| message: format!( | ||
| "Failed to create .lance-reserved file for table {}: {}", | ||
| table_name, e | ||
| ), | ||
| }) | ||
| })?; |
There was a problem hiding this comment.
When .lance-reserved already exists, the AlreadyExists / Precondition returned by rename_if_not_exists or the fallback PutMode::Create is indiscriminately wrapped as NamespaceError::Internal. In contrast, DirectoryNamespace::declare_table within the same crate explicitly maps such errors to NamespaceError::TableAlreadyExists via put_marker_file_atomic. Trigger scenarios: concurrent declare_table calls, retries after a previous incomplete declare operation. Callers cannot programmatically distinguish between "IO failures" and "table already declared". A directly applicable patch is attached.
| // Create the .lance-reserved file to mark the table as existing. | ||
| // Avoid `object_store.create(.lance-reserved)` / PutMode::Create on the | ||
| // final dotfile: some object stores implement Create via temp+rename | ||
| // using the final basename, which yields `..lance-reserved` temps that | ||
| // the underlying store rejects. Stage under a non-dot sibling, then | ||
| // rename_if_not_exists. | ||
| let reserved_file_path = table_path.clone().join(".lance-reserved"); | ||
| let staging_name = format!("lance-marker.staging.{}", Uuid::new_v4().simple()); | ||
| let staging_path = table_path.clone().join(staging_name.as_str()); | ||
|
|
||
| // Some object stores write via temp+rename and may not flush empty | ||
| // objects through to the underlying filesystem, so the conditional | ||
| // rename can fail with NotFound. Use a tiny non-empty payload. |
There was a problem hiding this comment.
Is this duplicated with dir.rs::put_marker_file_atomic?
Summary
Make
create_table_versionin the directory namespace (lance-namespace-impls)idempotent, strictly enforce version CAS, and make the
.lance-reservedmarkercreation safe across object stores.
Problem
In the namespace directory provider,
create_table_versionis currently unsafeagainst the retries and races that happen with real object-store deployments:
Non-idempotent retries. If a version is successfully created but the
client retries (network blip, or the coordinating node renamed staging →
final while the caller lost the response), the second attempt sees the
version already published and fails with
ConcurrentModification. Callerscannot distinguish a genuine conflict from a benign retry.
Loose version CAS. There is no enforced "must be
latest + 1" check,so a stale or duplicate write can fork the version chain.
Reserved-marker staging breaks on some object stores. The table is
reserved by writing a
.lance-reservedmarker. Some backends implementPutMode::Createas a temp-file + rename, producing temporary namescontaining
... Object stores and unified file systems that reject..path components then fail the marker write and the table can never be
created.
Changes
create_table_versiononly acceptsversion == latest + 1(
version == 1on an empty chain); any other value maps toNamespaceError::ConcurrentModification.exists with identical manifest content (
e_tag/size/ bytes), returnsuccess instead of
ConcurrentModification.content differs, fail with
ConcurrentModification.rename_if_not_exists(conditional no-replace) onto.lance-reserved; fallback to
PutMode::Createwhen rename is unsupported.Test plan
plus table-not-found and on-branch variants.
cargo test -p lance-namespace-impls --lib create_table_versionpasses.Scope
Targets the
dirnamespace provider (create_table_version/ reserved-markerlifecycle). The retry/conflict semantics apply to any object-store backend,
not a specific one.
Closes #7939