chore(deps): bump opendal to 0.58.1 - #20252
Conversation
Upgrade workspace opendal from 0.54.1 to 0.58.1 and object_store_opendal to 0.58.0 (0.58.1 is yanked). Migrate custom layers and call sites to the Service-based composition API, string schemes, jiff timestamps, and HttpTransport. Also bump fastrace/fastrace-opentelemetry/opentelemetry for compatibility with opendal-layer-fastrace 0.58.
🤖 CI Job Analysis
📊 Summary
❌ NO RETRY NEEDEDAll failures appear to be code/test issues requiring manual fixes. 🔍 Job Details
🤖 AboutAutomated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed). |
Any ideas? I'm not sure which version to use in this case. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d267e50a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| async fn acquire(&mut self) { | ||
| if self.permit.is_none() { | ||
| self.permit = Some(acquire_operation_permit(&self.semaphore, &self.labels).await); | ||
| } |
There was a problem hiding this comment.
Release permits after each raw I/O call
When max_concurrent_io_requests is enabled, acquire() stores the permit in self.permit but never clears it, so after a writer, lister, deleter, copier, or read stream performs its first operation, it occupies a slot until the entire handle is dropped. With a limit of 1, for example, an idle first writer blocks a second writer indefinitely rather than releasing capacity after write() completes; the assertion added at line 421 that the permit count returns to 1 after writer.write() will itself fail. Preserve the previous per-operation behavior by dropping/resetting the permit after each awaited raw I/O call.
AGENTS.md reference: AGENTS.md:L10-L11
Useful? React with 👍 / 👎.
…0.58 Drop object_store_opendal 0.58 (needs object_store 0.13) and add a local ObjectStore adapter over OpenDAL 0.58 for object_store 0.12 so delta and stage keep working with current deltalake/lance. Apply cargo fmt.
- Local ObjectStore adapter for object_store 0.12 (drop object_store_opendal 0.58) - Replace remove_all with delete_with().recursive(true) - Allow unused_braces for fastrace 0.7.18+ macro expansion - Migrate EE vacuum Access/OperatorBuilder mocks to Service/from_parts - Fix Scheme/lister lifetime/timestamp test API breakage
- Release ConcurrentLimitLayer permits per write/list/delete/copy call so long-lived IO handles do not pin the global semaphore (restore Databend semantics; matches test_io_handles_do_not_hold_operation_permits). - Bump workspace jiff floor to 0.2.28 (opendal-core requirement) and refresh cast/datetime goldenfiles for jiff 0.2.35 error message wording.
- Do not call skip_signature unconditionally for S3; OpenDAL 0.58's skip_signature always bypasses signing (unlike old allow_anonymous), which caused MinIO/stage PermissionDenied 403s. - Format timestamps with jiff::fmt::strtime::format so invalid directives (e.g. MySQL %i) still error after jiff made Display/strftime lenient.
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
Bump Apache OpenDAL from 0.54.1 to 0.58.1 (released 2026-07-31), and
object_store_opendalto 0.58.0 (0.58.1is yanked on crates.io).OpenDAL 0.55–0.58 rework the service/layer composition model (
Access→Service), switch schemes to strings, move metadata timestamps to jiff/opendal::raw::Timestamp, replaceHttpClient/HttpFetchwithHttpTransport, and return a finishedOperatorfromOperator::new(no.finish()).Fallout fixes
RuntimeLayer,ConcurrentLimitLayer,StorageMetricsLayer, metrics observe layer) for the newLayer/ServiceAPIsallow_anonymous→skip_signature;full_capability→capabilityservices::*_SCHEMEopendal_timestamp_to_chronofastrace/fastrace-opentelemetry/opentelemetrysolayers-fastracebuilds with OpenDAL 0.58Residual risk (blocked on ecosystem)
object_store_opendal0.58 requires object_store 0.13, while currentdeltalake(git rev) anddatafuse-extras/lancestill pin object_store 0.12. That leaves:databend-common-storages-deltadatabend-common-storages-stage(Lance path viaOpendalStore)failing with
OpendalStore: ObjectStoretrait version mismatch until those crates are upgraded together with workspaceobject_storeto 0.13.x.Tests
cargo check(see below)Validated with:
Lockfile: opendal 0.58.1 (direct), object_store_opendal 0.58.0, transitive opendal 0.55.0 still present via iceberg-rust.
Type of change
AI assistance
This change is