Skip to content

chore(deps): bump opendal to 0.58.1 - #20252

Open
Xuanwo wants to merge 5 commits into
mainfrom
xuanwo/bump-opendal-0.58.1
Open

chore(deps): bump opendal to 0.58.1#20252
Xuanwo wants to merge 5 commits into
mainfrom
xuanwo/bump-opendal-0.58.1

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Bump Apache OpenDAL from 0.54.1 to 0.58.1 (released 2026-07-31), and object_store_opendal to 0.58.0 (0.58.1 is yanked on crates.io).

OpenDAL 0.55–0.58 rework the service/layer composition model (AccessService), switch schemes to strings, move metadata timestamps to jiff/opendal::raw::Timestamp, replace HttpClient/HttpFetch with HttpTransport, and return a finished Operator from Operator::new (no .finish()).

Fallout fixes

  • Rewrite Databend custom layers (RuntimeLayer, ConcurrentLimitLayer, StorageMetricsLayer, metrics observe layer) for the new Layer / Service APIs
  • allow_anonymousskip_signature; full_capabilitycapability
  • Scheme enum matching → string schemes / services::*_SCHEME
  • Timestamp bridge via opendal_timestamp_to_chrono
  • Align fastrace / fastrace-opentelemetry / opentelemetry so layers-fastrace builds with OpenDAL 0.58

Residual risk (blocked on ecosystem)

object_store_opendal 0.58 requires object_store 0.13, while current deltalake (git rev) and datafuse-extras/lance still pin object_store 0.12. That leaves:

  • databend-common-storages-delta
  • databend-common-storages-stage (Lance path via OpendalStore)

failing with OpendalStore: ObjectStore trait version mismatch until those crates are upgraded together with workspace object_store to 0.13.x.

Tests

  • Unit Test — targeted cargo check (see below)
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Validated with:

cargo check -p databend-common-storage
cargo check -p databend-common-storages-fuse
cargo check -p databend-common-sql
cargo check -p databend-common-storages-parquet

Lockfile: opendal 0.58.1 (direct), object_store_opendal 0.58.0, transitive opendal 0.55.0 still present via iceberg-rust.

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe): dependency upgrade + API fallout fixes

AI assistance

  • AI usage: Agent-assisted dependency bump, OpenDAL 0.54→0.58 API migration (custom layers, scheme/timestamp/http client rewrites), targeted compile validation, draft PR.
  • Responsible human: @Xuanwo
  • The responsible human has read every line of this diff and can explain each change

This change is Reviewable

Upgrade workspace opendal from 0.54.1 to 0.58.1 and object_store_opendal
to 0.58.0 (0.58.1 is yanked). Migrate custom layers and call sites to the
Service-based composition API, string schemes, jiff timestamps, and
HttpTransport.

Also bump fastrace/fastrace-opentelemetry/opentelemetry for compatibility
with opendal-layer-fastrace 0.58.
@github-actions github-actions Bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Aug 4, 2026
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

🤖 CI Job Analysis

Workflow: 30926357907

📊 Summary

  • Total Jobs: 90
  • Failed Jobs: 7
  • Retryable: 0
  • Code Issues: 7

NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

  • linux / test_compat_meta_query: Not retryable (Code/Test)
  • linux / test_stateful_cluster: Not retryable (Code/Test)
  • linux / test_stateful_standalone: Not retryable (Code/Test)
  • linux / sqllogic / cluster (base, 2c, 2, http): Not retryable (Code/Test)
  • linux / sqllogic / standalone (base, 2c, http): Not retryable (Code/Test)
  • linux / sqllogic / cluster (base, 2c, 2, hybrid): Not retryable (Code/Test)
  • linux / sqllogic / standalone (base, 2c, hybrid): Not retryable (Code/Test)

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

@Xuanwo
Xuanwo marked this pull request as ready for review August 4, 2026 11:12
@Xuanwo

Xuanwo commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

object_store_opendal 0.58 requires object_store 0.13, while current deltalake (git rev) and datafuse-extras/lance still pin object_store 0.12. That leaves:

Any ideas? I'm not sure which version to use in this case.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d267e50a2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +336 to 339
async fn acquire(&mut self) {
if self.permit.is_none() {
self.permit = Some(acquire_operation_permit(&self.semaphore, &self.labels).await);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Release permits after each raw I/O call

When max_concurrent_io_requests is enabled, acquire() stores the permit in self.permit but never clears it, so after a writer, lister, deleter, copier, or read stream performs its first operation, it occupies a slot until the entire handle is dropped. With a limit of 1, for example, an idle first writer blocks a second writer indefinitely rather than releasing capacity after write() completes; the assertion added at line 421 that the permit count returns to 1 after writer.write() will itself fail. Preserve the previous per-operation behavior by dropping/resetting the permit after each awaited raw I/O call.

AGENTS.md reference: AGENTS.md:L10-L11

Useful? React with 👍 / 👎.

Xuanwo added 4 commits August 4, 2026 20:51
…0.58

Drop object_store_opendal 0.58 (needs object_store 0.13) and add a local
ObjectStore adapter over OpenDAL 0.58 for object_store 0.12 so delta and
stage keep working with current deltalake/lance. Apply cargo fmt.
- Local ObjectStore adapter for object_store 0.12 (drop object_store_opendal 0.58)
- Replace remove_all with delete_with().recursive(true)
- Allow unused_braces for fastrace 0.7.18+ macro expansion
- Migrate EE vacuum Access/OperatorBuilder mocks to Service/from_parts
- Fix Scheme/lister lifetime/timestamp test API breakage
- Release ConcurrentLimitLayer permits per write/list/delete/copy call so
  long-lived IO handles do not pin the global semaphore (restore Databend
  semantics; matches test_io_handles_do_not_hold_operation_permits).
- Bump workspace jiff floor to 0.2.28 (opendal-core requirement) and refresh
  cast/datetime goldenfiles for jiff 0.2.35 error message wording.
- Do not call skip_signature unconditionally for S3; OpenDAL 0.58's
  skip_signature always bypasses signing (unlike old allow_anonymous),
  which caused MinIO/stage PermissionDenied 403s.
- Format timestamps with jiff::fmt::strtime::format so invalid directives
  (e.g. MySQL %i) still error after jiff made Display/strftime lenient.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-chore this PR only has small changes that no need to record, like coding styles.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant