[feature](s3) Support S3 Express One Zone#65504
Open
0AyanamiRei wants to merge 11 commits into
Open
Conversation
…nt-MD5 for S3 Express
### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: apache#63409\n\nProblem Summary: The initial S3 Express implementation detected directory buckets and endpoints with broad substring matches in two separate locations. This could classify ordinary bucket names or endpoint paths as S3 Express and allowed the client configuration and upload checksum behavior to diverge. Centralize the decision in one helper, require the documented --x-s3 bucket suffix or an s3express endpoint host label, and pass the resulting context into the object storage client. Add unit coverage for valid and invalid names.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Unit Test (S3UTILTest.is_s3_express_context; ASAN UT rebuild started)\n- Behavior changed: No, except avoiding false-positive S3 Express detection\n- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
0AyanamiRei
marked this pull request as ready for review
July 13, 2026 01:38
0AyanamiRei
requested review from
Gabriel39,
gavinchou,
liaoxin01 and
morningman
as code owners
July 13, 2026 01:38
### What problem does this PR solve? Issue Number: N/A Related PR: apache#63409 Problem Summary: Document a production-ready design for S3 Express One Zone on Doris. The design audits the current PR, defines service-aware bucket capabilities, preserves ordinary S3 endpoint behavior, completes checksum and multipart flows across BE and FE, specifies unordered listing compatibility, rejects unsupported Cloud and Hadoop surfaces before I/O or persistence, and provides test, rollout, and data rollback gates. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change; validated with git diff --cached --check and source-path consistency checks) - Behavior changed: No - Does this need documentation: No (this commit is the implementation design)
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: Add end-to-end S3 Express One Zone Directory Bucket support across native BE and FE S3 paths. Use SDK-managed session authentication and zonal endpoint routing, CRC32C multipart propagation, Directory Bucket listing semantics, active multipart cleanup, and fail-fast gates for unsupported Cloud Storage Vault and Hadoop/S3A surfaces while preserving ordinary S3-compatible behavior. ### Release note Add native S3 Express One Zone Directory Bucket support for Doris S3 file access. Cloud Storage Vault, Hadoop/S3A-backed external surfaces, and presigned URLs remain unsupported and fail fast. ### Check List (For Author) - Test: Test code added but not run per request - Behavior changed: Yes. Official AWS Directory Buckets use SDK-managed Express session authentication, CRC32C, and Directory Bucket listing semantics; unsupported surfaces fail fast. - Does this need documentation: Yes, goal.md
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: Document how the S3 Express One Zone design maps to the implemented BE, FE, Hive, Cloud boundary, checksum, multipart, listing, and fail-fast changes. Record compatibility behavior, unexecuted tests, and remaining validation gaps so the implementation status is not confused with GA acceptance. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change) - Behavior changed: No - Does this need documentation: Yes, this commit adds the implementation summary
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: The previous S3 Express implementation expanded Directory Bucket awareness across listing, deletion, presigning, connectors, Cloud storage, checksums, and documentation. This change narrows support to official AWS endpoints and valid --x-s3 bucket names on Doris native known-object paths. AWS SDK endpoint rules provide the zonal endpoint and CreateSession authentication, Express clients use HTTPS and virtual-hosted addressing, Head/Get/range/Put and multipart create/upload/complete/abort are supported, exact S3 TVF and S3 Load paths use HeadObject, and Express uploads no longer send Content-MD5. General-purpose S3 and S3-compatible endpoints retain their existing client behavior. ### Release note Support native known-object reads and writes for Amazon S3 Express One Zone Directory Buckets. ### Check List (For Author) - Test: Not run (per user request) - Behavior changed: Yes. Official AWS Directory Buckets use SDK S3 Express routing and session authentication for native known-object operations. - Does this need documentation: Yes. The user documentation will be handled separately.
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: Align S3 Express support with the documented known-object contract. Only valid AWS Directory Buckets using a standard HTTPS regional endpoint, a matching region, and virtual-hosted-style access enable SDK endpoint rules and CreateSession. Exact S3 TVF and Broker Load paths use HeadObject and reject glob, range, query, empty-key, and directory-prefix inputs. Third-party S3-compatible endpoints retain conventional S3 authentication and checksum behavior even when a bucket name ends in --x-s3. BE known-object uploads omit Content-MD5 for Directory Buckets. Remove the prior generic Java operation and BE abort expansions. The existing SDK versions already provide the required endpoint and session APIs, so no SDK upgrade is included. ### Release note Support known-object access to Amazon S3 Express One Zone through native S3 paths. ### Check List (For Author) - Test: Not run (per user request) - Behavior changed: Yes. Valid AWS Directory Buckets use S3 Express session authentication for known-object access; unsupported listing and glob semantics are not enabled. - Does this need documentation: Yes. The user documentation is handled separately.
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: Synchronize the S3 Express branch with the latest master so the import-layer refactor and its unit tests use the current S3 filesystem interfaces. ### Release note None ### Check List (For Author) - Test: No need to test (base branch synchronization only) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#65504 Problem Summary: Amazon S3 Express One Zone directory buckets require directory-bucket listing rules and SDK-managed session authentication, while the existing S3 import paths used regular S3 clients. Scope the capability to one-shot INSERT SELECT from the S3 table-valued source and Broker Load WITH S3, validate native AWS directory-bucket context, select Express clients for discovery and reads, and preserve ordinary, third-party, legacy, and non-target import behavior. ### Release note Support importing from Amazon S3 Express One Zone directory buckets through one-shot INSERT SELECT S3 sources and Broker Load WITH S3. ### Check List (For Author) - Test: Unit Test - FE core targeted unit tests - FE filesystem targeted unit tests - BE targeted unit tests - Behavior changed: Yes, S3 Express reads are enabled only for the two scoped S3 import entry points. - Does this need documentation: Yes (follow-up documentation PR pending)
0AyanamiRei
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
July 18, 2026 12:51
### What problem does this PR solve? Issue Number: None Related PR: apache#63409 Problem Summary: The previous S3 Express implementation inferred directory-bucket behavior from user endpoints and changed generic S3/import abstractions beyond the intended feature. This refactor limits S3 Express selection to trusted one-shot INSERT SELECT and Broker Load read paths with an explicit AWS provider and a complete directory bucket name. Doris ignores the user endpoint for these scoped reads, derives known Regions from the bucket Zone ID, falls back to an unchecked user Region for future Zone prefixes, and lets the AWS SDK resolve the zonal endpoint and manage S3 Express session credentials. Ordinary S3 and non-target import, catalog, resource, storage, and write paths retain their existing behavior. ### Release note Add S3 Express One Zone read support for one-shot INSERT SELECT from S3 and Broker Load with S3. Provide the complete directory bucket name and AWS provider; endpoint is ignored, while region is used only as an unchecked fallback for an unknown Zone prefix. ### Check List (For Author) - Test: Unit Test - BE S3ClientFactoryTest: 12 tests passed - FE targeted S3 Express and import tests: 196 tests passed - Behavior changed: Yes. Only the scoped S3 import reads select SDK-managed S3 Express endpoint and session authentication. - Does this need documentation: Yes. The PR description documents the supported entry points and configuration contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #63409
Problem Summary:
Doris can use the existing S3 import pipeline to read object data from Amazon S3 Express One Zone, but it must first recognize a directory bucket and create an AWS SDK client with S3 Express endpoint rules and session authentication enabled. The previous implementation also inferred S3 Express from user-provided endpoints and changed shared import/storage abstractions. This PR narrows the feature to S3 Express semantics required by the import layer.
This PR supports S3 Express reads only for:
INSERT INTO ... SELECT ... FROM S3(...).WITH S3.After file discovery, both paths continue through the existing S3 import flow for exact paths, multiple paths, globs, format parsing, schema inference, column mapping, Range GET, and load execution.
Directory bucket, endpoint, and Region contract
An S3 Express directory bucket must be supplied as the complete bucket name embedded in the URI:
For example:
S3 Express mode is selected only when all of the following are true:
s3.provideris explicitly set toAWS.The complete directory bucket name is the source of truth. To avoid bucket Zone/Region and bucket Zone/endpoint Zone mismatches, users are recommended to provide the complete bucket name and omit both
s3.endpointands3.region.Doris enables AWS SDK automatic endpoint and S3 Express session-authentication behavior:
s3.endpointis not used asendpointOverride; the SDK derives the zonal data endpoint from the complete directory bucket name.usw2-az1tous-west-2. This derived Region takes precedence over a user-provided Region.s3.regionunchanged as the SDK Region, without consistency validation. If neither inference nor a fallback Region is available, the request fails with a clear error.use_path_stylevalue is still validated as a boolean, but the Express client does not propagate a path-style preference and always uses virtual-hosted style.CreateSession, caches the S3 Express session credentials, and refreshes them when needed.Recommended usage:
See the AWS documentation for S3 Express SDK session authentication and directory bucket endpoints and Zone IDs.
Implementation scope
ListObjectsV2discovery. It follows directory-bucket listing constraints by using slash-terminated prefixes and continuation-token pagination withoutStartAfter.This PR does not add S3 Express support to standalone S3/FILE TVF queries, CTAS, INSERT OVERWRITE, COPY, Streaming Insert, internal loads, external catalogs, resources, storage vaults, Doris internal storage, export, upload, multipart upload, delete, rename, checksum handling, or other write paths.
Release note
Support importing data from existing Amazon S3 Express One Zone directory buckets through one-shot
INSERT INTO ... SELECT ... FROM S3(...)and Broker LoadWITH S3.Check List (For Author)
Test
S3ClientFactoryTest: 12 passed.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)