Skip to content

[feature](s3) Support S3 Express One Zone#65504

Open
0AyanamiRei wants to merge 11 commits into
apache:masterfrom
0AyanamiRei:take-over-63409
Open

[feature](s3) Support S3 Express One Zone#65504
0AyanamiRei wants to merge 11 commits into
apache:masterfrom
0AyanamiRei:take-over-63409

Conversation

@0AyanamiRei

@0AyanamiRei 0AyanamiRei commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #63409

Problem Summary:

Doris can use the existing S3 import pipeline to read object data from Amazon S3 Express One Zone, but it must first recognize a directory bucket and create an AWS SDK client with S3 Express endpoint rules and session authentication enabled. The previous implementation also inferred S3 Express from user-provided endpoints and changed shared import/storage abstractions. This PR narrows the feature to S3 Express semantics required by the import layer.

This PR supports S3 Express reads only for:

  • Non-internal one-shot INSERT INTO ... SELECT ... FROM S3(...).
  • Broker Load using WITH S3.

After file discovery, both paths continue through the existing S3 import flow for exact paths, multiple paths, globs, format parsing, schema inference, column mapping, Range GET, and load execution.

Directory bucket, endpoint, and Region contract

An S3 Express directory bucket must be supplied as the complete bucket name embedded in the URI:

<bucket-base-name>--<zone-id>--x-s3

For example:

bucket-name--usw2-az1--x-s3

S3 Express mode is selected only when all of the following are true:

  • The statement is one of the two supported import entry points.
  • s3.provider is explicitly set to AWS.
  • The URI contains a valid complete directory bucket name.

The complete directory bucket name is the source of truth. To avoid bucket Zone/Region and bucket Zone/endpoint Zone mismatches, users are recommended to provide the complete bucket name and omit both s3.endpoint and s3.region.

Doris enables AWS SDK automatic endpoint and S3 Express session-authentication behavior:

  • A user-provided s3.endpoint is not used as endpointOverride; the SDK derives the zonal data endpoint from the complete directory bucket name.
  • Doris does not validate bucket Zone against configured Region, bucket Zone against endpoint Zone, or endpoint against configured Region. If endpoint or Region is supplied, no mismatch check is performed.
  • For currently known Zone prefixes, Doris derives the full Region from the bucket name, for example usw2-az1 to us-west-2. This derived Region takes precedence over a user-provided Region.
  • If AWS introduces a Region whose Zone prefix cannot yet be expanded by Doris, Doris uses the user-provided s3.region unchanged as the SDK Region, without consistency validation. If neither inference nor a fallback Region is available, the request fails with a clear error.
  • S3 Express always uses HTTPS and virtual-hosted-style addressing. A supplied use_path_style value is still validated as a boolean, but the Express client does not propagate a path-style preference and always uses virtual-hosted style.
  • The configured AWS credentials are base credentials. The AWS SDK automatically calls CreateSession, caches the S3 Express session credentials, and refreshes them when needed.

Recommended usage:

FROM S3(
    "uri" = "s3://bucket-name--usw2-az1--x-s3/import/**/part-*.parquet",
    "s3.provider" = "AWS",
    "s3.access_key" = "<access-key>",
    "s3.secret_key" = "<secret-key>",
    "use_path_style" = "false", -- use virtual-hosted style
    "format" = "parquet"
);

See the AWS documentation for S3 Express SDK session authentication and directory bucket endpoints and Zone IDs.

Implementation scope

  • FE attaches an internal marker only to the two supported import paths. Raw user property maps cannot enable the marker.
  • FE uses an SDK-managed Express client only for directory-bucket ListObjectsV2 discovery. It follows directory-bucket listing constraints by using slash-terminated prefixes and continuation-token pagination without StartAfter.
  • BE recognizes the same complete bucket-name format, resolves the effective Region, ignores endpoint override, forces HTTPS/virtual-hosted style, and lets the AWS C++ SDK select the S3 Express signer and session credential provider. Object reads then use the existing S3 read implementation.
  • Ordinary S3 buckets and S3-compatible services retain their existing endpoint, Region, addressing, and client behavior.

This PR does not add S3 Express support to standalone S3/FILE TVF queries, CTAS, INSERT OVERWRITE, COPY, Streaming Insert, internal loads, external catalogs, resources, storage vaults, Doris internal storage, export, upload, multipart upload, delete, rename, checksum handling, or other write paths.

Release note

Support importing data from existing Amazon S3 Express One Zone directory buckets through one-shot INSERT INTO ... SELECT ... FROM S3(...) and Broker Load WITH S3.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
      • FE core targeted unit tests: 58 passed.
      • FE filesystem targeted unit tests: 138 passed.
      • BE targeted S3ClientFactoryTest: 12 passed.
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
      • Only the two scoped S3 import entry points select SDK-managed S3 Express endpoint and session authentication.
      • User endpoint is ignored; known bucket Zone prefixes determine Region, and user Region is only an unchecked fallback for unknown prefixes.
      • Non-target entry points, ordinary S3 behavior, and S3 write behavior remain unchanged.
  • Does this need documentation?

    • No.
    • Yes. A follow-up documentation PR will describe the supported import entry points and configuration contract.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

yinzixin and others added 3 commits July 13, 2026 09:22
### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: apache#63409\n\nProblem Summary: The initial S3 Express implementation detected directory buckets and endpoints with broad substring matches in two separate locations. This could classify ordinary bucket names or endpoint paths as S3 Express and allowed the client configuration and upload checksum behavior to diverge. Centralize the decision in one helper, require the documented --x-s3 bucket suffix or an s3express endpoint host label, and pass the resulting context into the object storage client. Add unit coverage for valid and invalid names.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Unit Test (S3UTILTest.is_s3_express_context; ASAN UT rebuild started)\n- Behavior changed: No, except avoiding false-positive S3 Express detection\n- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei
0AyanamiRei marked this pull request as ready for review July 13, 2026 01:38
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#63409

Problem Summary: Document a production-ready design for S3 Express One Zone on Doris. The design audits the current PR, defines service-aware bucket capabilities, preserves ordinary S3 endpoint behavior, completes checksum and multipart flows across BE and FE, specifies unordered listing compatibility, rejects unsupported Cloud and Hadoop surfaces before I/O or persistence, and provides test, rollout, and data rollback gates.

### Release note

None

### Check List (For Author)

- Test: No need to test (documentation-only change; validated with git diff --cached --check and source-path consistency checks)

- Behavior changed: No

- Does this need documentation: No (this commit is the implementation design)
### What problem does this PR solve?

Issue Number: None

Related PR: apache#63409

Problem Summary: Add end-to-end S3 Express One Zone Directory Bucket support across native BE and FE S3 paths. Use SDK-managed session authentication and zonal endpoint routing, CRC32C multipart propagation, Directory Bucket listing semantics, active multipart cleanup, and fail-fast gates for unsupported Cloud Storage Vault and Hadoop/S3A surfaces while preserving ordinary S3-compatible behavior.

### Release note

Add native S3 Express One Zone Directory Bucket support for Doris S3 file access. Cloud Storage Vault, Hadoop/S3A-backed external surfaces, and presigned URLs remain unsupported and fail fast.

### Check List (For Author)

- Test: Test code added but not run per request
- Behavior changed: Yes. Official AWS Directory Buckets use SDK-managed Express session authentication, CRC32C, and Directory Bucket listing semantics; unsupported surfaces fail fast.
- Does this need documentation: Yes, goal.md
### What problem does this PR solve?

Issue Number: None

Related PR: apache#63409

Problem Summary: Document how the S3 Express One Zone design maps to the implemented BE, FE, Hive, Cloud boundary, checksum, multipart, listing, and fail-fast changes. Record compatibility behavior, unexecuted tests, and remaining validation gaps so the implementation status is not confused with GA acceptance.

### Release note

None

### Check List (For Author)

- Test: No need to test (documentation-only change)
- Behavior changed: No
- Does this need documentation: Yes, this commit adds the implementation summary
### What problem does this PR solve?

Issue Number: None

Related PR: apache#63409

Problem Summary: The previous S3 Express implementation expanded Directory Bucket awareness across listing, deletion, presigning, connectors, Cloud storage, checksums, and documentation. This change narrows support to official AWS endpoints and valid --x-s3 bucket names on Doris native known-object paths. AWS SDK endpoint rules provide the zonal endpoint and CreateSession authentication, Express clients use HTTPS and virtual-hosted addressing, Head/Get/range/Put and multipart create/upload/complete/abort are supported, exact S3 TVF and S3 Load paths use HeadObject, and Express uploads no longer send Content-MD5. General-purpose S3 and S3-compatible endpoints retain their existing client behavior.

### Release note

Support native known-object reads and writes for Amazon S3 Express One Zone Directory Buckets.

### Check List (For Author)

- Test: Not run (per user request)
- Behavior changed: Yes. Official AWS Directory Buckets use SDK S3 Express routing and session authentication for native known-object operations.
- Does this need documentation: Yes. The user documentation will be handled separately.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#63409

Problem Summary: Align S3 Express support with the documented known-object contract. Only valid AWS Directory Buckets using a standard HTTPS regional endpoint, a matching region, and virtual-hosted-style access enable SDK endpoint rules and CreateSession. Exact S3 TVF and Broker Load paths use HeadObject and reject glob, range, query, empty-key, and directory-prefix inputs. Third-party S3-compatible endpoints retain conventional S3 authentication and checksum behavior even when a bucket name ends in --x-s3. BE known-object uploads omit Content-MD5 for Directory Buckets. Remove the prior generic Java operation and BE abort expansions. The existing SDK versions already provide the required endpoint and session APIs, so no SDK upgrade is included.

### Release note

Support known-object access to Amazon S3 Express One Zone through native S3 paths.

### Check List (For Author)

- Test: Not run (per user request)
- Behavior changed: Yes. Valid AWS Directory Buckets use S3 Express session authentication for known-object access; unsupported listing and glob semantics are not enabled.
- Does this need documentation: Yes. The user documentation is handled separately.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65504

Problem Summary: Synchronize the S3 Express branch with the latest master so the import-layer refactor and its unit tests use the current S3 filesystem interfaces.

### Release note

None

### Check List (For Author)

- Test: No need to test (base branch synchronization only)
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65504

Problem Summary: Amazon S3 Express One Zone directory buckets require directory-bucket listing rules and SDK-managed session authentication, while the existing S3 import paths used regular S3 clients. Scope the capability to one-shot INSERT SELECT from the S3 table-valued source and Broker Load WITH S3, validate native AWS directory-bucket context, select Express clients for discovery and reads, and preserve ordinary, third-party, legacy, and non-target import behavior.

### Release note

Support importing from Amazon S3 Express One Zone directory buckets through one-shot INSERT SELECT S3 sources and Broker Load WITH S3.

### Check List (For Author)

- Test: Unit Test
    - FE core targeted unit tests
    - FE filesystem targeted unit tests
    - BE targeted unit tests
- Behavior changed: Yes, S3 Express reads are enabled only for the two scoped S3 import entry points.
- Does this need documentation: Yes (follow-up documentation PR pending)
### What problem does this PR solve?

Issue Number: None

Related PR: apache#63409

Problem Summary: The previous S3 Express implementation inferred directory-bucket behavior from user endpoints and changed generic S3/import abstractions beyond the intended feature. This refactor limits S3 Express selection to trusted one-shot INSERT SELECT and Broker Load read paths with an explicit AWS provider and a complete directory bucket name. Doris ignores the user endpoint for these scoped reads, derives known Regions from the bucket Zone ID, falls back to an unchecked user Region for future Zone prefixes, and lets the AWS SDK resolve the zonal endpoint and manage S3 Express session credentials. Ordinary S3 and non-target import, catalog, resource, storage, and write paths retain their existing behavior.

### Release note

Add S3 Express One Zone read support for one-shot INSERT SELECT from S3 and Broker Load with S3. Provide the complete directory bucket name and AWS provider; endpoint is ignored, while region is used only as an unchecked fallback for an unknown Zone prefix.

### Check List (For Author)

- Test: Unit Test
    - BE S3ClientFactoryTest: 12 tests passed
    - FE targeted S3 Express and import tests: 196 tests passed
- Behavior changed: Yes. Only the scoped S3 import reads select SDK-managed S3 Express endpoint and session authentication.
- Does this need documentation: Yes. The PR description documents the supported entry points and configuration contract.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants