Skip to content

[FLINK-39797] Support disabling automatic table creation#4420

Open
litiliu wants to merge 1 commit into
apache:masterfrom
litiliu:FLINK-39797-disable-auto-create-table
Open

[FLINK-39797] Support disabling automatic table creation#4420
litiliu wants to merge 1 commit into
apache:masterfrom
litiliu:FLINK-39797-disable-auto-create-table

Conversation

@litiliu
Copy link
Copy Markdown

@litiliu litiliu commented May 29, 2026

What is the purpose of the change

Introduce a framework-level sink option schema.create.mode to control target table creation behavior.

The new ERROR_IF_NOT_EXISTS mode allows users to skip automatic target table creation and fail fast when the target table does not exist. This is useful when target tables are managed by an external platform.

This PR currently enables ERROR_IF_NOT_EXISTS for Fluss and MaxCompute only. Other pipeline connectors keep their existing create-table behavior and will report unsupported when configured with schema.create.mode=ERROR_IF_NOT_EXISTS.

Brief change log

  • Add sink-level schema.create.mode with default CREATE_IF_NOT_EXISTS.
  • Add ERROR_IF_NOT_EXISTS handling through a framework-level MetadataApplier wrapper.
  • Add SupportsExistingSchemaValidation for sinks that can validate existing target schemas.
  • Support existing target schema validation in Fluss.
  • Support existing target schema validation in MaxCompute by extracting existing table/schema/primary-key checks from the original create table logic.
  • Keep Kafka, Elasticsearch, Doris, OceanBase, Iceberg, Paimon, Hudi, and StarRocks unsupported for ERROR_IF_NOT_EXISTS in this phase.

Verifying this change

  • Added parser/common wrapper tests for schema.create.mode.
  • Added Fluss tests for missing table fail-fast and existing schema validation.
  • Added MaxCompute tests for:
    • schemaExists() returns false when target table is missing
    • validateExistingSchema() passes when schema and primary key match
    • schema mismatch fails
    • primary key mismatch fails
    • ERROR_IF_NOT_EXISTS wrapper fails when target table is missing and does not create the table
  • Ran:
    • SchemaCreateModeMetadataApplierTest
    • MaxComputeMetadataApplierTest
    • MaxCompute spotless:check
    • git diff --check

Documentation

This introduces a new user-facing sink option:

  • schema.create.mode
    • CREATE_IF_NOT_EXISTS default
    • ERROR_IF_NOT_EXISTS

Documentation should mention that ERROR_IF_NOT_EXISTS is currently supported only by Fluss and MaxCompute. Other connectors will fail during schema application because they do not implement existing target schema validation yet.

@github-actions github-actions Bot added the docs Improvements or additions to documentation label May 29, 2026
@litiliu litiliu marked this pull request as draft May 29, 2026 08:14
@litiliu litiliu changed the title [FLINK-39797][fluss] Support disabling automatic table creation [FLINK-39797] Support disabling automatic table creation May 29, 2026
@litiliu litiliu force-pushed the FLINK-39797-disable-auto-create-table branch from 56f5a77 to 85e3cf9 Compare June 1, 2026 05:43
@litiliu litiliu force-pushed the FLINK-39797-disable-auto-create-table branch from 85e3cf9 to 86e3414 Compare June 1, 2026 05:55
@litiliu litiliu marked this pull request as ready for review June 1, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant