Skip to content

Glue DataLakeCatalog uses stale schema after ALTER ADD COLUMN #2085

Description

@alsugiliazova

Problem

For an Iceberg table in a Glue-backed DataLakeCatalog, consecutive ALTER TABLE ... ADD COLUMN commands succeed and the new columns are readable, but SHOW CREATE TABLE still returns the original schema. A subsequent attempt to drop one of the added columns also fails because ClickHouse cannot find it.

The same scenario works correctly with the REST catalog.

Tested with https://altinity-build-artifacts.s3.amazonaws.com/PRs/1841/78d272e18d7641866ac4f4ded573708cf005a3f9/build_arm_release/clickhouse-common-static_26.3.13.20001.altinityantalya_arm64.deb

Reproduction

Create an unpartitioned Iceberg table containing one nullable String column named name, then run:

SET allow_insert_into_iceberg = 1;

ALTER TABLE datalake_db.`namespace.table`
ADD COLUMN column_a Nullable(String);

ALTER TABLE datalake_db.`namespace.table`
ADD COLUMN column_b Nullable(Int64);

The columns are immediately available:

SELECT * FROM datalake_db.`namespace.table`;
Alice	\N	\N

Now run:

SHOW CREATE TABLE datalake_db.`namespace.table`;

Actual behavior

SHOW CREATE TABLE returns only the original column:

CREATE TABLE datalake_db.`namespace.table`
(
    `name` Nullable(String)
)
ENGINE = Iceberg('http://minio:9000/warehouse/data/')

column_a and column_b are missing even though both ALTER commands succeeded and SELECT * exposes both columns.

Dropping one of the added columns then fails:

SET allow_insert_into_iceberg = 1;
ALTER TABLE datalake_db.`namespace.table` DROP COLUMN column_a;
Code: 10. DB::Exception: Wrong column name.
Cannot find column `column_a` to drop.
(NOT_FOUND_COLUMN_IN_BLOCK)

Expected behavior

SHOW CREATE TABLE should return the current Iceberg schema:

(
    `name` Nullable(String),
    `column_a` Nullable(String),
    `column_b` Nullable(Int64)
)

All operations that resolve the table schema should use the same current metadata version.

DROP COLUMN column_a should succeed and leave name and column_b in the current schema.

Metadata

Metadata

Assignees

Labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions