Iceberg ALTER TABLE via REST catalog fails when catalog commit fails after metadata is already written to S3

[DimensionWieldr]

Describe the bug
PR #1841 adds Iceberg ALTER support by writing metadata to S3 and then committing to the REST catalog.
When running 150+ ALTERs (resulting in 150+ metadata files) it eventually throws a catalog commit failed error.

To Reproduce
Basically do some sort of ALTER many many times:

ALTER TABLE datalake.`namespace.iceberg_table` RENAME COLUMN some_column TO renamed_column

It is difficult to reproduce manually, but the relevant regression test can be run at:

cd clickhouse-regression/iceberg

using 26.3.13.20001.altinityantalya PR 1841 build:

./regression.py --clickhouse https://altinity-build-artifacts.s3.amazonaws.com/PRs/1841/78d272e18d7641866ac4f4ded573708cf005a3f9/build_amd_release/clickhouse-common-static_26.3.13.20001.altinityantalya_amd64.deb --only "/iceberg/iceberg engine/rest catalog/alter support/*" -l test.log

Expected behavior
Many consecutive or parallel ALTERs (modify, add, delete, rename, etc.) should work on Iceberg tables.

Actual Behaviour

Received exception from server (version 26.3.13):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Iceberg alter: catalog commit failed for 's3://warehouse/data/metadata/v158-374279d8-f15e-4221-a895-9f71e342ac4f.metadata.json' after metadata file was written successfully. (DATALAKE_DATABASE_ERROR)
(query: ALTER TABLE datalake_db_8534952e_8488_11f1_9125_cb5bdb943619.`namespace_851ef23c_8488_11f1_9125_cb5bdb943619.iceberg_table_851ef278_8488_11f1_9125_cb5bdb943619` RENAME COLUMN new_column_0277ad78_8489_11f1_9125_cb5bdb943619 TO renamed_151816d4_8489_11f1_9125_cb5bdb943619)

Server error log

2026.07.21 00:15:23.412753 [ 35 ] {} <Warning> AsynchronousMetrics: Hardware monitor 'thinkpad', sensor '2' exists but could not be read: errno: 6, strerror: No such device or address.
2026.07.21 00:15:23.418920 [ 35 ] {} <Warning> AsynchronousMetrics: Hardware monitor 'thinkpad', sensor '8' exists but could not be read: errno: 6, strerror: No such device or address.
2026.07.21 00:15:23.423055 [ 35 ] {} <Warning> Application: Integrity check of the executable skipped because the reference checksum could not be read.
2026.07.21 00:15:23.439637 [ 35 ] {} <Warning> ConfigProcessor: Include not found: compression
2026.07.21 00:15:23.448474 [ 35 ] {} <Warning> Access(local_directory): File /var/lib/clickhouse/access/users.list doesn't exist
2026.07.21 00:15:23.448477 [ 35 ] {} <Warning> Access(local_directory): Recovering lists in directory /var/lib/clickhouse/access/
2026.07.21 00:15:23.450034 [ 35 ] {} <Warning> ConfigProcessor: Include not found: networks
2026.07.21 00:15:23.453191 [ 35 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `sudo sh -c 'echo 1 > /proc/sys/kernel/task_delayacct'` or by using sysctl.
2026.07.21 00:19:33.897769 [ 44 ] {f22d42ed-de72-4caa-960e-3c878ca7fd7f} <Warning> RestCatalog(s3://bucket1/): Iceberg REST updateMetadata for namespace_851ef23c_8488_11f1_9125_cb5bdb943619.iceberg_table_851ef278_8488_11f1_9125_cb5bdb943619 failed: DB::HTTPException: Received error from remote server http://ice-rest-catalog:5000/v1/namespaces/namespace_851ef23c_8488_11f1_9125_cb5bdb943619/tables/iceberg_table_851ef278_8488_11f1_9125_cb5bdb943619. HTTP status code: 400 'Bad Request', body length: 118 bytes, body: '{"error":{"message":"Cannot set last added schema: no schema has been added","type":"ValidationException","code":400}}'
2026.07.21 00:19:33.902380 [ 44 ] {f22d42ed-de72-4caa-960e-3c878ca7fd7f} <Error> executeQuery: Code: 736. DB::Exception: Iceberg alter: catalog commit failed for 's3://warehouse/data/metadata/v158-374279d8-f15e-4221-a895-9f71e342ac4f.metadata.json' after metadata file was written successfully. (DATALAKE_DATABASE_ERROR) (version 26.3.13.20001.altinityantalya) (from 127.0.0.1:44822) (query 2, line 1) (in query: ALTER TABLE datalake_db_8534952e_8488_11f1_9125_cb5bdb943619.`namespace_851ef23c_8488_11f1_9125_cb5bdb943619.iceberg_table_851ef278_8488_11f1_9125_cb5bdb943619` RENAME COLUMN new_column_0277ad78_8489_11f1_9125_cb5bdb943619 TO renamed_151816d4_8489_11f1_9125_cb5bdb943619 ), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001790c6aa
1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000fb62b4e
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000fb624c9
3. DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000fb8df36
4. DB::Iceberg::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::shared_ptr<DB::IObjectStorage>, DB::DataLakeStorageSettings const&, DB::Iceberg::PersistentTableComponents&, String const&, DB::StorageID, std::shared_ptr<DataLake::ICatalog>) @ 0x000000001b096b10
5. DB::IcebergMetadata::alter(DB::AlterCommands const&, std::shared_ptr<DB::StorageObjectStorageConfiguration>, std::shared_ptr<DB::Context const>, DB::StorageID const&, std::shared_ptr<DataLake::ICatalog>) @ 0x000000001afc5a94
6. DB::DataLakeConfiguration<DB::StorageS3Configuration, DB::IcebergMetadata, true>::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, DB::StorageID const&, std::shared_ptr<DataLake::ICatalog>) @ 0x000000001a02d1f3
7. DB::StorageObjectStorage::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&) @ 0x000000001adffcd8
8. DB::StorageObjectStorageCluster::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&) @ 0x000000001ae16d34
9. DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001cbb7612
10. DB::InterpreterAlterQuery::execute() @ 0x000000001cbb31d3
11. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001d0ebc3e
12. DB::executeQuery(std::basic_string_view<char, std::char_traits<char>>, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001d0e6085
13. DB::TCPHandler::runImpl() @ 0x000000001ea8c6c0
14. DB::TCPHandler::run() @ 0x000000001eab17a4
15. Poco::Net::TCPServerConnection::start() @ 0x00000000243e4e47
16. Poco::Net::TCPServerDispatcher::run() @ 0x00000000243e5366
17. Poco::PooledThread::run() @ 0x000000002439de05
18. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000002439c38f
19. ? @ 0x0000000000094ac3
20. ? @ 0x0000000000126850

2026.07.21 00:19:33.902668 [ 44 ] {} <Error> TCPHandler: Code: 736. DB::Exception: Iceberg alter: catalog commit failed for 's3://warehouse/data/metadata/v158-374279d8-f15e-4221-a895-9f71e342ac4f.metadata.json' after metadata file was written successfully. (DATALAKE_DATABASE_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001790c6aa
1. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000fb62b4e
2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000fb624c9
3. DB::Exception::Exception<String&>(int, FormatStringHelperImpl<std::type_identity<String&>::type>, String&) @ 0x000000000fb8df36
4. DB::Iceberg::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::shared_ptr<DB::IObjectStorage>, DB::DataLakeStorageSettings const&, DB::Iceberg::PersistentTableComponents&, String const&, DB::StorageID, std::shared_ptr<DataLake::ICatalog>) @ 0x000000001b096b10
5. DB::IcebergMetadata::alter(DB::AlterCommands const&, std::shared_ptr<DB::StorageObjectStorageConfiguration>, std::shared_ptr<DB::Context const>, DB::StorageID const&, std::shared_ptr<DataLake::ICatalog>) @ 0x000000001afc5a94
6. DB::DataLakeConfiguration<DB::StorageS3Configuration, DB::IcebergMetadata, true>::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, DB::StorageID const&, std::shared_ptr<DataLake::ICatalog>) @ 0x000000001a02d1f3
7. DB::StorageObjectStorage::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&) @ 0x000000001adffcd8
8. DB::StorageObjectStorageCluster::alter(DB::AlterCommands const&, std::shared_ptr<DB::Context const>, std::unique_lock<std::timed_mutex>&) @ 0x000000001ae16d34
9. DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x000000001cbb7612
10. DB::InterpreterAlterQuery::execute() @ 0x000000001cbb31d3
11. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, std::unique_ptr<DB::ReadBuffer, std::default_delete<DB::ReadBuffer>>&, boost::intrusive_ptr<DB::IAST>&, std::shared_ptr<DB::ImplicitTransactionControlExecutor>, std::function<void ()>, DB::QueryResultDetails&) @ 0x000000001d0ebc3e
12. DB::executeQuery(std::basic_string_view<char, std::char_traits<char>>, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x000000001d0e6085
13. DB::TCPHandler::runImpl() @ 0x000000001ea8c6c0
14. DB::TCPHandler::run() @ 0x000000001eab17a4
15. Poco::Net::TCPServerConnection::start() @ 0x00000000243e4e47
16. Poco::Net::TCPServerDispatcher::run() @ 0x00000000243e5366
17. Poco::PooledThread::run() @ 0x000000002439de05
18. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000002439c38f
19. ? @ 0x0000000000094ac3
20. ? @ 0x0000000000126850

AI Suggested Fix Directions - not sure if helpful, but leaving it here just in case

1. Retry catalog commit on transient HTTP / conflict errors (PR Antalya 26.3: Iceberg client, handle namespaceExists error #1918 direction)
2. Use actual catalog current-schema-id for assert-current-schema-id, not new_schema_id - 1
3. Propagate REST catalog error into the 736 exception (or system.errors / query log) so failures are diagnosable
4. Consider commit ordering / atomicity — catalog-first or transactional pattern to avoid orphan metadata writes