HIVE-29643: Add missing primary keys to backend tables across differe…#6521
Open
saihemanth-cloudera wants to merge 5 commits into
Open
HIVE-29643: Add missing primary keys to backend tables across differe…#6521saihemanth-cloudera wants to merge 5 commits into
saihemanth-cloudera wants to merge 5 commits into
Conversation
asf-ci-hive
added
tests pending
tests unstable
tests passed
and removed
tests pending
tests unstable
labels
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the standalone metastore schema (Hive 4.3.0) to add missing primary keys to several transactional/compaction-related backend tables across supported RDBMS vendors, which is needed for enabling/operating HA mode reliably.
Changes:
- Add new surrogate identity/serial primary keys to previously keyless transactional tables (e.g.,
TXN_COMPONENTS,WRITE_SET, etc.) across Postgres/Oracle/MySQL/MSSQL/Derby schemas. - Replace certain unique indexes with proper primary keys (e.g.,
TXN_TO_WRITE_ID,NEXT_WRITE_ID) in both base schema and upgrade scripts. - Update test DB cleanup initialization inserts to specify column lists for sequence tables whose schemas gained new PK columns.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/utils/TestTxnDbUtil.java | Updates inserts into NEXT_LOCK_ID / NEXT_COMPACTION_QUEUE_ID to work with new PK columns. |
| standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-4.2.0-to-4.3.0.postgres.sql | Adds PKs/identity columns and converts some unique indexes to PKs for Postgres upgrades. |
| standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.3.0.postgres.sql | Adds PKs/identity columns and moves certain uniqueness from indexes to PK constraints in the Postgres base schema. |
| standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-4.2.0-to-4.3.0.oracle.sql | Adds PKs/identity columns and converts some unique indexes to PKs for Oracle upgrades. |
| standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.3.0.oracle.sql | Adds PKs/identity columns and converts some unique indexes to PKs in the Oracle base schema. |
| standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-4.2.0-to-4.3.0.mysql.sql | Adds auto-increment PKs and converts some unique indexes to PKs for MySQL upgrades. |
| standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.3.0.mysql.sql | Adds auto-increment PKs and replaces some unique indexes with PK constraints in the MySQL base schema. |
| standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-4.2.0-to-4.3.0.mssql.sql | Adds IDENTITY PKs and replaces some unique indexes with PK constraints for SQL Server upgrades. |
| standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.3.0.mssql.sql | Adds IDENTITY PKs and replaces some unique indexes with PK constraints in the SQL Server base schema. |
| standalone-metastore/metastore-server/src/main/sql/derby/upgrade-4.2.0-to-4.3.0.derby.sql | Adds identity PKs and converts some unique indexes to PKs for Derby upgrades. |
| standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.3.0.derby.sql | Adds identity PKs and replaces some unique indexes with PK constraints in the Derby base schema. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+6
to
+10
| ALTER TABLE TXN_COMPONENTS ADD COLUMN TC_ID BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY; | ||
| ALTER TABLE COMPLETED_TXN_COMPONENTS ADD COLUMN CTC_ID BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY; | ||
| ALTER TABLE COMPACTION_METRICS_CACHE ADD COLUMN CMC_ID BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY; | ||
| ALTER TABLE WRITE_SET ADD COLUMN WS_ID BIGINT PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY; | ||
|
|
Member
|
Sounds like the AI comments are valid, can you check them please? |
|
| ); | ||
| INSERT INTO TMP_TXN_COMPONENTS (TC_TXNID, TC_DATABASE, TC_TABLE, TC_PARTITION, TC_OPERATION_TYPE, TC_WRITEID) | ||
| SELECT TC_TXNID, TC_DATABASE, TC_TABLE, TC_PARTITION, TC_OPERATION_TYPE, TC_WRITEID FROM TXN_COMPONENTS; | ||
| RENAME TABLE TXN_COMPONENTS TO TXN_COMPONENTS_OLD, TMP_TXN_COMPONENTS TO TXN_COMPONENTS; |
| ); | ||
| INSERT INTO TMP_COMPLETED_TXN_COMPONENTS (CTC_TXNID, CTC_DATABASE, CTC_TABLE, CTC_PARTITION, CTC_TIMESTAMP, CTC_WRITEID, CTC_UPDATE_DELETE) | ||
| SELECT CTC_TXNID, CTC_DATABASE, CTC_TABLE, CTC_PARTITION, CTC_TIMESTAMP, CTC_WRITEID, CTC_UPDATE_DELETE FROM COMPLETED_TXN_COMPONENTS; | ||
| RENAME TABLE COMPLETED_TXN_COMPONENTS TO COMPLETED_TXN_COMPONENTS_OLD, TMP_COMPLETED_TXN_COMPONENTS TO COMPLETED_TXN_COMPONENTS; |
| ); | ||
| INSERT INTO TMP_COMPACTION_METRICS_CACHE (CMC_DATABASE, CMC_TABLE, CMC_PARTITION, CMC_METRIC_TYPE, CMC_METRIC_VALUE, CMC_VERSION) | ||
| SELECT CMC_DATABASE, CMC_TABLE, CMC_PARTITION, CMC_METRIC_TYPE, CMC_METRIC_VALUE, CMC_VERSION FROM COMPACTION_METRICS_CACHE; | ||
| RENAME TABLE COMPACTION_METRICS_CACHE TO COMPACTION_METRICS_CACHE_OLD, TMP_COMPACTION_METRICS_CACHE TO COMPACTION_METRICS_CACHE; |
| ); | ||
| INSERT INTO TMP_WRITE_SET (WS_DATABASE, WS_TABLE, WS_PARTITION, WS_TXNID, WS_COMMIT_ID, WS_OPERATION_TYPE) | ||
| SELECT WS_DATABASE, WS_TABLE, WS_PARTITION, WS_TXNID, WS_COMMIT_ID, WS_OPERATION_TYPE FROM WRITE_SET; | ||
| RENAME TABLE WRITE_SET TO WRITE_SET_OLD, TMP_WRITE_SET TO WRITE_SET; |
| ); | ||
| INSERT INTO TMP_MIN_HISTORY_WRITE_ID (MH_TXNID, MH_DATABASE, MH_TABLE, MH_WRITEID) | ||
| SELECT MH_TXNID, MH_DATABASE, MH_TABLE, MH_WRITEID FROM MIN_HISTORY_WRITE_ID; | ||
| RENAME TABLE MIN_HISTORY_WRITE_ID TO MIN_HISTORY_WRITE_ID_OLD, TMP_MIN_HISTORY_WRITE_ID TO MIN_HISTORY_WRITE_ID; |
saihemanth-cloudera
force-pushed
the
HIVE-29643
branch
from
July 17, 2026 17:51
56c2bf1 to
a160945
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…nt types of vendors
What changes were proposed in this pull request?
Added primary keys to the backend schema tables
Why are the changes needed?
To add support HA mode in DB, we need to have primary keys for the existing tables.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests