HIVE-29736: MSCK REPAIR TABLE throws "partition already exists" exception when Metastore URI contains an explicit port and HDFS uses HA Nameservice

[Aggarwal-Raghav]

What changes were proposed in this pull request?

This is a behaviour change b/w Hive3 vs Hive4 done in HIVE-23347

It fixes an edge case in HiveMetaStoreChecker where MSCK REPAIR TABLE crashes with a MetastoreException when comparing partitions on an HDFS HA cluster. When a partition's Metastore URI contains an explicit port (e.g., :8020 ) but the physical Hadoop FileSystem returns paths normalized to the HA nameservice.

The PR updates the validation loop to ignore these leftover physical directories if they perfectly map to a partition that was already successfully matched against the Metastore database.

For example:

Desc formatted tbl -> hdfs://namenode/
Desc formatted tbl partition.. -> hdfs://namenode:8020/;

the checkTable will detemine the diff of HDFS vs HMS lookup and it compares complete Path URI and picks the already existing / register partition as a candidate for msck operation. Causing the following stacktrace:

org.apache.hadoop.hive.metastore.api.MetastoreException: The partition 'dummy_tbl_name:xxxx=xxxx/xxxxxxx=xxxxxx/dt=2026-06-07' already exists for table aip_auditlog
	at org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.findUnknownPartitions(HiveMetaStoreChecker.java:468)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:401)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.checkTable(HiveMetaStoreChecker.java:287)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreChecker.checkMetastore(HiveMetaStoreChecker.java:162)
	at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1812)
	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:411)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:255)
	at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Why are the changes needed?

To ensure msck passes.

Does this PR introduce any user-facing change?

NO

How was this patch tested?

On Cluster and wrote a resonating issue Junit

[Aggarwal-Raghav]

I'm guessing the spark written partition are also contributing to this issue.

Sample repro on cluster without HA service enabled:

create table bug (id int) partitioned by (country string);
alter table bug add partition (country='ABC');
desc formatted bug;                                                           -- gave location hdfs://namenode:8020/
desc formatted bug partition(country='ABC');
alter table bug partition (country='ABC') SET LOCATION 'hdfs://namenode/...../';               -- So removed :8020
msck repair table bug;

NOTE, ensure hdfs dfs -ls hdfs://namenode/path_to_table/ and hdfs://namenode:8020/path_to_table/ both are working in cluster. In localhost (my local mac setup) it was not working for me but on VM/cluster it was repro-ing

[Aggarwal-Raghav]

@deniskuzZ , @dengzhhu653 can you please review this. This is very impactful bug, causing msck repair fail in prod.
This PR will change the behaviour going forward. Please provide feedback on this?
HOT code for this PR:

HiveMetaStoreChecker#checkTable() inside this allPartDirs.remove(partPath) is crucual

[Aggarwal-Raghav]

UT failure seems un-related, passing in local, Re-triggering CI

 -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hive.metastore.TestMetastoreLeaseLeader
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.74 s -- in org.apache.hadoop.hive.metastore.TestMetastoreLeaseLeader
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  34.608 s
[INFO] Finished at: 2026-07-16T07:17:47+05:30
[INFO] ------------------------------------------------------------------------

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud