HIVE-29735: TxnStoreMutex.releaseLocks does not release the connection on failure

[deniskuzZ]

What changes were proposed in this pull request?

Complete the transaction on all paths in releaseLocks by calling TransactionContext.close() in a finally block

Why are the changes needed?

BugFix

Does this PR introduce any user-facing change?

No

How was this patch tested?

Local repro

[difin]

LGTM

[Aggarwal-Raghav]

@deniskuzZ , will it cause this stacktrace :-)

2026-07-14T16:45:08,491 ERROR [Metastore Scheduled Worker 13] service.AcidTxnCleanerService: Unexpected exception in thread: Metastore Scheduled Worker 13, message: Unable to lock TxnCleaner due to: PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1", "MT_KEY2") VALUES(?, 0)]; Deadlock found when trying to get lock; try restarting transaction; nested exception is com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
java.lang.RuntimeException: Unable to lock TxnCleaner due to: PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1", "MT_KEY2") VALUES(?, 0)]; Deadlock found when trying to get lock; try restarting transaction; nested exception is com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
	at org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:85) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.metastore.txn.service.AcidTxnCleanerService.run(AcidTxnCleanerService.java:62) ~[hive-exec-4.0.1.jar:4.0.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.springframework.dao.DeadlockLoserDataAccessException: PreparedStatementCallback; SQL [INSERT INTO "AUX_TABLE" ("MT_KEY1", "MT_KEY2") VALUES(?, 0)]; Deadlock found when trying to get lock; try restarting transaction; nested exception is com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
	at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:268) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1575) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:81) ~[hive-exec-4.0.1.jar:4.0.1]
	... 7 more
Caused by: com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:123) ~[mysql-jdbc-driver.jar:8.0.30]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-jdbc-driver.jar:8.0.30]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:916) ~[mysql-jdbc-driver.jar:8.0.30]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061) ~[mysql-jdbc-driver.jar:8.0.30]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009) ~[mysql-jdbc-driver.jar:8.0.30]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320) ~[mysql-jdbc-driver.jar:8.0.30]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994) ~[mysql-jdbc-driver.jar:8.0.30]
	at org.apache.hive.com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hive.com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331) ~[spring-jdbc-5.3.39.jar:5.3.39]
	at org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:81) ~[hive-exec-4.0.1.jar:4.0.1]
	... 7 more

[Aggarwal-Raghav]

@deniskuzZ , can you please share the repo case/stacktrace of error you faced? I faced the above stacktrace today in prod as per timestamp. HIVE-28669 is backported in 4.0.1 in my case.

So I was thinking of this in acquireLock()? Did you face this or my stacktrace is different or maybe they are two halves of the same coin?

diff --git i/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStoreMutex.java w/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStoreMutex.java
index 3dad12fc1f..8ea4b0cbd9 100644
--- i/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStoreMutex.java
+++ w/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStoreMutex.java
@@ -81,7 +81,14 @@ public LockHandle acquireLock(String key) throws MetaException {
           jdbcResource.getJdbcTemplate().update("INSERT INTO \"AUX_TABLE\" (\"MT_KEY1\", \"MT_KEY2\") VALUES(:key, 0)", paramSource);
           context.createSavepoint();
         } catch (DataAccessException e) {
-          if (!jdbcResource.getDatabaseProduct().isDuplicateKeyError(e)) {
+          boolean isDeadlock = false;
+          try {
+            isDeadlock = jdbcResource.getDatabaseProduct().isDeadlock(TxnUtils.getSqlException(e));
+          } catch (IllegalArgumentException ex) {
+            // Ignore if we can't extract a SQLException
+          }
+
+          if (!jdbcResource.getDatabaseProduct().isDuplicateKeyError(e) && !isDeadlock) {
             throw new RuntimeException("Unable to lock " + key + " due to: " + SqlRetryHandler.getMessage(e), e);
           }
           //if here, it means a concrurrent acquireLock() inserted the 'key'

[deniskuzZ]

@Aggarwal-Raghav
Connection to backend db got lost during TxnStoreMutex.LockHandleImpl.releaseLocks() and there is no rollback or commit happening in the catch block.

2026-06-27 00:35:58,992 WARN com.zaxxer.hikari.pool.ProxyConnection: [Thread-16]: mutex - Connection com.mysql.cj.jdbc.ConnectionImpl@383f41dd marked as broken because of SQLSTATE(08S01), ErrorCode(0)
com.mysql.cj.jdbc.exceptions.CommunicationsException: The last packet successfully received from the server was 1,141,329 milliseconds ago. The last packet sent successfully to the server was 1,141,329 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:165) ~[mysql-connector-j.jar:9.0.0]
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:55) ~[mysql-connector-j.jar:9.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:987) ~[mysql-connector-j.jar:9.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1165) ~[mysql-connector-j.jar:9.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1100) ~[mysql-connector-j.jar:9.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1466) ~[mysql-connector-j.jar:9.0.0]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1083) ~[mysql-connector-j.jar:9.0.0]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) ~[hive-exec.jar]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) ~[hive-exec.jar]
at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) ~[spring-jdbc-5.3.39.jar:5.3.39]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648) ~[spring-jdbc-5.3.39.jar:5.3.39]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) ~[spring-jdbc-5.3.39.jar:5.3.39]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:981) ~[spring-jdbc-5.3.39.jar:5.3.39]
at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:331) ~[spring-jdbc-5.3.39.jar:5.3.39]
at org.apache.hadoop.hive.metastore.txn.TxnStoreMutex$LockHandleImpl.releaseLocks(TxnStoreMutex.java:186) ~[hive-exec.jar]

As a result subsequent calls are failing with connections closed in Initiator.

ERROR org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-16]:
  Initiator loop caught unexpected exception this time through the loop
org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback;
  uncategorized SQLException for SQL [SELECT "MT_COMMENT", "MT_KEY2" FROM "AUX_TABLE"
  WHERE "MT_KEY1" = ? for update]; ... Connection is closed
...
at org.apache.hadoop.hive.metastore.txn.TxnStoreMutex.acquireLock(TxnStoreMutex.java:79)
at org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:123)

I think HIVE-28669: Deadlock found when TxnStoreMutex trying to acquireLock might be related to your case

[deniskuzZ]

added test

[Aggarwal-Raghav]

Thanks @deniskuzZ for adding test. I'll go through it later and it might be slightly different from my issue as i have HIVE-28669 merged in my custom branch. I think its a bug. I'll investigate it in upcoming days and will raise a seprate JIRA if i find its open on master branch.

Your current PR LGTM

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud