[WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore#6416
[WIP]HIVE-29520: Add configuration to enable/disable ACID functionality in Hive Metastore#6416VenuReddy2103 wants to merge 1 commit intoapache:masterfrom
Conversation
VenuReddy2103
changed the title
|
why do we need new toggle for that? |
| taskNames.removeAll(ImmutableSet.of(MetastoreConf.ACID_OPEN_TXNS_COUNTER_SERVICE_CLASS, | ||
| MetastoreConf.ACID_HOUSEKEEPER_SERVICE_CLASS, | ||
| MetastoreConf.ACID_TXN_CLEANER_SERVICE_CLASS, | ||
| MetastoreConf.MATERIALZIATIONS_REBUILD_LOCK_CLEANER_TASK_CLASS)); |
There was a problem hiding this comment.
Hive provides support for iceberg tables materialized views
|
VenuReddy2103
force-pushed
the
HIVE-29520
branch
from
09ebdbd to
25bd551
Compare
|
| MetastoreConf.getStringCollection(configuration, MetastoreConf.ConfVars.TASK_THREADS_REMOTE_ONLY); | ||
| if (!MetastoreConf.getBoolVar(configuration, MetastoreConf.ConfVars.METASTORE_SUPPORT_ACID)) { | ||
| taskNames.removeAll(ImmutableSet.of(MetastoreConf.ACID_OPEN_TXNS_COUNTER_SERVICE_CLASS, | ||
| MetastoreConf.ACID_HOUSEKEEPER_SERVICE_CLASS, |
There was a problem hiding this comment.
can we honor this METASTORE_SUPPORT_ACID per task, such as CompactionHouseKeeperService.java. If the task is disabled completely, then make runFrequency return 0.
There was a problem hiding this comment.
Have tried to handle at LeaderElection.LeadershipStateListener level most of the cases like CompactorTasks, HouseKeepingTasks instead of leaf level (MetastoreTaskThread). Thought we can avoid all those instance creations, setConf()/init() which internally gets TxnUtils.getTxnStore(conf) and internally setConf() of txnHandler etc if we do at LeadershipStateListener level.
But in case of CompactionHouseKeeperService, a part of its functionality((i.e., txnHandler::purgeCompactionHistory alone) is required when MetastoreConf.ConfVars.METASTORE_SUPPORT_ACID is disabled. So have handled at leaf level.
What is your opinion ?
4 participants
What changes were proposed in this pull request?
Introduce a configuration to enable/disable ACID functionality in Hive Metastore
Why are the changes needed?
With the increasing adoption of the Apache Iceberg table format, many deployments no longer rely on ACID (transactional) tables provided by Apache Hive. In such environments, ACID-related functionality in the Hive Metastore becomes unnecessary.
Currently, Hive Metastore initializes and runs several background threads and services that are specifically designed to support ACID operations (e.g., compaction handling, transaction management, cleanup tasks). These components continue to run regardless of whether ACID tables are actually used in the deployment.
Running ACID-related services in environments that do not use ACID tables leads to:
Providing a mechanism to disable ACID functionality would streamline such deployments and improve efficiency.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added unittests