THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
As operations modify the table more and more delta files are created and need to be compacted to maintain adequate performance. There are three types of compactions, minor, major and rebalance.
- Minor compaction takes a set of existing delta files and rewrites them to a single delta file per bucket.
- Major compaction takes one or more delta files and the base file for the bucket and rewrites them into a new base file per bucket. Major compaction is more expensive but is more effective.
- More information about rebalance compaction can be found here: Rebalance compaction
...
Configuration key | Values | Location | Notes | ||||
Default: org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager Value required for transactions: org.apache.hadoop.hive.ql.lockmgr.DbTxnManager | Client/ | DummyTxnManager replicates pre Hive-0.13 behavior and provides no transactions. | |||||
| hive.txn.strict.locking.mode | Default: true | Client/ HiveServer2 | In strict mode non-ACID resources use standard R/W lock semantics, e.g. INSERT will acquire exclusive lock. In non-strict mode, for non-ACID resources, INSERT will only acquire shared lock, which allows two concurrent writes to the same partition but still lets lock manager prevent DROP TABLE etc. when the table is being written to (as of Hive 2.2.0). | ||||
hive.txn.timeout deprecated. Use metastore.txn.timeout instead | Default: 300 | Client/ Metastore | Time after which transactions are declared aborted if the client has not sent a heartbeat, in seconds. It's critical that this property has the same value for all components/services.5 | ||||
| hive.txn.heartbeat.threadpool.size deprecated - but still in use | Default: 5 | Client/ HiveServer2 | The number of threads to use for heartbeating (as of Hive 1.3.0 and 2.0.0). | ||||
| hive.timedout.txn.reaper.start deprecated | Default: 100s | Metastore | Time delay of first reaper (the process which aborts timed-out transactions) run after the metastore starts (as of Hive 1.3.0). Controls AcidHouseKeeperServcie above. | ||||
hive.timedout.txn.reaper.interval deprecated | Default: 180s | Metastore | Time interval describing how often the reaper (the process which aborts timed-out transactions) runs (as of Hive 1.3.0). Controls AcidHouseKeeperServcie above. | ||||
hive.txn.max.open.batch deprecated. Use metastore.txn.max.open.batch instead | Default: 1000 | Client | Maximum number of transactions that can be fetched in one call to open_txns().1 | ||||
| hive.max.open.txns deprecated. Use metastore.max.open.txns instead. | Default: 100000 | HiveServer2/ Metastore | Maximum number of open transactions. If current open transactions reach this limit, future open transaction requests will be rejected, until the number goes below the limit. (As of Hive 1.3.0 and 2.1.0.) | ||||
| hive.count.open.txns.interval deprecated. Use metastore.count.open.txns.interval instead. | Default: 1s | HiveServer2/ Metastore | Time in seconds between checks to count open transactions (as of Hive 1.3.0 and 2.1.0). | ||||
| hive.txn.retryable.sqlex.regex deprecated. Use metastore.txn.retryable.sqlex.regex instead. | Default: "" (empty string) | HiveServer2/ Metastore | Comma separated list of regular expression patterns for SQL state, error code, and error message of retryable SQLExceptions, that's suitable for the Hive metastore database (as of Hive 1.3.0 and 2.1.0). For an example, see Configuration Properties. | ||||
| hive.compactorcompaction.initiatormerge.onenabled | Default: false Value required for transactions: true (for exactly one instance of the Thrift metastore service) | HiveServer2 | Enables merge-based compaction which is a compaction optimization when few ORC delta files are present | ||||
| hive.compactor.initiator.duration.update.interval | Default: 60s | HiveServer2 | Time in seconds that drives the update interval of compaction_initiator_duration metric. | ||||
hive.compactor.initiator.on deprecated. Use metastore.compactor.initiator.on instead. | Default | Metastore | Default: false Value required for transactions: true (for exactly one instance of the Thrift metastore service) | Metastore | Whether to run the cleaner initiator thread on this metastore instance. BeforePrior to Hive 40.0 Cleaner thread can be started/stopped with config hive.compactor.initiator.on. This config helps to enable/disable initiator/cleaner threads independently | ||
Default: 0 Value required for transactions: > 0 on at least one instance of the Thrift metastore service | Metastore | How many compactor worker threads to run on this metastore instance.2 | |||||
3.0 it's critical that this is enabled on exactly one standalone metastore service instance (not enforced yet). As of Hive 1.3.0 this property may be enabled on any number of standalone metastore instances. | |||||||
| hive.compactor.cleaner.duration.update.interval | Default: 60s | HiveServer2 | Time in seconds that drives the update interval of compaction_cleaner_duration metric. | Default: 86400 | Metastore | Time in seconds after which a compaction job will be declared failed and the compaction re-queued. | |
| .run.interval .on deprecated. Use metastore.compactor.cleaner.on instead. | Default: 5000 | Metastore | false Value required for transactions: true (for exactly one instance of the Thrift metastore service) | Metastore | Whether to run the cleaner thread on this metastore instance. Before Hive 4.0.0 Cleaner thread can be started/stopped with config hive.compactor.initiator.on. This config helps to enable/disable initiator/cleaner threads independently Time in milliseconds between runs of the cleaner thread. (Hive 0.14.0 and later.) | ||
| hive.compactor.checkcleaner.threads.num interval | Default: 3001 | Metastore | HiveServer2 | Enables parallelization of the cleaning directories after compaction, that includes many file | |||
| hive.compactor.deltacompact.numinsert.only threshold | Default: 10true | Metastore | HiveServer2 | Whether the compactor should compact insert-only tables. A safety switchNumber of delta directories in a table or partition that will trigger a minor compaction. | |||
| hive.compactor.deltacrud.pctquery.thresholdbased | Default: 0.1false | Metastore | HiveServer2 | Means compaction on full CRUD tables is done via queries. Compactions on insert-only tables will always run via queries regardless of the value of this configuration.Percentage (fractional) size of the delta files relative to the base that will trigger a major compaction. 1 = 100%, so the default 0.1 = 10%. | |||
| hive.compactor.abortedtxngather.thresholdstats | Default: 1000 | Metastore | Number of aborted transactions involving a given table or partition that will trigger a major compaction. | ||||
| hive.compactor.aborted.txn.time.threshold | Default: 12h | Metastore | Age of table/partition's oldest aborted transaction when compaction will be triggered. Default time unit is: hours. Set to a negative number to disable. | ||||
| Default: 500 | Metastore | Maximum number of delta files that the compactor will attempt to handle in a single job (as of Hive 1.3.0).4 | |||||
| Default: "" (empty string) | Metastore | Used to specify name of Hadoop queue to which Compaction jobs will be submitted. Set to empty string to let Hadoop choose the queue (as of Hive 1.3.0). | |||||
| Compaction History | |||||||
hive.compactor.history.retention.succeeded | Default: 3 | Metastore | Number of successful compaction entries to retain in history (per partition). | ||||
hive.compactor.history.retention.failed | Default: 3 | Metastore | Number of failed compaction entries to retain in history (per partition). | ||||
hive.compactor.history.retention.attempted | Default: 2 | Metastore | Number of attempted compaction entries to retain in history (per partition). | ||||
hive.compactor.initiator.failed.compacts.threshold | Default: 2 | Metastore | Number of of consecutive failed compactions for a given partition after which the Initiator will stop attempting to schedule compactions automatically. It is still possible to use ALTER TABLE to initiate compaction. Once a manually initiated compaction succeeds auto initiated compactions will resume. Note that this must be less than hive.compactor.history.retention.failed. | ||||
hive.compactor.history.reaper.interval | Default: 2m | Metastore | Controls how often the process to purge historical record of compactions runs. | ||||
true | HiveServer2 | If set to true MAJOR compaction will gather stats if there are stats already associated with the table/partition. Turn this off to save some resources and the stats are not used anyway. This is a replacement for the HIVE_MR_COMPACTOR_GATHER_STATS config, and works both for MR and Query based compaction. | |||||
| metastore.compactor.initiator.failed.retry.time | Default: 7d | Metastore | Time after Initiator will ignore metastore.compactor.initiator.failed.compacts.threshold and retry with compaction again. This will try to auto heal tables with previous failed compaction without manual intervention. Setting it to 0 or negative value will disable this feature. | ||||
| metastore.compactor.long.running.initiator.threshold.warning | Default: 6h | Metastore | Initiator cycle duration after which a warning will be logged. Default time unit is: hours | ||||
| metastore.compactor.long.running.initiator.threshold.error | Default: 12h | Metastore | Initiator cycle duration after which an error will be logged. Default time unit is: hours | ||||
| hive.compactor.worker.sleep.time | Default: 10800ms | HiveServer2 | Time in milliseconds for which a worker threads goes into sleep before starting another iteration in case of no launched job or error | ||||
| hive.compactor.worker.max.sleep.time | Default: 320000ms | HiveServer2 | Max time in milliseconds for which a worker threads goes into sleep before starting another iteration used for backoff in case of no launched job or error | ||||
hive.compactor.worker.threads deprecated. Use metastore.compactor.worker.threads instead. | Default: 0 Value required for transactions: > 0 on at least one instance of the Thrift metastore service | Metastore | How many compactor worker threads to run on this metastore instance.2 | ||||
Default: 86400s | Metastore | Time in seconds after which a compaction job will be declared failed and the compaction re-queued. | |||||
| hive.compactor.cleaner.run.interval | Default: 5000ms | Metastore | Time in milliseconds between runs of the cleaner thread. (Hive 0.14.0 and later.) | ||||
Default: 300s | Metastore | Time in seconds between checks to see if any tables or partitions need to be compacted.3 | |||||
Default: 10 | Metastore | Number of delta directories in a table or partition that will trigger a minor compaction. | |||||
Default: 0.1 | Metastore | Percentage (fractional) size of the delta files relative to the base that will trigger a major compaction. 1 = 100%, so the default 0.1 = 10%. | |||||
Default: 1000 | Metastore | Number of aborted transactions involving a given table or partition that will trigger a major compaction. | |||||
| hive.compactor.aborted.txn.time.threshold | Default: 12h | Metastore | Age of table/partition's oldest aborted transaction when compaction will be triggered. Default time unit is: hours. Set to a negative number to disable. | ||||
| Default: 500 | Metastore | Maximum number of delta files that the compactor will attempt to handle in a single job (as of Hive 1.3.0).4 | |||||
| Default: "" (empty string) | Metastore | Used to specify name of Hadoop queue to which Compaction jobs will be submitted. Set to empty string to let Hadoop choose the queue (as of Hive 1.3.0). | |||||
| hive.compactor.request.queue | Default: 1 | HiveServer2 | Enables parallelization of the checkForCompaction operation, that includes many file metadata checks and may be expensive | ||||
| hive.split.grouping.mode | Default: query (Allowed values: query, compactor) | HiveServer2 | This is set to compactor from within the query based compactor. This enables the Tez SplitGrouper to group splits based on their bucket number, so that all rows from different bucket files for the same bucket number can end up in the same bucket file after the compaction. | ||||
hive.txn.xlock.iow | Default: true | HiveServer2 | Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire Exclusive locks for transactional tables. This ensures that inserts (w/o overwrite) running concurrently are not hidden by the INSERT OVERWRITE. | ||||
hive.txn.xlock.write | Default: true | HiveServer2 | Manages concurrency levels for ACID resources. Provides better level of query parallelism by enabling | ||||
| metastore.acidmetrics.ext.on | Default: true | HiveServer2 | Whether to collect additional acid related metrics outside of the acid metrics service. | ||||
| Compaction History | |||||||
hive.compactor.history.retention.succeeded deprecated. Use metastore.compactor.history.retention.succeeded instead | Default: 3 | Metastore | Number of successful compaction entries to retain in history (per partition). | ||||
hive.compactor.history.retention.failed deprecated. Use metastore.compactor.history.retention.failed instead. | Default: 3 | Metastore | Number of failed compaction entries to retain in history (per partition). | ||||
hive.compactor.history.retention.attempted deprecated. Use metastore.compactor.history.retention.did.not.initiate instead. | Default: 2 | Metastore | Number of attempted compaction entries to retain in history (per partition). | ||||
hive.compactor.initiator.failed.compacts.threshold deprecated. Use metastore.compactor.initiator.failed.compacts.threshold instead. | Default: 2 | Metastore | Number of of consecutive failed compactions for a given partition after which the Initiator will stop attempting to schedule compactions automatically. It is still possible to use ALTER TABLE to initiate compaction. Once a manually initiated compaction succeeds auto initiated compactions will resume. Note that this must be less than hive.compactor.history.retention.failed. | ||||
| metastore.compactor.initiator.failed.compacts.threshold | Default: 2 (Allowed between 1 and 20) | Metastore | Number of consecutive compaction failures (per table/partition) after which automatic compactions will not be scheduled any more. Note that this must be less than hive.compactor.history.retention.failed. | ||||
hive.compactor.history.reaper.interval deprecated. metastore.acid.housekeeper.interval handles it. | Default: 2m | Metastore | Controls how often the process to purge historical record of compactions runs. | ||||
ACID metrics | |||||||
metastore.acidmetrics.check.interval | Default: 300s | Metastore | Time in seconds between acid related metric collection runs. | ||||
| metastore.acidmetrics.thread.on | Default: true | Metastore | Whether to run acid related metrics collection on this metastore instance. | ||||
| metastore.deltametrics.delta.num.threshold | Deafult: 100 | Metastore | The minimum number of active delta files a table/partition must have in order to be included in the ACID metrics report. | ||||
| metastore.deltametrics.delta.pct.threshold | Default: 0.01 | Metastore | Percentage (fractional) size of the delta files relative to the base directory. Deltas smaller than this threshold count as small deltas. Default 0.01 = 1%.) | ||||
| metastore.deltametrics.max.cache.size | Default: 100 (Allowed between 0 and 500) | Metastore | Size of the ACID metrics cache, i.e. max number of partitions and unpartitioned tables with the most deltas that will be included in the lists of active, obsolete and small deltas. Allowed range is 0 to 500. | ||||
| metastore.deltametrics.obsolete.delta.num.threshold | Default: 100 | Metastore | The minimum number of obsolete delta files a table/partition must have in order to be included in the ACID metrics report. | ||||
1metastore1hive.txn.max.open.batch controls how many transactions streaming agents such as Flume or Storm open simultaneously. The streaming agent then writes that number of entries into a single file (per Flume agent or Storm bolt). Thus increasing this value decreases the number of delta files created by streaming agents. But it also increases the number of open transactions that Hive has to track at any given time, which may negatively affect read performance.
...
5If the value is not the same active transactions may be determined to be "timed out" and consequently Aborted. This will result in errors like "No such transaction...", "No such lock ..."
Configuration Values to Set
...
for Hive ACID (INSERT, UPDATE, DELETE)
In addition to the new parameters listed above, some existing parameters need to be set to support INSERT ... VALUES, UPDATE, and DELETE.
...
If the data in your system is not owned by the Hive user (i.e., the user that the Hive metastore runs as), then Hive will need permission to run as the user who owns the data in order to perform compactions. If you have already set up HiveServer2 to impersonate users, then the only additional work to do is assure that Hive has the right to impersonate users from the host running the Hive metastore. This is done by adding the hostname to hadoop.proxyuser.hive.hosts in Hadoop's core-site.xml file. If you have not already done this, then you will need to configure Hive to act as a proxy user. This requires you to set up keytabs for the user running the Hive metastore and add hadoop.proxyuser.hive.hosts and hadoop.proxyuser.hive.groups to Hadoop's core-site.xml file. See the Hadoop documentation on secure mode for your version of Hadoop (e.g., for Hadoop 2.5.1 it is at Hadoop in Secure Mode).
Compaction pooling
More in formation on compaction pooling can be found here: Compaction pooling
...
If a table is to be used in ACID writes (insert, update, delete) then the table property "transactional=true" must must be set on that table, starting with Hive 0.14.0. Note, once a table has been defined as an ACID table via TBLPROPERTIES ("transactional"="true"), it cannot be converted back to a non-ACID table, i.e., changing TBLPROPERTIES ("transactional"="false") is not allowed. Also, hive.txn.manager must be set to org.apache.hadoop.hive.ql.lockmgr.DbTxnManager either in hive-site.xml or in the beginning of the session before any query is run. Without Without those, inserts will be done in the old style; updates and deletes will be prohibited prior to HIVE-11716. Since HIVE-11716 operations . Operations on ACID tables without DbTxnManager are not allowed. However, this does not apply to Hive 0.13.0. Additional to "transactional=true", "transactional_properties=insert_only" can be defined to allow only insert transactions.
If a table owner does not wish the system to automatically determine when to compact, then the table property "NO_AUTO_COMPACTION" can be set. This will prevent all automatic compactions. Manual compactions can still be done with Alter Table/Partition Compact statements.
Table properties are set with the TBLPROPERTIES clause when a table is created or altered, as described in the Create Table and Alter Table Properties sections of Hive Data Definition Language. The "transactional" and "NO_AUTO_COMPACTION" table properties are case-sensitive in Hive releases 0.x and 1.0, but they are case-insensitive starting with release 1.1.0 (HIVE-8308).insensitive.
More compaction related options can be set via TBLPROPERTIES as of Hive 1.3.0 and 2.1.0. They can be set at both table-level via CREATE TABLE, and on request-level via ALTER TABLE/PARTITION COMPACT. These are used to override the Warehouse/table wide settings. For example, to override an MR property to affect a compaction job, one can add "compactor.<mr property name>=<value>" in either CREATE TABLE statement or when launching a compaction explicitly via ALTER TABLE. The "<mr property name>=<value>" will be set on JobConf of the compaction MR job. Similarly, "tblprops.<prop name>=<value>" can be used to set/override any table property which is interpreted by the code running on the cluster. Finally, "compactorthreshold.<prop name>=<value>" can be used to override properties from the "New Configuration Parameters for Transactions" table table above that end with ".threshold" and control when compactions are triggered by the system. Examples:
...
| Code Block | ||
|---|---|---|
| ||
ALTER TABLE table_name COMPACT 'minor'
WITH OVERWRITE TBLPROPERTIES ("compactor.mapreduce.map.memory.mb"="3072"); -- specify compaction map job properties
ALTER TABLE table_name COMPACT 'major'
WITH OVERWRITE TBLPROPERTIES ("tblprops.orc.compress.size"="8192"); -- change any other Hive table properties |
Talks and Presentations
The Art of Compaction by Kokila N at a Cloudera meetup.
Transactional Operations In Hive by Eugene Koifman at Dataworks Summit 2017, San Jose, CA, USA
...