Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Configuration key

Values

Location

Notes

hive.txn.manager 

Default: org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager

Value required for transactions: org.apache.hadoop.hive.ql.lockmgr.DbTxnManager

Client/
HiveServer2

DummyTxnManager replicates pre Hive-0.13 behavior and provides no transactions.

hive.txn.strict.locking.modeDefault: true

Client/ HiveServer2

In strict mode non-ACID resources use standard R/W lock semantics, e.g. INSERT will acquire exclusive lock. In non-strict mode, for non-ACID resources, INSERT will only acquire shared lock, which allows two concurrent writes to the same partition but still lets lock manager prevent DROP TABLE etc. when the table is being written to (as of Hive 2.2.0).

hive.txn.timeout deprecated. Use metastore.txn.timeout instead

Default: 300

Client/
HiveServer2/

Metastore 

Time after which transactions are declared aborted if the client has not sent a heartbeat, in seconds. It's critical that this property has the same value for all components/services.5

hive.txn.heartbeat.threadpool.size deprecated - but still in useDefault: 5

Client/

HiveServer2

The number of threads to use for heartbeating (as of Hive 1.3.0 and 2.0.0).
hive.timedout.txn.reaper.start deprecatedDefault: 100sMetastoreTime delay of first reaper (the process which aborts timed-out transactions) run after the metastore starts (as of Hive 1.3.0). Controls AcidHouseKeeperServcie above.

hive.timedout.txn.reaper.interval deprecated

Default: 180sMetastore

Time interval describing how often the reaper (the process which aborts timed-out transactions) runs (as of Hive 1.3.0). Controls AcidHouseKeeperServcie above.

hive.txn.max.open.batch deprecated. Use 

metastore.txn.max.open.batch instead

Default: 1000

Client

Maximum number of transactions that can be fetched in one call to open_txns().1

hive.max.open.txns deprecated. Use metastore.max.open.txns instead.Default: 100000

HiveServer2/

 Metastore

Maximum number of open transactions. If current open transactions reach this limit, future open transaction requests will be rejected, until the number goes below the limit. (As of Hive 1.3.0 and 2.1.0.)
hive.count.open.txns.interval deprecated. Use metastore.count.open.txns.interval instead.Default: 1s

HiveServer2/

 Metastore

Time in seconds between checks to count open transactions (as of Hive 1.3.0 and 2.1.0).
hive.txn.retryable.sqlex.regex deprecated. Use metastore.txn.retryable.sqlex.regex instead.Default: "" (empty string)

HiveServer2/

 Metastore

Comma separated list of regular expression patterns for SQL state, error code, and error message of retryable SQLExceptions, that's suitable for the Hive metastore database (as of Hive 1.3.0 and 2.1.0).

For an example, see Configuration Properties.

hive.compactor.initiator.on deprecated. Use 

metastore.compactor.initiator.on instead.

Default: false

Value required for transactions: true (for exactly one instance of the Thrift metastore service)

Metastore

Whether to run the initiator thread on this metastore instance. Prior to Hive 1.3.0 it's critical that this is enabled on exactly one standalone metastore service instance (not enforced yet).

As of Hive 1.3.0 this property may be enabled on any number of standalone metastore instances.


hive.compactor.cleaner.on deprecated. Use 

metastore.compactor.cleaner.on instead.

Default: false

Value required for transactions: true (for exactly one instance of the Thrift metastore service)

Metastore

Whether to run the cleaner thread on this metastore instance.

Before Hive 4.0.0 Cleaner thread can be started/stopped with config hive.compactor.initiator.on. This config helps to enable/disable initiator/cleaner threads independently

hive.compactor.crud.query.based

Default: false

HiveServer2Means compaction on full CRUD tables is done via queries. Compactions on insert-only tables will always run via queries regardless of the value of this configuration.
hive.compactor.gather.stats

Default: true

HiveServer2If set to true MAJOR compaction 
will gather stats if there are stats already associated with the table/partition.
Turn this off to save some resources and the stats are not used anyway.
This is a replacement for the HIVE_MR_COMPACTOR_GATHER_STATS config, and works both for MR and Query based 
compaction.
hive.compactor.worker.sleep.time

Default: 10800ms

HiveServer2Time in milliseconds for which a worker threads goes into sleep before starting another iteration 
in case of no launched job or error
hive.compactor.worker.max.sleep.time

Default: 320000ms

HiveServer2Max time in milliseconds for which a worker threads goes into sleep before starting another iteration 
used for backoff in case of no launched job or error

hive.compactor.worker.threads deprecated. Use 

metastore.compactor.worker.threads instead.

Default: 0

Value required for transactions: > 0 on at least one instance of the Thrift metastore service

Metastore

How many compactor worker threads to run on this metastore instance.2

hive.compactor.worker.timeout

Default: 86400s

Metastore

Time in seconds after which a compaction job will be declared failed and the compaction re-queued.

hive.compactor.cleaner.run.intervalDefault: 5000msMetastoreTime in milliseconds between runs of the cleaner thread. (Hive 0.14.0 and later.)

hive.compactor.check.interval

Default: 300s

Metastore

Time in seconds between checks to see if any tables or partitions need to be compacted.3

hive.compactor.delta.num.threshold

Default: 10

Metastore

Number of delta directories in a table or partition that will trigger a minor compaction.

hive.compactor.delta.pct.threshold

Default: 0.1

Metastore

Percentage (fractional) size of the delta files relative to the base that will trigger a major compaction. 1 = 100%, so the default 0.1 = 10%.

hive.compactor.abortedtxn.threshold

Default: 1000

Metastore

Number of aborted transactions involving a given table or partition that will trigger a major compaction.

hive.compactor.aborted.txn.time.thresholdDefault: 12hMetastoreAge of table/partition's oldest aborted transaction when compaction will be triggered. Default time unit is: hours. Set to a negative number to disable.

hive.compactor.max.num.delta

Default: 500MetastoreMaximum number of delta files that the compactor will attempt to handle in a single job (as of Hive 1.3.0).4

hive.compactor.job.queue

Default: "" (empty string) Metastore Used to specify name of Hadoop queue to which Compaction jobs will be submitted. Set to empty string to let Hadoop choose the queue (as of Hive 1.3.0).

hive.txn.xlock.iow

Default: trueHiveServer2

Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire Exclusive locks for

transactional tables. This ensures that inserts (w/o overwrite) running concurrently

are not hidden by the INSERT OVERWRITE.

hive.txn.xlock.write


Default: trueHiveServer2

Manages concurrency levels for ACID resources. Provides better level of query parallelism by enabling 
shared writes and write-write conflict resolution at the commit step.
- If true - exclusive writes are used:
  - INSERT OVERWRITE acquires EXCLUSIVE locks
  - UPDATE/DELETE acquire EXCL_WRITE locks
  - INSERT acquires SHARED_READ locks
- If false - shared writes, transaction is aborted in case of conflicting changes:
  - INSERT OVERWRITE acquires EXCL_WRITE locks
  - INSERT/UPDATE/DELETE acquire SHARED_READ locks

Compaction History

hive.compactor.history.retention.succeeded deprecated. Use 

metastore.compactor.history.retention.succeeded instead
Default: 3MetastoreNumber of successful compaction entries to retain in history (per partition).

hive.compactor.history.retention.failed deprecated. Use 

metastore.compactor.history.retention.failed instead.
Default: 3MetastoreNumber of failed compaction entries to retain in history (per partition).

hive.compactor.history.retention.attempted deprecated. Use 

metastore.compactor.history.retention.did.not.initiate instead.
Default: 2MetastoreNumber of attempted compaction entries to retain in history (per partition).

hive.compactor.initiator.failed.compacts.threshold deprecated. Use 

metastore.compactor.initiator.failed.compacts.threshold instead.
Default: 2MetastoreNumber of of consecutive failed compactions for a given partition after which the Initiator will stop attempting to schedule compactions automatically. It is still possible to use ALTER TABLE to initiate compaction. Once a manually initiated compaction succeeds auto initiated compactions will resume. Note that this must be less than hive.compactor.history.retention.failed.

hive.compactor.history.reaper.interval deprecated. 

metastore.acid.housekeeper.interval handles it.
Default: 2mMetastoreControls how often the process to purge historical record of compactions runs.

...