Versions Compared

Old Version 456

changes.mady.by.user Lefty Leverenz

Saved on

compared with

New Version 457

changes.mady.by.user Andrew Sears

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: HIVE-13029 hive.llap.io.allocator properties

...

Info
titleVersion information

As of Hive 0.14.0 (HIVE-7211), a configuration name that starts with "hive." is regarded as a Hive system property. With the hive.conf.validation option true (default), any attempts to set a configuration property that starts with "hive." which is not registered to the Hive system will throw an exception.

Query and DDL Execution

hive.execution.engine

...

  • Default Value: 1,000,000,000 prior to Hive 0.14.0; 256 MB (256,000,000) in Hive 0.14.0 and later
  • Added In: Hive 0.2.0; default changed in 0.14.0 with HIVE-7158 (and HIVE-7917)

...

  • Default Value: (empty)
  • Added In: Hive 0.14.0 with HIVE-7553

The locations of the plugin jars, which can be comma-separated folders or jars. They can be renewed (added, removed, or updated) by executing the Beeline reload command without having to restart HiveServer2. These jars can be used just like the auxiliary classes in hive.aux.jars.path for creating UDFs or SerDes.

hive.exec.scratchdir
  • Default Value: /tmp/${user.name} in Hive 0.2.0 through 0.8.0; /tmp/hive-${user.name} in Hive 0.8.1 through 0.14.0; or /tmp/hive in Hive 0.14.0 and later
  • Added In: Hive 0.2.0; default changed in 0.8.1 and in 0.14.0 with HIVE-6847 and HIVE-8143

...

Hive 0.14.0 and later:  HDFS root scratch directory for Hive jobs, which gets created with write all (733) permission. For each connecting user, an HDFS scratch directory ${hive.exec.scratchdir}/<username> is created with ${hive.scratch.dir.permission}.

Also see hive.start.cleanup.scratchdir and hive.scratchdir.lock.

hive.scratch.dir.permission

...

  • Default Value: 0.55
  • Added In: Hive 0.7.0 with HIVE-1830

This number means how much memory the local task can take to hold the key/value into an in-memory hash table when this map join is followed by a group by. If the local task's memory usage is more than this number, the local task will abort by itself. It means the data of the small table is too large to be held in memory.

hive.mapjoin.check.memory.rows

...

  • Default Value: 100000
  • Added In: Hive 0.7.0 with HIVE-1642

Initial capacity of mapjoin hashtable if statistics are absent, or if hive.hashtable.key.count.adjustment is set to 0.

hive.hashtable.key.count.adjustment
  • Default Value: 1.0
  • Added In: Hive 0.14.0 with HIVE-7616

Adjustment to mapjoin hashtable size derived from table and column statistics; the estimate of the number of keys is divided by this value. If the value is 0, statistics are not used and hive.hashtable.initialCapacity is used instead.

hive.hashtable.loadfactor

...

In the process of Mapjoin, the key/value will be held in the hashtable. This value means the load factor for the in-memory hashtable.

hive.debug.localtask
  • Default Value: false
  • Added In: Hive 0.7.0 with HIVE-1642
hive.outerjoin.supports.filters
hive.optimize.skewjoin
  • Default Value: false
  • Added In: Hive 0.6.0

...

Whether to execute jobs in parallel.  Applies to MapReduce jobs that can run in parallel, for example jobs processing different source tables before a join.  As of Hive 0.14, also applies to move tasks that can run in parallel, for example moving files to insert targets during multi-insert.

hive.exec.parallel.thread.number

...

For conditional joins, if input stream from a small alias can be directly applied to the join operator without filtering or projection, the alias need not be pre-staged in the distributed cache via a mapred local task. Currently, this is not working with vectorization or Tez execution engine.

hive.udtf.auto.progress
  • Default Value: false
  • Added In: Hive 0.5.0

...

  • Default Value:
    • Hive 0.x and 1.x: false
    • Hive 2.0 and later: true (HIVE-12325)
  • Added In: Hive 0.10.0 with HIVE-3432

If the bucketing/sorting properties of the table exactly match the grouping key, whether to perform the group by in the mapper by using BucketizedHiveInputFormat. The only downside to this is that it limits the number of mappers to the number of files.

hive.map.groupby.sorted.testmode
  • Default Value: false
  • Added In: Hive 0.11.0 with HIVE-4281
  • Removed In: Hive 2.0.0 with HIVE-12325

If the bucketing/sorting properties of the table exactly match the grouping key, whether to perform the group by in the mapper by using BucketizedHiveInputFormat. If the test mode is set, the plan is not converted, but a query property is set to denote the same. (This configuration property was removed in release 2.0.0.)

hive.groupby.orderby.position.alias

...

  • Default Value: -1 in Hive 0.13.0 and 0.13.1, 1073741824 (1 GB) in Hive 0.14.0 and later 
  • Added In: Hive 0.13.0 with HIVE-3990; default changed in Hive 0.14.0 with HIVE-7397

...

In older Hive versions (0.10 and earlier) no distinction was made between partition columns or non-partition columns while displaying columns in DESCRIBE TABLE. From version 0.12 onwards, they are displayed separately. This flag will let you get the old behavior, if desired. See test-case in patch for HIVE-6689.

hive.limit.query.max.table.partition

...

Uses sampling on order-by clause for parallel execution.

hive.optimize.sampling.orderby.number
  • Default Value: 1000
  • Added In: Hive 0.12.0 with HIVE-1402

...

  • Default Value: true
  • Added In: Hive 1.2.0 with HIVE-6617

Whether to enable support for SQL2011 reserved keywords. When enabled, will support (part of) SQL2011 reserved keywords.

hive.explain.user
  • Default Value: false
  • Added In: Hive 1.2.0 with HIVE-9780

Whether to show explain result at user level. When enabled, will log EXPLAIN output for the query at user level.

hive.typecheck.on.insert
  • Default Value: true
  • Added In: Hive 0.12.0 with HIVE-5297 for insert partition
  • Extended In: Hive 1.2 with HIVE-10307 for alter, describe partition, etc.

...

  • Default Value: default

  • Added In: Hive 1.1.0 with HIVE-7313

Expects one of [memory, ssd, default].

Define the storage policy for temporary tables. Choices between memory, ssd and default. See HDFS Storage Types and Storage Policies.

...

When applicable, this optimization rewrites distinct aggregates from a single-stage to multi-stage aggregation. This may not be optimal in all cases. Ideally, whether to trigger it or not should be a cost-based decision. Until Hive formalizes the cost model for this, this is config driven.

hive.optimize.point.lookup

  • Default Value: true

  • Added In: Hive 2.0.0 with HIVE-11461

...

  • Default Value: true

  • Added In: Hive 2.1.0 with HIVE-13027

Whether to enable Log4j2's asynchronous logging. Asynchronous logging can give significant performance improvement as logging will be handled in a separate thread that uses the LMAX disruptor queue for buffering log messages.

Refer to https://logging.apache.org/log4j/2.x/manual/async.html for benefits and drawbacks.

hive.msck.repair.batch.size

...

hive.metastore.schema.verification
  • Default Value: false 
  • Added In: Hive 0.12.0 with HIVE-3764

...

hive.direct.sql.max.query.length 
  • Default Value: 100
  • Added In: Hive 1.3.0 and 2.1.0 (but not 2.0.x) with HIVE-12349

...

Number of threads to use to read file metadata in background to cache it.

hive.metastore.initial.metadata.count.enabled 
  • Default Value: true
  • Added In: Hive 2.1.0 with HIVE-12628

...

This property can also be used to find the users if a custom-configured LDAP query returns a group instead of a user (as of Hive 2.1.1). For details, see Support for Groups in Custom LDAP Query.

hive.server2.authentication.ldap.groupClassKey

...

hive.server2.idle.session.timeout
  • Default Value:
    • Hive 0.x, 1.0.x, 1.1.x, 1.2.0: 0ms
    • Hive 1.2.1+, 1.3+, 2.x+: 7d (HIVE-9842) 
  • Added In: Hive 0.14.0 with HIVE-5799

...

  • Default Value: false
  • Added In: Hive 2.0.0 with HIVE-12485 

SSL certificate keystore password for HiveServer2 WebUI.

...

hive.server2.webui.spnego.principal
  • Default Value: HTTP/_HOST@EXAMPLE.COM
  • Added In: Hive 2.0.0 with HIVE-12485

...

Whether ORC low-level cache should use direct allocation.

hive.llap.io.allocator.nmap
  • Default value: false
  • Added In: Hive 2.1.0 with HIVE-13029

Whether ORC low-level cache should use memory mapped allocation (direct I/O)

hive.llap.io.allocator.nmap.path
  • Default value: /tmp
  • Added In: Hive 2.1.0 with HIVE-13029

The directory location for mapping NVDIMM/NVMe flash storage into the ORC low-level cache.

LLAP CBO

hive.llap.auto.allow.uber

...

hive.conf.hidden.list
  • Default Value: javax.jdo.option.ConnectionPassword,hive.server2.keystore.password
  • Added In: Hive 1.2.2 with HIVE-9013

...

Some parameters are added automatically when they match one of the regex specifications for the white list in HiveConf.java (for example, hive.log.trace.id in Hive 2.0.0 – see HIVE-12419).

Note that the hive.conf.restricted.list checks are still enforced after the white list check.

...

hive.lockmgr.zookeeper.default.partition.name
  • Default Value: __HIVE_DEFAULT_ZOOKEEPER_PARTITION__
  • Added In: Hive 0.7.0 with HIVE-1293

...