THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Apache HCatalog's behaviour can be modified through use of a few configuration parameters specified in jobs submitted to it. This document details all the various knobs that users have available to them, and what they accomplish.
Setup
The properties described in this page are meant to be job-level properties set on HCatalog through the jobConf passed into it. This means that this page is relevant for Pig users of HCatLoader/HCatStorer, or MapReduce users of HCatInputFormat/HCatOutputFormat. For a MapReduce user of HCatalog, these must be present as key-values in the Configuration (JobConf/Job/JobContext) used to instantiate HCatOutputFormat or HCatInputFormat. For Pig users of HCatStorer, these parameters are set using the Pig "set" command before instantiating an HCatLoader/HCatStorer.
Storage Directives
| Property | Default | Description |
|---|---|---|
| hcat.pig.storer.external.location | Not set | An override to specify where HCatStorer will write to, defined from Pig jobs, either directly by user, or by using org.apache.hive.hcatalog.pig.HCatStorerWrapper. HCatalog will write to this specified directory, rather than writing to the table/partition directory specified/calculatable by the metadata. This will be used in lieu of the table directory if this is a table-level write (unpartitioned table write) or in lieu of the partition directory if this is a partition-level write. This parameter is used only for non-dynamic-partitioning jobs which have multiple write destinations. |
| hcat.dynamic.partitioning.custom.pattern | Not set | For dynamic partitioning jobs, simply specifying a custom directory is not good enough, since it writes to multiple destinations, and thus, instead of a directory specification, it requires a pattern specification. That's where this parameter comes in. For example, if one had a table that was partitioned by keys country and state, with a root directory location of /apps/hive/warehouse/geo/ , then a dynamic partition write into it that writes partitions (country=US,state=CA) & (country=IN,state=KA) would create two directories: /apps/hive/warehouse/geo/country=US/state=CA/ and /apps/hive/warehouse/geo/country=IN/state=KA/ . If we wanted a different patterned location, and specified hcat.dynamic.partitioning.custom.pattern="/ext/geo/${country}-${state}", it would create the following two partition dirs: /ext/geo/US-CA and /ext/geo/IN-KA . Thus, it allows us to specify a custom dir location pattern for all the writes, and will interpolate each variable it sees when attempting to create a destination location for the partitions. See Dynamic Partitioning: External Tables for another example.
|
hcat.append.limit (Hive 0.15.0 and later) | Not set | hcat.append.limit allows an HCatalog user to specify a custom append limit. By default, while appending to an existing directory HCatalog will attempt to avoid naming clashes and try to append _a_NNN, where NNN is a number, to the desired filename to avoid clashes. However, by default, it only tries for NNN from 0 to 999 before giving up. This can cause an issue for some tables with an extraordinarily large number of files. Ideally, this should be fixed by the user changing their usage pattern and doing some manner of compaction, but in the meanwhile, until they can, setting this parameter can be used to bump that limit. (Added in Hive 0.15.0 with HIVE-9381.) |
Cache Behaviour Directives
HCatalog maintains a cache of HiveClients to talk to the metastore, managing a cache of 1 metastore client per thread, defaulting to an expiry of 120 seconds. For people that wish to modify the behaviour of this cache, a few parameters are provided:
| Property | Default | Description |
|---|---|---|
| hcatalog.hive.client.cache.expiry.time | 120 | Allows users to override the expiry time specified -- this is an int, and specifies number of seconds. |
| hcatalog.hive.client.cache.disabled | false | Allows people to disable the cache altogether if they wish to. This is useful in highly multithreaded usecases. |
Input Split Generation Behaviour
| Property | Default | Description |
|---|---|---|
| hcat.desired.partition.num.splits | not set | This is a hint/guidance that can be provided to HCatalog to pass on to underlying InputFormats, to produce a "desired" number of splits per partition. This is useful when we have a few large files and we want to increase parallelism by increasing the number of splits generated. It is not yet so useful in cases where we would want to reduce the number of splits for a large number of files. It is not at all useful, also, in cases where there are a large number of partitions that this job will read. Also note that this is merely an optimization hint, and it is not guaranteed that the underlying layer will be capable of using this optimization. Also, mapreduce parameters mapred.min.split.size and mapred.max.split.size can be used in conjunction with this parameter to tweak/optimize jobs. |
Data Promotion Behaviour
In some cases where a user of HCatalog (such as some older versions of Pig) does not support all the datatypes supported by Hive, there are a few configuration parameters provided to handle data promotions/conversions to allow them to read data through HCatalog. On the write side, it is expected that the user pass in valid HCatRecords with data correctly.
| Property | Default | Description |
|---|---|---|
| hcat.data.convert.boolean.to.integer | false | promotes boolean to int on read from HCatalog |
| hcat.data.tiny.small.int.promotion | false | promotes tinyint/smallint to int on read from HCatalog |
HCatRecordReader Error Tolerance Behaviour
While reading, it is understandable that data might contain errors, but we may not want to completely abort a task due to a couple of errors. These parameters configure how many errors we can accept before we fail the task.
| Property | Default | Description |
|---|---|---|
| hcat.input.bad.record.threshold | 0.0001f | A float parameter, defaults to 0.0001f, which means we can deal with 1 error every 10,000 rows, and still not error out. Any greater, and we will. |
| hcat.input.bad.record.min | 2 | An int parameter, defaults to 2, which is the minimum number of bad records we encounter before applying hcat.input.bad.record.threshold parameter, this is to prevent an initial/early bad record from resulting in a task abort because the ratio of errors it got was too high. |
Navigation Links
Previous: Installation from Tarball
Next: Load and Store Interfaces
General: HCatalog Manual – WebHCat Manual – Hive Wiki Home – Hive Project Site