...
Property | Default | Description |
---|---|---|
hcat.pig.storer.external.location | Not set | An override to specify where HCatStorer will write to, defined from pig jobs, either directly by user, or by using org.apache.hive.hcatalog.pig.HCatStorerWrapper. HCat will write to this specified directory, rather than writing to the table/partition directory specified/calculatable by the metadata. This will be used in lieu of the table directory if this is a table-level write (unpartitioned table write) or in lieu of the partition directory if this is a partition-level write. This parameter is used only for non-dynamic-partitioning jobs which have multiple write destinations. |
hcat.dynamic.partitioning.custom.pattern | Not set | For dynamic partitioning jobs, simply specifying a custom directory is not good enough, since it writes to multiple destinations, and thus, instead of a directory specification, it requires a pattern specification. That's where this parameter comes in. For example, if one had a table that was partitioned by keys country and state, with a root directory location of /apps/hive/warehouse/geo/ , then a dynamic partition write into it that writes partitions (country=US,state=CA) & (country=IN,state=KA) would create two directories: /apps/hive/warehouse/geo/country=US/state=CA/ and /apps/hive/warehouse/geo/country=IN/state=KA/ . If we wanted a different patterned location, and specified hcat.dynamic.partitioning.custom.pattern="/ext/geo/${country}-${state}", it would create the following two partition dirs: /ext/geo/US-CA and /ext/geo/IN-KA . Thus, it allows us to specify a custom dir location pattern for all the writes, and will interpolate each variable it sees when attempting to create a destination location for the partitions. See Dynamic Partitioning: External Tables for another example.
|
hcat.append.limit | Not set | hcat.append.limit allows a hcat user to specify a custom append limit. By default, while appending to an existing directory, hcat will attempt to avoid naming clashes and try to append _a_NNN where NNN is a number to the desired filename to avoid clashes. However, by default, it only tries for NNN from 0 to 999 before giving up. This can cause an issue for some tables with an extraordinarily large number of files. Ideally, this should be fixed by the user changing their usage pattern and doing some manner of compaction, but in the meanwhile, until they can, setting this parametercan be used to bump that limit. |
Cache behaviour directives:
...