Page History

...

It is also important to note that all partitions created during a single run are part of one transaction; therefore if any part of the process fails, none of the partitions will be added to the table.

External Tables

Starting in HCatalog 0.5, dynamic partitioning on external tables was broken (HCATALOG-500). This issue was fixed in Hive 0.12.0 by creating dynamic partitions of external tables in locations based on metadata rather than user specifications (HIVE-5011). In a future release, users will be . Starting in Hive 0.13.0, users are able to customize the locations (see by specifying a path pattern in the job configuration property hcat.dynamic.partitioning.custom.pattern (HIVE-6109).Static  Static partitions for external tables can have user-specified locations in all Hive releases.

For example, in Hive 0.12.0 if a table named user_logs is partitioned by (year, month, day, hour, minute, country) and stored at external location "hdfs://hcat/data/user_logs", then the locations of its dynamic partitions have the standard Hive format which includes keys as well as values, such as:

• hdfs://hcat/data/user_logs/year=2013/month=12/hour=06/minute=10/country=US

In Hive 0.13.0 and later, hcat.dynamic.partitioning.custom.pattern can be configured to a custom path pattern. For example, the pattern "${year}/${month}/${day}/${hour}/${minute}/${country}" omits keys from the path:

• hdfs://hcat/data/user_logs/2013/12/06/10/US

Each dynamic partition column must be present in the custom location path in the format ${column_name}, and the custom location path must consist of all dynamic partition columns. Other valid custom path strings include:

•  

  data/${year}/${month}/${day}/${country}

• ${year}­‐${month}­‐${day}/country=${country}

• output/yr=${year}/mon=${month}/day=${day}/geo=${country}

See HCatalog Configuration Properties for another example. Also see the PDF attachment to HIVE-6019 for details of the implementation.

Hive Dynamic Partitions

Information about Hive dynamic partitions is available here:

...