...
- Infers the schema of the Hive table from the Avro schema.
- Reads all Avro files within a table against a specified schema, taking advantage of Avro's backwards compatibility abilities
- Supports arbitrarily nested schemas.
- Translates all Avro data types into equivalent Hive types. Most types map exactly, but some Avro types don't exist in Hive and are automatically converted by the AvroSerde.
- Understands compressed Avro files.
Wiki Markup Transparently converts the Avro idiom of handling nullable types as Union\[T, null\] into just T and returns null when appropriate.
- Writes any Hive table to Avro files.
- Has worked reliably against our most convoluted Avro schemas in our ETL process.
For general information about SerDes, see Hive SerDe in the Developer Guide. Also see SerDe for details about input and output processing.
Requirements
The AvroSerde has been built and tested against Hive 0.9.1 and Avro 1.5.
...
Avro type | Becomes Hive type | Note | ||
---|---|---|---|---|
null | void | |||
boolean | boolean | |||
int | int | |||
long | bigint | |||
float | float | |||
double | double | |||
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="410ed3ed7dab9e53-02867ceb-4d554933-abf4abde-d8384cff29a353fd516deb8d"><ac:plain-text-body><![CDATA[ | bytes | binary | Bytes are converted to Array[smallint] prior to Hive 0.12.0 | ]]></ac:plain-text-body></ac:structured-macro> |
string | string | |||
record | struct | |||
map | map | |||
list | array | |||
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="76ae7795c443e3d0-626c82b1-4fc84c01-8717958a-7adf53d6fe0e935d9d2f4085"><ac:plain-text-body><![CDATA[ | union | union | Unions of [T, null] transparently convert to nullable T, other types translate directly to Hive's unions of those types. However, unions were introduced in Hive 7 and are not currently able to be used in where/group-by statements. They are essentially look-at-only. Because the AvroSerde transparently converts [T,null], to nullable T, this limitation only applies to unions of multiple types or unions not of a single type and null. | ]]></ac:plain-text-body></ac:structured-macro> |
enum | string | Hive has no concept of enums | ||
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="bb89e2a2a5b66f59-8096dc43-4f644ead-bf1a89fd-9d33b48d2333185191d2c6ef"><ac:plain-text-body><![CDATA[ | fixed | binary | Fixeds are converted to Array[smallint] prior to Hive 0.12.0 | ]]></ac:plain-text-body></ac:structured-macro> |
...