...

Hoodie uses parquet as its default storage format for Copy on Write and Merge On Read operations where users are forced to store and query data in parquet. Introduce Orc as an underlying storage format for Hoodie to expose ORC Read Optimized views.

Implementation

SchemaSchema 

Hudi use avro schema for input record, and parquet writes record by using avro schema. But orc cann't support avro directlynatively, need to do some transform work.

HoodieOrcWriter

Image Removed

Key Steps

Implementation Steps

• Add storage type option(e.g "hoodie.table.storage.type=ORC") to hoodie.properties.

• Impement a HoodieOrcWriter just like HoodieParquetWriter to write data in ORC.
• Impement a OrcReaderIterator just like ParquetReaderIterator to write data in ORC.
• Change or create new implementation of HoodieInputFormat to support ORC writing (As of now it strictly writes Parquet)
• Rework HoodieBloomIndex to work with ORC.

Rollout/Adoption Plan

• What impact (if any) will there be on existing users?

                 None. Only one columnar storage format can be used for each hudi data set, controlled by "hoodie.table.storage.type" option.

• If we are changing behavior how will we phase out the older behavior?

                Parquet is the default,  

• If we need special migration tools, describe them here.
  • NA
• When will we remove the existing behavior?
  • Is not required.

Test Plan

<Describe in few sentences how the HIP will be tested. How will we know that the implementation works as expected? How will we know nothing broke?>

...