The Div macro wraps content in a div tag with optional class and styles. This allows us to use macros such as the Style macro below.

Proposers

• lamber-ken

Approvers

• Unknown User (vinoth)  
•  Unknown User (nagarwal) 

Status

Current state:  

	Current State

Discussion thread: here

JIRA: here

Released: N/A

Abstract

All Hudi datasets are preserved as Parquet on DFS. Since ORC is widely accepted and supported too. Goal is to  provide ORC as a serving layer to back Hudi datasets so that users can have more control over the columnar format they wish to use.

Hoodie uses parquet as its default storage format for Copy on Write and Merge On Read operations where users are forced to store and query data in parquet. Introduce Orc as an underlying storage format for Hoodie to expose ORC Read Optimized views.

Implementation

Schema 

Hudi use avro schema for input record, and parquet writes record by using avro schema. But orc cann't support avro natively, need to do some transform job.

Implementation Steps

• Add storage type option(e.g "hoodie.table.storage.type=ORC") to hoodie.properties.

• Impement a HoodieOrcWriter just like HoodieParquetWriter to write data in ORC.
• Impement a OrcReaderIterator just like ParquetReaderIterator to read data in ORC.
• Change or create new implementation of HoodieInputFormat to support ORC writing (As of now it strictly writes Parquet)
• Rework HoodieBloomIndex to work with ORC.

Rollout/Adoption Plan

• What impact (if any) will there be on existing users?

                 None. Only one columnar storage format can be used for each hudi data set, controlled by "hoodie.table.storage.type" option.

• If we are changing behavior how will we phase out the older behavior?

                Parquet is the default,  

• If we need special migration tools, describe them here.
  • NA
• When will we remove the existing behavior?
  • Is not required.

Test Plan

<Describe in few sentences how the HIP will be tested. How will we know that the implementation works as expected? How will we know nothing broke?>

May be rework in the future (Dataframe)

Hudi use avro schema for input record and store the schema infomation to commit meta, and parquet writer will use the avro schema.

But the schema of orc incompatible with avro schema, so we should store StructType( spark provides ) infomation to commit meta.