...

• Unknown User (vinoth)  
•  Unknown User (nagarwal) 

Status

Current state:  

	Current State
Status title Under Discussion
Status colour Yellow title In Progress
Status colour Red title ABANDONED
Status colour Green title Completed
Status colour Blue title INactive

Discussion thread: here

JIRA: here

Released: N/A

Abstract

All Hudi datasets are preserved as Parquet on DFS. Since ORC is widely accepted and supported too. Goal is to  provide ORC as a serving layer to back Hudi datasets so that users can have more control over the columnar format they wish to use.

Background

Hoodie uses parquet as its default storage format for Copy on Write and Merge On Read operations where users are forced to store and query data in parquet. Introduce Orc as an underlying storage format for Hoodie to expose ORC Read Optimized views.

Implementation

Schema 

Hudi use avro schema for input record, and parquet writes record by using avro schema. But orc cann't support avro natively, need to do some transform job.

Implementation

...

 Steps

• Add storage type option(e.g "hoodie.table.storage.type=ORC") to hoodie.properties.

Implement a HoodieOrcWriter just like
• Impement a HoodieOrcWriter just like 
• HoodieParquetWriter to write data in ORC.
• Impement a OrcReaderIterator just like ParquetReaderIterator to read data in ORC.:Implementation done and capture in HUDI-57 (Effort needs to be tested)
• Change or create new implementation of HoodieInputFormat to support ORC writing (As of now it strictly writes Parquet)

• <Implementation and Design approach will be attached here>

• Rework HoodieBloomIndex to work with ORC:<Implementation and Design approach will be attached here>.

Rollout/Adoption Plan

• <What What impact (if any) will there be on existing users?>

                 None. Only one columnar storage format can be used for each hudi data set, controlled by "hoodie.table.storage.type" option.

• If <If we are changing behavior how will we phase out the older behavior?>

                Parquet is the default,  

• If <If we need special migration tools, describe them here.>
  • NA
• <When When will we remove the existing behavior?>
  • Is not required.

Test Plan

<Describe in few sentences how the HIP will be tested. How will we know that the implementation works as expected? How will we know nothing broke?>

May be rework in the future (Dataframe)

Hudi use avro schema for input record and store the schema infomation to commit meta, and parquet writer will use the avro schema.

But the schema of orc incompatible with avro schema, so we should store StructType( spark provides ) infomation to commit meta.

Image Added