Proposers

• pingle wang
• Taher Koitawala

Approvers

• Unknown User (vinoth)  : REQUESTED_INFO
•  Unknown User (nagarwal)

Status

Current state:  "Accepted"

Discussion thread: here

JIRA: here

Released: N/A

Abstract

All Hudi datasets are preserved as Parquet on DFS. Since ORC is widely accepted and supported too. Goal is to  provide ORC as a serving layer to back Hudi datasets so that users can have more control over the columnar format they wish to use.

Background

Hoodie uses parquet as its default storage format for Copy on Write and Merge On Read operations where users are forced to store and query data in parquet. Introduce Orc as an underlying storage format for Hoodie to expose ORC Read Optimized views.

Implementation

• Implement a HoodieOrcWriter just like HoodieParquetWriter to write data in ORC:

• Implementation done and capture in HUDI-57 (Effort needs to be tested)

• Change or create new implementation of HoodieInputFormat to support ORC writing (As of now it strictly writes Parquet)

• <Implementation and Design approach will be attached here>

• Rework HoodieBloomIndex to work with ORC:

• <Implementation and Design approach will be attached here>

Rollout/Adoption Plan

• <What impact (if any) will there be on existing users?>
• <If we are changing behavior how will we phase out the older behavior?>
• <If we need special migration tools, describe them here.>
• <When will we remove the existing behavior?>

Test Plan

<Describe in few sentences how the HIP will be tested. How will we know that the implementation works as expected? How will we know nothing broke?>