You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 14 Next »

Proposers

Approvers

Status

Current state:  


Current State

UNDER DISCUSSION

(tick)

IN PROGRESS


ABANDONED


COMPLETED


INACTIVE


Discussion thread: here

JIRA: here

Released: N/A

Abstract

All Hudi datasets are preserved as Parquet on DFS. Since ORC is widely accepted and supported too. Goal is to  provide ORC as a serving layer to back Hudi datasets so that users can have more control over the columnar format they wish to use.

Background

Hoodie uses parquet as its default storage format for Copy on Write and Merge On Read operations where users are forced to store and query data in parquet. Introduce Orc as an underlying storage format for Hoodie to expose ORC Read Optimized views.

Implementation

Schema Compatibility

Hudi use avro schema for input record, and parquet writes record by using avro schema. But orc cann't support avro directly, need to do some wrapper work.

HoodieOrcWriter

Key Steps

  • Implement a HoodieOrcWriter just like HoodieParquetWriter to write data in ORC:
  • Implementation done and capture in HUDI-57 (Effort needs to be tested)
  • Change or create new implementation of HoodieInputFormat to support ORC writing (As of now it strictly writes Parquet)
  • <Implementation and Design approach will be attached here>
  • Rework HoodieBloomIndex to work with ORC:
  • <Implementation and Design approach will be attached here>

Rollout/Adoption Plan

  • <What impact (if any) will there be on existing users?>
  • <If we are changing behavior how will we phase out the older behavior?>
  • <If we need special migration tools, describe them here.>
  • <When will we remove the existing behavior?>

Test Plan

<Describe in few sentences how the HIP will be tested. How will we know that the implementation works as expected? How will we know nothing broke?>

TODO (Dataframe)

Hudi use avro schema for input record and store the schema infomation to commit meta, and parquet writer will use the avro schema.

But the schema of orc incompatible with avro schema, so we should store StructType( spark provides ) infomation to commit meta.

  • No labels