...

Approvers

Status

Current state: 

	Current State
Status title Under Discussion
Status colour Yellow title In Progress
Status colour Red title ABANDONED
Status colour Green title Completed
Status colour Blue title INactive

Discussion thread: here

JIRA: Hudi-841

Released: <Hudi Version>

Abstract

Now we want to use Aliyun DataLake analytics  service to analytics hudi dataset.  So  we need to sync the meta to  Aliyun DataLake analytics , but the hudi-hive-sync just support hive.  Hudi as open datalake  engine, will support more meta service and analytics engine.

So, I am proposing to support for abstract the common hudi-sync. Then  other service like aws alue、aliyun datalake analytics can implement.

Background

Currently Hudi only supports sync dataset metadata to Hive through hive jdbc and IMetaStoreClient. When you need to sync to other frameworks, such as aws glue, aliyun DataLake analytics, etc.

...

• Unit tests
• Integration tests
• Test on the cluster for a larger dataset.