...

Proposers

• liwei

Approvers

• Shaofeng Li
• Vinoth Chandar
• Gary Li Approved

Status

Current state: 

	Current State
Status title Under Discussion
Status colour Yellow title In Progress
Status colour Red title ABANDONED
Status colour Green title Completed
Status colour Blue title INactive

Discussion thread: here

JIRA: Hudi-841

Released: <Hudi Version>

Abstract

Now we want to use Aliyun DataLake analytics  service to analytics hudi dataset.  So  we need to sync the meta to  Aliyun DataLake analytics , but the hudi-hive-sync just support hive.  Hudi as open datalake  engine, will support more meta service and analytics engine.

So, I am proposing to support for abstract the common hudi-sync. Then  other service like aws

...

glue、aliyun datalake analytics can implement.

Background

Currently Hudi only supports sync dataset metadata to Hive through hive jdbc and IMetaStoreClient. When you need to sync to other frameworks, such as aws glue, aliyun DataLake analytics, etc.

...

The organizational form of Option1 will be clearer, and user dependence will be clearer; but an additional module will be added.
Option2 has no changes to the current module organization, but the general interface implementation is placed in the hudi-hive-sync module. The user-defined implementation depends on this module, which is a bit semantically strange.

I personally prefer Option2Option1.

2.2 Code (class) structure

Image AddedImage Removed

Among them, AbstractHoodieSyncClient is an abstract synchronization client, the default implementation is HoodieHiveClient; users can customize to implement Client.
The abstract methods in AbstractHoodieSyncClient are below

public abstract void createTable(String tableName, MessageType storageSchema,
                                 String inputFormatClass, String outputFormatClass, String serdeClass);

public abstract boolean doesTableExist(String tableName);

public abstract Option<String> getLastCommitTimeSynced(String tableName);

public abstract void updateLastCommitTimeSynced(String tableName);

public abstract void addPartitionsToTable(String tableName, List<String> partitionsToAdd);

public abstract void updatePartitionsToTable(String tableName, List<String> changedPartitions);

If option1 is used, then AbstractHoodieSyncClient will be put into the hudi-common-sync module; if option2 is used, then AbstractHoodieSyncClient will be put into hudi-hive-sync.
Image Removed
Image Added

AbstractSyncTool is an abstract synchronization tool. All synchronization tools must inherit this class, and the default implementation is HiveSyncTool.

...

• Unit tests
• Integration tests
• Test on the cluster for a larger dataset. 

...