...
Table of Contents |
---|
Proposers
Approvers
- Shaofeng Li
- Vinoth Chandar
- Gary Li Approved
Status
Current state:
Current State | |||||||||
---|---|---|---|---|---|---|---|---|---|
| |||||||||
| |||||||||
| |||||||||
| |||||||||
|
Discussion thread: here
JIRA: Hudi-841
Released: <Hudi Version>
Abstract
Now we want to use Aliyun DataLake analytics service to analytics hudi dataset. So we need to sync the meta to Aliyun DataLake analytics , but the hudi-hive-sync just support hive. Hudi as open datalake engine, will support more meta service and analytics engine.
So, I am proposing to support for abstract the common hudi-sync. Then other service like aws
...
glue、aliyun datalake analytics can implement.
Background
Currently Hudi only supports sync dataset metadata to Hive through hive jdbc and IMetaStoreClient. When you need to sync to other frameworks, such as aws glue, aliyun DataLake analytics, etc.
...
The organizational form of Option1 will be clearer, and user dependence will be clearer; but an additional module will be added.
Option2 has no changes to the current module organization, but the general interface implementation is placed in the hudi-hive-sync module. The user-defined implementation depends on this module, which is a bit semantically strange.
I personally prefer Option2Option1.
2.2 Code (class) structure
Among them, AbstractHoodieSyncClient is an abstract synchronization client, the default implementation is HoodieHiveClient; users can customize to implement Client.
The abstract methods in AbstractHoodieSyncClient are below
public abstract void createTable(String tableName, MessageType storageSchema,
String inputFormatClass, String outputFormatClass, String serdeClass);
public abstract boolean doesTableExist(String tableName);
public abstract Option<String> getLastCommitTimeSynced(String tableName);
public abstract void updateLastCommitTimeSynced(String tableName);
public abstract void addPartitionsToTable(String tableName, List<String> partitionsToAdd);
public abstract void updatePartitionsToTable(String tableName, List<String> changedPartitions);
If option1 is used, then AbstractHoodieSyncClient will be put into the hudi-common-sync module; if option2 is used, then AbstractHoodieSyncClient will be put into hudi-hive-sync.
AbstractSyncTool is an abstract synchronization tool. All synchronization tools must inherit this class, and the default implementation is HiveSyncTool.
...
- Unit tests
- Integration tests
- Test on the cluster for a larger dataset.
...