Proposers
Approvers
Status
Current state:
Current State | |
---|---|
UNDER DISCUSSION | |
IN PROGRESS | |
ABANDONED | |
COMPLETED | |
INACTIVE |
Discussion thread: here
JIRA: Hudi-841
Released: <Hudi Version>
Abstract
Now we want to use Aliyun DataLake analytics service to analytics hudi dataset. So we need to sync the meta to Aliyun DataLake analytics , but the hudi-hive-sync just support hive. Hudi as open datalake engine, will support more meta service and analytics engine.
So, I am proposing to support for abstract the common hudi-sync. Then other service like aws alue、aliyun datalake analytics can implement.
Background
Currently Hudi only supports sync dataset metadata to Hive through hive jdbc and IMetaStoreClient. When you need to sync to other frameworks, such as aws glue, aliyun DataLake analytics, etc.
You need to copy a lot of code from HoodieHiveClient, which creates a lot of redundant code. So need to redesign the hudi-hive-sync module to support other frameworks and reuse current code as much as possible. Only the interface is provided by Hudi, and the implement is customized by different services as hive 、aws glue、aliyun DataLake analytics.
Implementation
To solve the above problems, we need to refactor the current module organization and code implementation.
2.1 module organization
2.1.1 Option1
- Added hudi-common-sync module, which is used to place common interface implementation
- The hudi-hive-sync module remains unchanged and depends on the hudi-common-sync module
2.1.2 Option2
- Keep the current module unchanged, that is, no new module is added;
The organizational form of Option1 will be clearer, and user dependence will be clearer; but an additional module will be added.
Option2 has no changes to the current module organization, but the general interface implementation is placed in the hudi-hive-sync module. The user-defined implementation depends on this module, which is a bit semantically strange.
I personally prefer Option1.
2.2 Code (class) structure
Among them, AbstractHoodieSyncClient is an abstract synchronization client, the default implementation is HoodieHiveClient; users can customize to implement Client.
The abstract methods in AbstractHoodieSyncClient are below
public abstract void createTable(String tableName, MessageType storageSchema,
String inputFormatClass, String outputFormatClass, String serdeClass);
public abstract boolean doesTableExist(String tableName);
public abstract Option<String> getLastCommitTimeSynced(String tableName);
public abstract void updateLastCommitTimeSynced(String tableName);
public abstract void addPartitionsToTable(String tableName, List<String> partitionsToAdd);
public abstract void updatePartitionsToTable(String tableName, List<String> changedPartitions);
If option1 is used, then AbstractHoodieSyncClient will be put into the hudi-common-sync module; if option2 is used, then AbstractHoodieSyncClient will be put into hudi-hive-sync.
AbstractSyncTool is an abstract synchronization tool. All synchronization tools must inherit this class, and the default implementation is HiveSyncTool.
Rollout/Adoption Plan
- No impact on the existing users because the existing HoodieHiveClient implement not changed
- New meta client can implement
Test Plan
- Unit tests
- Integration tests
- Test on the cluster for a larger dataset.