THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Abstract
Background
System overview
Bytelake is a metadata management system compatible with Hive metastore, and can manage the data lake related metadata. It is designed to store metadata by using pluggable storage, which can be a distributed file system, kv store or a database.
Features:
- Pluggable: The storage layer is designed to be pluggable without depending on a particular system. It can be a distributed file system, kv store, database or any storage system.
- Snapshot Management:
- Concurreny Control: Each hudi table has a
version
which is atomic and self-increasing to archive snapshot isolation and optimistic concurrency control. Theversion
is maintained in the metastore and persisted in the storage layer. - Multiple Conflict Resolution Strategies: conflict check strategy at partition, file group and column level.
- Compatible with Hive:
- Lightweight and Easy to expand: The metastore server is a stateless service that can be scaled horizontally.