Apache Kylin : Analytical Data Warehouse for Big Data
Page History
...
-
For Hudi souce integration:
- Add new config item in kylin.property for Hudi source type(e.g: isHudiSouce=true, HudiType=MOR)
- Add new ISouce interface and implemenation using Hudi native client api
- Use Hudi client API's optimal view query api on top of hive external table to extract the source hudi dataset
-
For Hudi cuboid storage(TBD):
- Add new config item in kylin.property for Hudi storage type for cuboid(e.g: isHudiCuboidStorage=true)
- Add new ITarget interface and implementation using Hudi write api for interm store and operations of cuboid files
-
For cube rebuild with new Hudi souce type(TBD):
- Use Hudi's incremental query api to only extract the changed data from last time of Cube segement's timestamp
- Use Hudi's upset API to merge the changed data & former history data of cuboid
-
For cube merge with new Hudi cuboid storage type(TBD):
- Use Hudi's upset API to merge the 2 cuboid files
...
Overview
Content Tools
ThemeBuilder
Apps