...
| Component | Main Work |
|---|
| MetaService | - Manage the relationship between ETL job and table in
Table Store , including source table, sink table. - Manage the finished timestamp barrier of each table in
Table Store - Interaction between Flink and MetaService, such as register ETL job, get consistency version of table and ect.
|
Table Store | Catalog | - Register source and sink table with ETL job id.
Create table based on a consistency version from MetaService
|
| Source and SplitEnumerator | - SplitEnumerator managers the snapshot, split and timestamp barrier for specific table.
- Source read data and timestamp barrier from split, broadcast timestamp barrier
- Notify MetaService to update the completed timestamp barrier for tables.
- Notify MetaService to cleanup the information of the terminated ETL job.
|
| Sink | - Write data to table store and commit data with timestamp barrier
|
Flink | Timestamp Barrier Mechanism | The detailed and main work is in the above table |
| Planner | - Register job to MetaService to create relationship between source and sink tables.
- Create table based on a consistency version from MetaService
|
| JobManager | Add a listener and call back it when the job ends
|
| Improvement | - Delta Join in Flink to improve state resource
- Storage & Computation Decouple in Table Store
|
Constraint
The current FLIP design has two constraints and it may continue to improve in the future
...
- Timestamp Barrier Coordination and Generation
- Timestamp Barrier Checkpoint and Recovery
- Timestamp Barrier Replay Data Implementation
- Timestamp Barrier Alignment and Computation In Operator
- Introduce Delta Join in Flink To Optimize Improve State Resource
- Introduce MetaService module and implement source/sink in Table Store and etc
- Job and Table management in MetaService such as exception handling, data revision and etc
...