...
Strong Consistency
It will grantee guarantee strong data consistency among queries above. Query gets the minimum version of all the related tables according to the source tables and the dependencies between them, which ensure data consistency between related tables. For the examples above, Query1, Query2 and Query3 will get Min(table1 version, table2 version) for table1 and table2, Min(table3 version) for table3.
Weak Consistency
It doesn't grantee guarantee the data consistency among queries above, but only the data consistency of a single query. At this time, each query can get its latest version of tables, this ensures better data freshness.
...
There are two layers in Global Checkpoint
: MetaService
and JobManager
. MetaService
regards each ETL job as a single node, manages the global checkpoin checkpoint in the ETL Topology
; JobManager
interacts with MetaService
and manages the global checkpoint in each ETL job.
There are two parts in the global checkpoint processing: interaction between MetaService
and JobManager
, and interaction between JobManager
and Source Node
.
...
MetaService
manages completed checkpoints of each table and grantees guarantees data consistency in OLAP query. OLAP query should request versions of source tables from MetaService
, and MetaService
calculates snapshot ids of tables based on the dependencies between tables, completed checkpoints and snapshots in each table and consistency type requirement. OLAP reads data from tables according to the given snapshot ids, which ensure the data consistency for it.
...
Flink jobs read snapshots from Table Store. When a job fails, it will read reread snapshots according to the latest checkpoint. The relationship between checkpoint and snapshot is determined. The failed job can read the same data from the same snapshot according to the same checkpoint, which means the job will read determined data from Table Store before and after failover.
...