Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Strong Consistency

It will grantee guarantee strong data consistency among queries above. Query gets the minimum version of all the related tables according to the source tables and the dependencies between them, which ensure data consistency between related tables. For the examples above, Query1, Query2 and Query3 will get Min(table1 version, table2 version) for table1 and table2, Min(table3 version) for table3.

  • Weak Consistency

It doesn't grantee guarantee the data consistency among queries above, but only the data consistency of a single query. At this time, each query can get its latest version of tables, this ensures better data freshness.

...

There are two layers in Global Checkpoint: MetaService and JobManager. MetaService regards each ETL job as a single node, manages the global checkpoin checkpoint in the ETL Topology; JobManager interacts with MetaService and manages the global checkpoint in each ETL job.
There are two parts in the global checkpoint processing: interaction between MetaService and JobManager, and interaction between JobManager and Source Node.

...

MetaService manages completed checkpoints of each table and grantees guarantees data consistency in OLAP query. OLAP query should request versions of source tables from MetaService, and MetaService calculates snapshot ids of tables based on the dependencies between tables, completed checkpoints and snapshots in each table and consistency type requirement. OLAP reads data from tables according to the given snapshot ids, which ensure the data consistency for it.

...

Flink jobs read snapshots from Table Store. When a job fails, it will read reread snapshots according to the latest checkpoint. The relationship between checkpoint and snapshot is determined. The failed job can read the same data from the same snapshot according to the same checkpoint, which means the job will read determined data from Table Store before and after failover.

...