Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Flink creates DynamicTableSource for each source table independently when it generates plan for a SQL job, sources are not aware of each other. In some situation we need to generate unified parameters for all sources in the same job, such as consistency requirements . For example, we use Flink as OLAP cluster for our HTAP system[1] which will synchronize data from an OLTP system to column storage at the database level, and then perform OLAP queries in Flink cluster on a unified version to ensure consistency of results. Similarly, we also mentioned in FLIP-276 [1]2] that it is necessary to support consistent queries across multiple tables. We would like to introduce customized provider in planner for Flink SQL job to generate dynamic parameters for all source tables, then Flink job can create sources based on these parameters.

...

This is a newly added feature, so there will be no compatibility issues

Test Plan

UT & IT


[1] https://www.vldb.org/pvldb/vol15/p3411-chen.pdf

[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-276%3A+Data+Consistency+of+Streaming+and+Batch+ETL+in+Flink+and+Table+Store

...