...

Therefore, we plan to introduce the ClickHouse DataStream Connector & Flink ClickHouse SQL Connector & Catalog, and do the function adaptation of BalancedClickhouseDataSource, so as to make the integration of Flink and ClickHouse more convenient and easy to use.

Proposed Changes

1. Introduce ClickHouse DataStream Connector.
2. Introduce ClickHouse SQL Connector.
3. Introduce ClickHouse Catalog.

Things to confirm

About ClickHouseDynamicTableSource

...

1. Support writing distributed and local tables
   1. write into a distributed table
      1. We don't need to care about the load balancing of ClickHouse table.
      2. In the case of asynchronous writing to distributed tables, data may be lost. When using the synchronous write mechanism, there may be a certain delay.
      3. And Writing data into a  distributed table will cause the pressure to the network & io loading of ClickHouse cluster. This limitation has nothing to do with how to implement the connector, which is the limitation of ClickHouse itself. 
   2. write into local tables
      1. The write frequency is controlled according to the data batch size and batch interval to achieve a balance between part merge pressure and data real-time for ClickHouse.
      2. The BalancedClickhouseDataSource can  ensure theoretically the load balance of each ClickHouse instance through random routing, but it is only detect the instance through periodic Ping, shields the currently inaccessible instances, and has no failover. Therefore, once we try to write to the failed node, we will lose data. We can introduce a retry mechanism that can configure parameters to minimize this situation.
      3. Enhance the route strategies to ClickHouse instance. e.g. round robin, shard key hash.
2. ClickHouse does not support the transaction feature, so we do not need to consider the consistency of data writing

Proposed Changes

1. Introduce ClickHouse DataStream Connector.
2. Introduce ClickHouse SQL Connector.
3. Introduce ClickHouse Catalog.

Related Classes

ClickHouseDynamicTableSource

...

Compatibility, Deprecation, and Migration Plan

• Introduce ClickHouse SQL ClickHouse  connector for users
• It will be a new feature, so we needn't phase out the older behavior.
• we don't need special migration tools

...