...
Users in large infrastructure setups often need to process and/or join data that lives live in different one or more Kafka clusters in Flink. In addition, multiple Kafka clusters may be relevant when Kafka consumers need to migrate Kafka clusters for failover or upgrades.
Some of the challenging use cases that this feature solves are:
...
This source will extend the KafkaSource to be able to read from multiple a dynamic number of Kafka clusters within a single source and introduces an interface to enable more sophisticated automation/coordination between Kafka and Flink infrastructure.
...