Versions Compared

Old Version 7

changes.mady.by.user Mason Chen

Saved on

compared with

New Version 8

changes.mady.by.user Mason Chen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Shortly describe the system that your connector will connect to. In case there is already a connector available, please explain why you're introducing either a new one or an improvement via this FLIP. 

This source will extend the KafkaSource to be able to read from multiple Kafka clusters within a single source. In addition, the source can adjust the clusters and topics the source . consumes from dynamically, without Flink job restart.

Some of the challenging use cases that these features solve are:

  1. Transparent Kafka cluster migration addition/removal without Flink job restart.
  2. Transparent Kafka topic migration addition/removal without Flink job restart.
  3. Direct integration with Hybrid Source.

Public Interfaces

Please include the Flink interfaces that you will use when implementing your connector. You most likely will use one or more of these:

...

The source will use the FLIP-27: Refactor Source Interface

...

 to integrate it with Flink.

This proposal does not include any changes to existing public interfaces. A new MultiClusterKafkaSource builder will serve as the public API and all other APIs will be marked as Internal in this proposal. Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this connector.

Proposed Changes

Describe the new connector you are proposing in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change. Please include details on implementation including what destinations your connector will support, batch and stream support, async or synchronous connector implementation, etc.

...