Versions Compared

Old Version 29

changes.mady.by.user Maximilian Michels

Saved on

compared with

New Version 30

changes.mady.by.user Mason Chen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Other required functionality leverages and composes the existing KafkaSource implementation for discovering Kafka topic partition offsets, committing offsets, doing the actual Kafka Consumer polling, snapshotting state, and split coordination per Kafka cluster. We will reuse the code of the KafkaSource in order to achieve this.

To the source more user friendly, a MultiClusterKafkaSourceBuilder will be provided (e.g. batch mode should not turn on KafkaMetadataService discovery, should only be done at startup).

...