Versions Compared

Old Version 60

changes.mady.by.user Mason Chen

Saved on

compared with

New Version 61

changes.mady.by.user Mason Chen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Metadata is first discovered on job start and new metadata is updated discovered via polling mechanism (a source configuration  see the source configuration in next section) for streaming jobs with eventual consistency. The metadata update is designed with eventual consistency–repeated metadata polling will eventually get the correct metadata and reconcile the job accordingly. For batch jobs,  this polling mechanism should be disabled.

To the source more user friendly, a MultiClusterKafkaSourceBuilder will be provided (e.g. batch mode should not turn on KafkaMetadataService discovery, should only be done at startup).

...