Versions Compared

Old Version 6

changes.mady.by.user Konstantine Karantasis

Saved on

compared with

New Version 7

changes.mady.by.user Konstantine Karantasis

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Key formatKey exampleValue formatValue example
topic-${topic-name}-:connector-${connector-name}
topic-foo-:connector-some-source

"topic": {
  "connector": string,
  "task": int32,
  "name": string
}

"topic": {
  "connector": "some-source",
  "task": 0,
  "name": "foo"
}

The topic name can be safely separated by the connector name because the delimiter : (colon character) is not a valid topic name character in Kafka. 

Connect workers reading this topic are computing a map from connector names to sets of topic names. This map represents an up-to-date view of which connectors are using which topics at any time based on the key of these records and whether the value entry of the record is null or non-null. If the most recent record value for a specific topic and a specific connector is non-null, then this topic is considered part of the set of topics that this connector is using. If the most recent record has a null value (is therefore a tombstone record), then the connector is not using this topic anymore. Tombstone records are interpreted by Connect workers as deletion actions of previous topic status records for this connector.

...

  1. Expose topic tracking information for each connector programmatically by extending the Connector abstract class. Implementation of topic tracking and observability through the Connect REST API seems adequate to support a rich set of use cases. 
  2. Expose topic tracking information for each connector as a metric. This solution is not as flexible and easy to consume as introducing a Connect REST API endpoint. Additionally, Connect has already been preferring exposing similar information via its REST API instead of jmx. It seems preferable for this KIP to continue on that path.
  3. Introduce a new, separate, internal topic to record topic tracking information. This seems unnecessary at this point because the status.storage.topic is a low volume topic that already shares the same specifications with the topic that would be required by this feature. Keeping topic tracking information in the status.storage.topic keeps the migration and compatibility implications very lightweight.