THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Table of Contents |
|---|
Status
Current state: Under Discussion
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
This is a followup to KIP-154 and KIP-464.
...
The Kafka Connect distributed worker configuration also does not support defining or passing other topic settings when the Connect worker creates the configuration, offset, and status topics.
Public Interfaces
The allowed values for the following existing properties will be changed to also allow -1 to signal that the broker's default replication factor should be used for the newly created topics:
...
Also, the distributed worker configuration will be changed to recognize additional optional properties that match the following patterns:
Property pattern | Type | Default | Possible Values | Description |
|---|---|---|---|---|
config.storage.<topic-specific-setting> | several | broker value | Additional topic-specific settings used when creating the internal Kafka topic where Connect stores connector configurations. Here "<topic-specific-setting>" corresponds to any of the Kafka topic-level configurations for the version of the Kafka broker where the topic should be created. | |
offset.storage.<topic-specific-setting> | several | broker value | Additional topic-specific settings used when creating the internal Kafka topic where Connect stores source offsets for source connectors. Here "<topic-specific-setting>" corresponds to any of the Kafka topic-level configurations for the version of the Kafka broker where the topic should be created. | |
status.storage.<topic-specific-setting> | several | broker value | Additional topic-specific settings used when creating the internal Kafka topic where Connect stores connector and task statuses. Here "<topic-specific-setting>" corresponds to any of the Kafka topic-level configurations for the version of the Kafka broker where the topic should be created. |
These additional properties (without the prefix) will be passed to the new topic requests created by the Connect worker when it attempts to create the internal topics.
Proposed Changes
When the Connect distributed worker starts up, it currently attempts to create the configuration, offsets, and/or status topics used by the worker if any of those topics do not exist. With this proposal, the additional properties will be passed to the broker during these new topic requests. Note that like with KIP-154, these properties will not be used if the corresponding topic already exists.
Compatibility, Deprecation, and Migration Plan
These changes are backward compatible, and existing Connect distributed worker configurations will continue to work with no change in behavior.
Rejected Alternatives
None