THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
KAFKA-2798 (0.9.0.0) introduces capabilities for each source connector and sink connector to inherit their client configurations from the worker properties. Within the worker properties, any configuration that has a prefix of "producer." or "consumer." are applied to all source connectors and sink connectors respectively. The intent of the original change was to introduce a namespaced set of configurations to isolate the settings for different components, but it leaves users with a great deal of inflexibility on the connector level, as all connectors are forced to exhibit those same set of properties.
In this KIP, I propose adding support of supporting connector-level configurability of producer/consumer client configs that are optionally provided into the users can optionally provide within connector properties to override worker-level settings.
| Jira | ||||||
|---|---|---|---|---|---|---|
|
New or Changed Public Interfaces
...
- Upon initialization, all Kafka Connect source connectors will iterate through their list of connector properties, looking for those with keys prefixed by "producer."
- Upon initialization,
...
- all Kafka Connect sink connectors will iterate through their list of connector properties, looking for those with keys prefixed by "consumer."
Proposed Changes
Connector properties that are prefixed with "producer." and "consumer." are now used to feed into the producer and consumer clients embedded within source and sink connectors respectively. The prefixes will be removed via a String::substring() invocation, and the remainder of the configuration key will be used as the client configuration key. The value is fed directly to the client as the configuration value.
...