...

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]Withdrawn

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]6387

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Describe the problems you are trying to solveThe Connect worker configuration must specify the set of configuration properties for the worker's consumers and producers used to read and write to the internal topics, and except for the basic broker connection information the configuration properties for Connect's producers and consumers that are used for source and sink connectors. Currently, the configuration between the worker's internal producers and consumers and the producers and consumers used for connectors are unrelated. To simplify configuration in some situations, the producer and consumer used for connectors should inherit from the worker-level configuration properties.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

No public interfaces (code or configuration definitions) will change.

Proposed Changes

Connect worker configurations define connection properties for the following types of connections being made to the Kafka cluster:

1. the worker group membership and producers/consumers for internal topics, with no prefix
2. producers for source connectors, prefixed with "producer." (e.g., "producer.retries=1").
3. consumers for sink connectors, prefixed with "consumer." (e.g., "consumer.max.partition.fetch.bytes=10485760").

Currently, any top-level configurations are set for the internal producer/consumers do not affect the producers and consumers used for connectors. So, if the same configuration is to be overridden for all consumers/producers, the configuration must be set in multiple places.

This proposal suggests changing the behavior so that the configuration for the producer and consumer used for connectors (items 2 and 3 in the list above) inherits any configurations defined at the top-level (item 1 in the list above). The producer.* and consumer.* properties would override any inherited properties, resulting in exactly the same behavior as todayDescribe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Rejected Alternatives

NOTE: After further evaluation, there are a number of cases where this behavior might actually change how the producers and consumers used for connectors are configured. Consider a worker configuration includes the following configuration properties:

retries=2
max.partition.fetch.bytes=262144

This configuration file does not define "consumer.max.partition.fetch.bytes" or "producer.max.partition.fetch.bytes", so currently these would default to 1048576. However, after this proposed change if implemented the "consumer.max.partition.fetch.bytes" and "producer.max.partition.fetch.bytes" values would be the inherited value of 262144, not the default 1048576.

In short, implementing this change would break backward compatibility. We could implement a configuration switch that controls whether the configurations are inherited, but this adds complexity to the already-complex configuration mechanism. Therefore, withdrawing this KIP.

Rejected Alternatives

Using a new configuration property to control whether the configuration properties are inherited would work, but would also add unnecessary complexityIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.