This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"]

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread] 

JIRA: KAKFA-3741here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Describe the problems you are trying to solve.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

There is presently no way in Kafka Streams to provide default configs for the internal topics that are created. Anyone that wants to provide topic configs can only do it via the StateStoreSupplier API, which is cumbersome and only applies to changelog topics. This will usually result in having to create the topics first (which involves resolving the generated name), and/or changing the configuration after the topics have been created.

We should provide a prefix that can be used when creating a StreamsConfig. The prefix will mark these configs as Topic Configs and should be applied at the time any internal topics are created.

Public Interfaces

We will add the following field and method to StreamsConfig

/**
 * Prefix used to provide default topic configs to be applied when creating internal topics.
 * These should be valid properties from {@link org.apache.kafka.common.config.TopicConfig TopicConfig}.
 * It is recommended to use {@link #topicPrefix(String)}.
 */
public static final String TOPIC_PREFIX = "topic.";
 
/**
 * Prefix a property with {@link #TOPIC_PREFIX}
 * used to provide default topic configs to be applied when creating internal topics.
 *
 * @param topicProp the topic property to be masked
 * @return TOPIC_PREFIX + {@code topicProp}
 */
public static String topicPrefix(final String topicProp) {
    return TOPIC_PREFIX + topicProp;
}

Proposed Changes

We will add the field and method described above to StreamsConfig. At topic creation time we will extract all properties from StreamsConfig that have the TOPIC_PREFIX and use these as the default configs when creating any internal topics. Any configs that are currently provided, i.e., via a StateStoreSupplier, will override any of the defaults that have been set in StreamsConfig. Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Rejected Alternatives

• None

Rejected Alternatives

Adding each individual topic config to StreamsConfig. We already have these params duplicated elsewhereIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.