Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka has a feature that allows for the auto-creation of topics. Upon a produce or consume request, when fetching metadata for a topic that does not yet exist, the topic may be created if the broker enables topic auto creation. In the early days of Kafka, this was one of only a few ways to create topics. However, with the addition of AdminClient, this functionality is no longer the recommended way to create topics. 

...

However, many cloud users rely on the auto-create functionality. They often use software that they did not write and can not easily change that relies on auto-creation. Thus, moving the auto-create functionality to the producer will still support topic auto-creation, but will do away with some of the negative aspects of the current implementation. Auto-creation can then be configured on a client-by-client basis.

Public Interfaces

The producer will have a few new configurations added. Default values of -1 will use broker defaults or raise an exception.

...

The goal is to deprecate automatic topic creation on the broker. At some point in the future, it may be useful to set the broker's auto.create.topics.enable default to false. However, for compatibility reasons, doing so is not possible at this time.

Proposed Changes

The idea is to move the configuration and ability to automatically create topics to the producer.  Broker auto-topic creation will be deprecated. In the producer, auto-creation of a topic will occur through a specific request rather than through a side effect of requesting metadata. 

Auto-creation will try to use client-side configurations if set. However, if none exist, as represented by -1, broker-side defaults can be used if the broker supports KIP 464. Auto-creation is not supported for older brokers if no client configurations are given.

Compatibility, Deprecation, and Migration Plan

This KIP argues for deprecation of the broker config auto.create.topics.enable. However, the default will remain true, so there will be no compatibility issues with current usage of the config and its functionality.

...

• • Systems that rely on auto-creation would be encouraged to set the producer config auto.create.topics.enable to true, but could also rely on the deprecated broker config.
    • Systems with older broker versions would need to set auto.create.num.partitions and auto.create.replication.factor upon allowing producer auto-creation.
    • Setting both the broker and producer auto.create.topics.enable to true is not advised. 
  • Systems that utilize auto-creation on the consumer will no longer be able to do so with the producer config.
    • Enabling the broker config or making further changes to the code is required.

Rejected Alternatives

• Enabling true by default on the producer config:
• Setting both the broker config and the producer config to true could result in both attempting to create new topics and result in excess requests to the broker.
• Topic auto-creation is a feature that is not supported as it once was, so defaulting to false supports this trend towards using admin client to create topics. Auto-creation does not work on older brokers without configuration client-side anyway, so it might be more clear that configuration must be done in order to keep using this feature.