THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
The producer will have a few new configurations added. Default values of -1 will use broker defaults or raise an exception.
| Name | Description | Type | Default | Valid Values | Importance |
|---|---|---|---|---|---|
| auto.create.topics.enable | Enable auto-creation of topics on the producer. Even if the broker config `auto.create.topics.enable` does not allow auto-creation server-side, topics can still be created automatically client-side. | boolean | false | medium | |
| auto.create.num.partitions | The default number of log partitions per topic for auto-created topics. | int | -1 | medium | |
| auto.create.replication.factor | The default replication factors for automatically created topics. | int | -1 | medium |
The goal is to deprecate automatic topic creation on the broker. At some point in the future, it may be useful to set the broker's auto.create.topics.enable default to false. However, for compatibility reasons, doing so is not possible at this time. Likewise, it would make sense to change the default of the consumer's allow.auto.create.topics to false since it has no effect if the broker configuration is false.
Proposed Changes
The idea is to move the configuration and ability to automatically create topics to client-side on the producer. Broker auto-topic creation will be deprecated. This includes deprecating the configuration 'auto.create.topics.enable' on the broker and 'allow.auto.create.topics' on the consumer. In the producer, auto-creation of a topic will occur through a specific request rather than through a side effect of requesting metadata. Unlike the configuration in the consumer, the ability to automatically create topics client-side will be independent of the server-side automatic topic creation controlled by the broker configuration.
Auto-creation will try to use client-side configurations if set. However, if none exist, as represented by -1, broker-side defaults can be used if the broker supports KIP 464. Auto-creation is not supported for older brokers if no client configurations are given.
...
In order to automatically create topics with the producer, the producer's auto.create.topics.enable config must be set to true and the broker config should be set to false. If the broker does not support KIP 464, auto.create.num.partitions and auto.create.replication.factor must be specified.
...
- Systems that rely on auto-creation would be encouraged to set the producer config auto.create.topics.enable to true, but could also rely on the deprecated broker config if it is enabled.
- Systems with older broker versions would need to set auto.create.num.partitions and auto.create.replication.factor upon allowing producer auto-creation.Setting both
- Enabling the configuration on the broker and producer auto.create.topics.enable to true is not advised. the producer should not have any negative consequences, but there is no benefit in enabling both either.
- Systems that utilize auto-creation on the consumer will no longer be able to do so with only the producer config enabled.
- Enabling the broker config or making further changes to the code is required.
- Systems that rely on auto-creation would be encouraged to set the producer config auto.create.topics.enable to true, but could also rely on the deprecated broker config if it is enabled.
...
- Enabling true by default on the producer config: Setting both the broker config and the producer config to true could result in both attempting to create new topics and result in excess requests to the broker.
- Topic auto-creation is a feature that is not supported as it once was, so defaulting to false supports this trend towards using admin client to create topics. Auto-creation does not work on older brokers without configuration client-side anyway, so it might be more clear that configuration must be done in order to keep using this feature.