You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Status

Current state: Under Discussion

Discussion thread: TBD

JIRA:  Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

When consumer subscribes to a non-existent topic, it does not have control over whether the topic is automatically created or not. We have the broker configuration auto.create.topics.enable that controls topic auto-creation globally. If there is a lag between the consumers and producers starting up, we usually do not want the consumer to be able to automatically create the topic as this will use the default number of partitions and replication factor. Consumers should thus have the ability to be able to disable auto topic creation, if needed.

To work around the lack of such a configuration, consumers avoid subscribing to specific topics and use wildcards instead, like in Streams. This is not optimal as it means clients will ask for and receive metadata for topics and partitions in the cluster, which could be large.

Public Interfaces

We will add a new consumer configuration auto.create.topics.enable which is similar in spirit to its broker counterpart. If set to false, the topic being subscribed to will not be auto-created, regardless of what this configuration is set to on the broker.

Configuration Name: auto.create.topics.enable
Type: BOOLEAN
Explanation: This configuration controls whether the topic is automatically created when subscribing to a non-existent topic. This configuration is only applicable with 1.0 or higher versions of the broker. A topic will be auto-created only if this configuration as well the corresponding server-side configuration are both set to true.
Default Value: TRUE
Priority: MEDIUM

Proposed Changes

Whether the topic should be auto-created will be included in MetadataRequest sent by the consumer. MetadataRequest v4 had earlier introduced a way to specify if a topic should be auto-created when requesting metadata for specific topics.

Compatibility, Deprecation, and Migration Plan

NA

Rejected Alternatives

None

  • No labels