Status

Current state: Draft Under Discussion

Discussion thread: here 

JIRA: here

Motivation

Similiar to KIP-787: MM2 manage Kafka resources with custom Admin implementation motivations. This KIP motivation is to make sure connect can use a custom admin to create topics.

At the moment Connect can either create topics automatically or using Admin client when topic.creation.enable enabled set to true. This setup assume that Kafka cluster either has auto.create.topics.enable or that the connect client has admin access to create all needed topics.

While the current approach simplifies creating topics in connect, it created a problem for any organization that runs any sort of resource management or federated solutions where these systems are usually the only application allowed to create topics in Kafka’s ecosystem to control capacity and budget planning.

The downside of connect using AdminClient directly is that Connect create topics using AdminClient whenever topic.creation.enable is true ignoring any capacity safeguard set by the ecosystem. This increase the disk usage on the cluster. The team that runs the cluster will only notice the capacity issue when their disk usage hits the threshold of their alerts.

Currently there are 3 ways to deal with this issue:

...