...

• The cluster has a large number of topics and total number of partitions is close to the limit.

• Most of the dynamic topics do not contain keyed messages. (See limitation section)

• The data retention period is relatively short.

This proposal is based on what is described in KIP-500 group of changes, mainly the removal of zookeeper and migrating metadata to an internal Raft quorum. 

We have implemented the described changes and deployed it in various setups internally in production environment.

Public Interfaces

Configuration Changes

...

Compatibility, Deprecation, and Migration Plan

...

The proposed change is compatible with Kafka clients backwards and forwards with current constraints

• When older clients access new broker and cannot interpret 'mode', we suggest the administrator to manually set delete.topic.partition.enable to true. This will restrict the Metadata  request to be above the specified version

...

• otherwise it will get LEADER_NOT_AVAILABLE.

...

• When newer clients access older broker (with new version org.apache.kafka.common.PartitionInfo class), the default value of the partition status field of the

...

• 'mode' field is ReadWrite, hence there will be no impact.

Upgrading the cluster from any older version is possible , support for this feature that you need to set delete.topic.partition.enable=true, and the Metadata protocol request version of Topic is above the specified versionwith the above situation handled.

Rejected Alternatives