Status

Current state: [Accepted]Under Discussion

Discussion thread: here

JIRA: KAFKA-7358 3333 Alternative Partitioner to Support "Always Round-Robin" Selection

Motivation

In my organisation, we have been using kafka as the basic publish-subscribe messaging system provider. Our goal is the to send event-based (secure, encrypted) SQL messages reliably, and process them accordingly. For us, the message keys represent some metadata which we use to either ignore messages (if a loopback loop-back to the sender), or log some information. We have the following use case for messaging:

...

3) A group of consumers (for each data centre with a unique consumer-group ID) are will process messages from their respective partitions. 1 consumer per partition.

...

To address this issue, we are proposing an alternative, and more concrete partitioner - "KeylessPartitionerRoundRobinPartitioner". We use "KeylessRound-Robin" as the new partitioner does not focus on the key or paritions. The partitioner code will almost be identical to DefaultPartitioner.partition() method, except that it will simply execute the "Null Key and No Partition" logic from DefaultPartitioner. The following is the content of partition() method for our new partitioner.

...

We would also like to clarify that this is not code "Duplication". We do not wish to change the DefaultPartitioner class, but want to "Reuse" certain portion of its logic to achieve our goalthis solution.

Public Interfaces

There is no requirement to change any interfaces. We simply use the existing paritioner.class config in server.properties and use a different class name. But we are not changing the default value, which is DefaultPartitioner.

...