This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, ConnectorUtils in Kafka provides functionality for assigning continuous elements in groups. However, there are scenarios where this approach falls short, particularly when a more balanced distribution of elements across groups is desired. In such cases, a round-robin assignment strategy can offer a more equitable allocation of elements.

Consider a scenario where a Kafka Connect connector needs to distribute data across multiple destinations or partitions in a balanced manner. The existing functionality of ConnectorUtils may result in suboptimal assignments, leading to uneven distribution and potential performance bottlenecks. Introducing a round-robin assignment strategy would enable connectors to achieve a more balanced distribution of elements, thereby enhancing overall system efficiency and performance.

For example, suppose we have 5 elements {a, b, c, d, e} and want to assign them into 3 groups. In current assignment strategy, the result will be {{a, b}, {c, d}, {e}}, we keep current strategy and add a round-robin strategy, the result will be {{a, d}, {b, e}, {c}}. In this example, elements "a", "b", and "c" are evenly distributed among three groups using the round-robin assignment strategy.

Public Interfaces

Will add a new static method to connect/api/src/main/java/org/apache/kafka/connect/util/ConnectorUtils.java

public static <T> List<List<T>> groupElementsRoundRobin(List<T> elements, int numGroups)

Proposed Changes

We propose extending ConnectorUtils with a round-robin assignment strategy to facilitate more balanced distribution of continuous elements across different groups. This strategy would evenly distribute elements among the available groups, ensuring a more equitable allocation and minimizing potential imbalances.

Compatibility, Deprecation, and Migration Plan

This proposal does not introduce any compatibility issues or deprecations. Existing functionality in ConnectorUtils will remain unchanged, and the round-robin assignment strategy will be introduced as an additional option.

Test Plan

Unit tests are enough to cover all cases.

Rejected Alternatives

Define a Strategy interface, then user can pass different strategy implementation. This solution is incompatible with current ConnectorUtils. Current strategy and round-robin strategy are enough for most scenarios.