Current state: Under Discussion
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here
Currently, ConnectorUtils in Kafka Connect API provides functionality for assigning continuous elements in groups. However, there are scenarios where this approach falls short, particularly when a more balanced distribution of elements across groups is desired. In such cases, a round-robin assignment strategy can offer a more equitable allocation of elements.
Consider a scenario where a Kafka Connect connector needs to distribute data across multiple destinations or partitions in a balanced manner. The existing functionality of ConnectorUtils may result in suboptimal assignments, leading to uneven distribution and potential performance bottlenecks. Introducing a round-robin assignment strategy would enable connectors to achieve a more balanced distribution of elements, thereby enhancing overall system efficiency and performance.
For example, suppose we have 5 elements {a, b, c, d, e} and want to assign them into 3 groups. In current assignment strategy, the result will be {{a, b}, {c, d}, {e}}, we keep current strategy and add a round-robin strategy, the result will be {{a, d}, {b, e}, {c}}. In this example, elements "a", "b", and "c" are evenly distributed among three groups using the round-robin assignment strategy.
Will add a new static method to connect/api/src/main/java/org/apache/kafka/connect/util/ConnectorUtils.java
public static <T> List<List<T>> groupElementsRoundRobin(List<T> elements, int numGroups)
Extend ConnectorUtils with a round-robin assignment strategy to facilitate more balanced distribution of continuous elements across different groups. This strategy would evenly distribute elements among the available groups, ensuring a more equitable allocation and minimizing potential imbalances.
This proposal does not introduce any compatibility issues or deprecations. Existing functionality in ConnectorUtils will remain unchanged, and the round-robin assignment strategy will be introduced as an additional option.
No other classes depend on this util. The change only adds a new static method and won't break existing method. So, unit tests are enough to cover all cases.
Define a Strategy interface, then user can pass different strategy implementation. This solution is incompatible with current ConnectorUtils. Current strategy and round-robin strategy are enough for most scenarios.