You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Status

Current state: Under Discussion

Discussion thread: N/A

JIRA: KAFKA-19740 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Following KIP-1022: Formatting and Updating Features, the group.coordinator.rebalance.protocols server configuration has become redundant. KIP-848 (New Group Coordinator), KIP-932 (Share groups), and KIP-1071 (Streams rebalance protocol) now implement group.version , streams.version , and share.version  respectively. These feature versions enable cluster-wide upgrades and downgrades through kafka-features.sh, eliminating the need for server-level configuration to control feature versions.

Having a server-level configuration is problematic because individual brokers can be misconfigured, resulting in a cluster operating in a mixed state. Moreover, the current implementation shows transitional behavior across different coordinator types. While share groups have completely decoupled from group.coordinator.rebalance.protocols, the new group coordinator and streams still rely on this configuration. This means that even when the feature version is upgraded via kafka-features.sh, the new capabilities remain unavailable unless the server configuration is also updated. This dual requirement exists because the feature flag mechanism was introduced later in the development cycle, resulting in a transitional period where both systems coexist. This can lead to user confusion when feature upgrades don't immediately enable new capabilities.

Public Interfaces

  • org.apache.kafka.coordinator.group.GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG

Proposed Changes

  • The group.coordinator.rebalance.protocols configuration will be deprecated in Kafka 4.x and removed in version 5.0, following the same approach as Share Groups. The configuration will be decoupled from feature functionality, meaning it will be accepted but ignored during the transition period. Both New Group Coordinator and Streams will follow the Share Groups model where functionality is controlled solely by group.version  and streams.version  feature via kafka-features.sh. This eliminates the dual requirement of setting both configuration and feature flags, providing a consistent experience across all coordinator types. Users should use kafka-features.sh for all feature version management, and can safely remove the configuration from their server.properties files as it no longer affects feature functionality.

Compatibility, Deprecation, and Migration Plan

For the group coordinator, after users upgrade, the system will maintain the same version as before the upgrade because feature information is persisted. Users can upgrade using kafka-features.sh without concern for the config value, as it has been decoupled from the feature version.

As for Share Groups and Streams rebalance protocol, the same applies. Additionally, since neither has reached general access yet, there are no migration concerns.


Test Plan


Rejected Alternatives


References

  • KIP-848: The Next Generation of the Consumer Rebalance Protocol
  • KIP-932: Queues for Kafka (Share Group)
  • KIP-1022: Formatting and Updating Features
  • KIP-1071: Rebalance Protocol for Kafka Streams
  • No labels