Page History

...

• The topology metadata that was provided through the StreamsGroupInitialize API.

• The latest member metadata that each member provided through the StreamsGroupHeartbeat API.

• The current target assignment generated by the assignor.

We are delaying also include extensions of the Admin API and the command-line tools to describe and modify streams groups to a follow-up KIP. 

Rebalance Process

Similar as in KIP-848 consumer groups, the rebalance process is entirely driven by the group coordinator, and based on the same three epochs: group epoch, assignment epoch, and member epoch. The main difference is the assigned resource - instead of assigning partitions to consumers, we assign tasks consisting of a subtopology ID and a partition number, and each task can be assigned to a client in three roles: as an active task, as a standby task or as a warm-up task.

...

• The group epoch will be bumped every time the topology is updated through the StreamsGroupInitialize API.

• The partition metadata for the group, which is tracked by the group coordinator, is the partition metadata of all input topics (i.e., user source topics and internal repartition topics we read from).

• The group epoch is bumped every time a member:

  • When a client member with an assigned warm-up task reports a task changelog offset and task changelog end offset whose difference is less that acceptable.recovery.lag.

  • Updates When a member updates its topology ID, rack ID, client tags or process ID. Note: Typically, these do not change within the lifetime of a Streams client, so this only happens when a member with static membership rejoins with an updated configuration.

...

We make core assignment options configurable centrally on the broker, without relying on each clients configuration. This allows tuning a streams group without redeploying the streams application. The three core assignment options will be introduced on the broker-side: acceptable.recovery.lag, maxnum.warmup.replicas and num.standby.replicas. They can be configured both globally on the broker, and dynamically for specific streams groups through the IncrementalAlterConfigs and DescribeConfigs RPCs.

...