Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. The co-ordinator for a group detects changes to the number of partitions for the subscribed list of topics. The co-ordinator then increments the group's generation id in zookeeper.
  2. The controller watches the generation id changes for every consumer group and sends a UpdateConsumerMetadataRequest to all brokers with the new generation id for the affected group.
  3. The co-ordinator also closes the socket connection to all consumers in the group to trigger a rebalance operation.
  4. On losing connection to the co-ordinator, the consumers go through the co-ordinator re-discovery process. If the consumer receives a higher generation id in either the ConsumerMetadataResponse or the HeartbeatResponse, it stops fetching data, commits offsets and sends a JoinGroupRequest to the co-ordinator. 
  5. The co-ordinator waits for all consumers to send it the JoinGroupRequest for the group. Once it receives all expected JoinGroupRequests, it computes the new partition assignment and returns the updated assignment and the new generation id in the JoinGroupResponse
  6. On receiving the JoinGroupResponse, the consumer stores the new generation id and the consumer id locally and starts fetching data for the returned set of partitions.

Offset commits during rebalance