Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

With wildcard subscription (for example, whitelist and blacklist), the consumers are responsible to discover matching topics through topic metadata request. That is, its topic metadata request will contain an empty topic list, whose response then will return the partition info of all topics, it will then filter the topics that match its wildcard expression, and then update the subscription list. Again, if the subscription list has changed from the previous values, it will start the consumer re-join group procedure. 

Co-ordinator failover or connection loss to the co-ordinator

  1. On co-ordinator failover, the controller elects a new leader for a subset of the consumer groups affected due to the co-ordinator failure. As part of becoming the leader for a subset of the offset topic partitions, the co-ordinator reads metadata for each group that it is responsible for, from zookeeper. The metadata includes the group's consumer ids, the generation id and the subscribed list of topics. Until the co-ordinator has read all metadata from zookeeper, it returns the CoordinatorNotReady error code in HeartbeatResponse, ConsumerMetadataResponse and JoinGroupResponse.
  2. As part of electing a new consumer co-ordinator, the controller also sends the UpdateConsumerMetadataRequest to all brokers. If a consumer sends a ConsumerMetadataRequest to a broker before the broker has received the updated group metadata through the UpdateConsumerMetadataRequest from the controlller, the ConsumerMetadataResponse will return stale information about the co-ordinator. The consumer will receive NotCoordinatorForGroup error code on the heartbeat/commit offset responses. On receiving the NotCoordinatorForGroup error code, the consumer backs off and resends the ConsumerMetadataRequest.
  3. The consumer does not stop fetching data during the co-ordinator failover and re-discovery process.

Partition changes for subscribed topics

  1. The co-ordinator for a group detects changes to the number of partitions for the subscribed list of topics. The co-ordinator then increments the group's generation id in zookeeper.
  2. The controller watches the generation id changes for every consumer group and sends a UpdateConsumerMetadataRequest to all brokers with the new generation id for the affected group.
  3. The co-ordinator also closes the socket connection to all consumers in the group to trigger a rebalance operation.