Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-848: The Next Generation of the Consumer Rebalance Protocol introduces new status in consumer group like group epoch, target assignment epoch, member epoch, and target assignment. Adding this information to kafka-consumer-groups.sh command can help administrator to have a detailed view of the state of the groups.

Public Interfaces

Client API changes

Admin

ConsumerGroupDescription

Add new fields groupEpoch and targetAssignmentEpoch.

public class ConsumerGroupDescription {
    private final String groupId;
    private final boolean isSimpleConsumerGroup;
    private final Collection<MemberDescription> members;
    private final String partitionAssignor;
    private final GroupType type;
    private final ConsumerGroupState state;
    private final Node coordinator;
    private final Set<AclOperation> authorizedOperations;
	private final Optional<Integer> groupEpoch;
    private final Optional<Integer> targetAssignmentEpoch;

	// ...

	/**
	 * The epoch of the consumer group.
	 */
	public Optional<Integer> groupEpoch() {
		return groupEpoch;
	}

	/**
	 * The epoch of the target assignment.
	 */
	public Optional<Integer> targetAssignmentEpoch() {
		return targetAssignmentEpoch;
	}
}

MemberDescription

Add a new field memberEpoch.

public class MemberDescription {
    private final String memberId;
    private final Optional<String> groupInstanceId;
    private final String clientId;
    private final String host;
    private final MemberAssignment assignment;
    private final Optional<MemberAssignment> targetAssignment;
    private final Optional<Integer> memberEpoch;

	// ...

	/**
	 * The epoch of the group member.
	 */
	public Optional<Integer> memberEpoch() {
		return memberEpoch;
	}
}

Proposed Changes

kafka-consumer-groups.sh

--describe --members

Add GROUP-EPOCH, TARGET-ASSIGNMENT-EPOCH, and CONSUMER-EPOCH information. CONSUMER-EPOCH is equal to memberEpoch.

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members
GROUP		CONSUMER-ID            HOST            CLIENT-ID			#PARTITIONS		GROUP-EPOCH 	TARGET-ASSIGNMENT-EPOCH		CONSUMER-EPOCH
my-group	T4tbGKsvT7CtsxVBH5J2QQ /127.0.0.1      consumer-my-group-1	1				1				1							1

--describe --members --verbose

Add GROUP-EPOCH, TARGET-ASSIGNMENT-EPOCH, CONSUMER-EPOCH, and TARGET-ASSIGNMENT information. CONSUMER-EPOCH is equal to memberEpoch.

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose
GROUP		CONSUMER-ID            HOST            CLIENT-ID			#PARTITIONS		ASSIGNMENT		TARGET-ASSIGNMENT		GROUP-EPOCH 	TARGET-ASSIGNMENT-EPOCH		CONSUMER-EPOCH
my-group	T4tbGKsvT7CtsxVBH5J2QQ /127.0.0.1      consumer-my-group-1	1				(0)				(0)						1				1							1

--state

Add GROUP-EPOCH and TARGET-ASSIGNMENT-EPOCH information.

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state
GROUP		COORDINATOR (ID)          ASSIGNMENT-STRATEGY  	STATE           	#MEMBERS 	GROUP-EPOCH		TARGET-ASSIGNMENT-EPOCH
my-group	localhost:61316  (0)      uniform      		 	Stable			 	1			1				1

Compatibility, Deprecation, and Migration Plan

This proposal adds new optional fields to Admin client and kafka-consumer-groups.sh, so classic consumer groups will be unaffected.

Test Plan

The feature will be thoroughly tested with unit and integration tests.

Rejected Alternatives

N/A