...
Initial state with a group of OZKCCs:
Gliffy Diagram |
---|
| |
---|
size | 600 |
---|
name | Initial state with a group of OZKCCs |
---|
|
Begin migration from OZKCCs to MEZKCCs:
Gliffy Diagram |
---|
| |
---|
size | 600 |
---|
name | Begin migration from OZKCCs to MEZKCCs |
---|
|
The group has fully migrated to MEZKCCs while still using zookeeper-based coordination:
Gliffy Diagram |
---|
| |
---|
size | 600 |
---|
name | The group has fully migrated to MEZKCCs while still using zookeeper-based coordination |
---|
|
The coordination mode toggle is applied so that the group of MEZKCCs use uses kafka-based coordination:
Gliffy Diagram |
---|
| |
---|
size | 600 |
---|
name | The coordination mode toggle is applied so that the group of MEZKCCs uses kafka-based coordination |
---|
|
Begin migration from MEZKCCs to KCs:
Gliffy Diagram |
---|
| |
---|
size | 600 |
---|
name | Begin migration from MEZKCCs to KCs |
---|
|
Final state with a group of KCs:
Gliffy Diagram |
---|
| |
---|
size | 600 |
---|
name | Final state with a group of KCs |
---|
|
Rejected Alternatives
- Adapt
KafkaConsumer
to understand zookeeper-based coordination. This approach was rejected since it introduces a zookeeper dependency into kafka-clients
. - Build a wrapper class comprised of a
ZookeeperConsumerConnector
and KafkaConsumer
. When the coordination mode trigger is fired, toggle consumption to the corresponding consumer. This approach was rejected for several reasons:- It requires a transformation from
ZookeeperConsumerConnector
's or KafkaConsumer
's consumption API into the wrapper class consumption API while properly tracking offsets.- Should it adopt
ZookeeperConsumerConnector
's API of providing KafkaStream
s? - Should it adopt
KafkaConsumer
's polling API that provides ConsumerRecords
?
- It introduces yet another consumer client to kafka
- Users would need to change their code to use the new client
- Embed a
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator
inside ZookeeperConsumerConnector
instead of a KafkaConsumer
. This approach was rejected because ConsumerCoordinator
is in the "internals" package and is subject to API changes without notice. Since API changes to ConsumerCoordinator
might require changes to the KIP's proposed ZookeeperConsumerConnector
running kafka-based coordination, this KIP instead opts for embedding the user-facing KafkaConsumer
. - Regarding the subtask of providing a global state for
ConsumerRebalanceListener
to preserve existing behavior, we had considered just instantiating an EKC per consumer thread id so that kafka-based coordination would solve the problem of mapping partitions to consumer threads for us instead of stitching together DescribeGroupsResponse
and zookeeper state. We ultimately went against this approach due to the added complexity of managing many EKCs. Another downside of this approach is that a standard partition assignment strategy using kafka-based coordination would give equal weight to a ZookeeperConsumerConnector
consumer thread and a KafkaConsumer
, causing an uneven partition ownership distribution across the group. - Merge the
/consumers/<group id>/ids
and /consumers/<group id>/migration/ids
directories by simply defining a new znode data version 2 for MEZKCCs. This was rejected to avoid any possibility of breaking clients as they parse the znode.