The following is a draft design that uses a high-available consumer coordinator at the broker side to handle consumer rebalance. By migrating the rebalance logic from the consumer to the coordinator we can resolve the consumer split brain problem and help thinner the consumer client. With this design we hope to incorporates:
KAFKA-167: Move partition assignment to the broker
KAFKA-364: Add ability to disable rebalancing in ZooKeeper consumer
And solves:
1. Consumer Scalability Goal
...