Page History

...

After Kafka moved the rebalancing responsibilities from brokers to clients, many other applications (Kafka Connect, Kafka Streams, Confluent Schema Registry) have relied on the Group Membership Protocol to implement resource allocation amongst distributed processesto implement resource allocation amongst distributed processes.

Kafka Connect's WorkerCoordinator class extends AbstractCoordinator, and uses the rebalance mechanism to distribute tasks to its workers. In the same spirit, Confluent Kafka Registry - and its SchemaRegistryCoordinator - relies in the AbstractCoordinator class for leadership election (which instance of a Schema Registry cluster can accept writes). These two are in addition to the ConsumerCoordinator that the Kafka Consumer uses internally, and indirectly Kafka Streams.

We've adopted this generic, extensible protocol to implement a framework that solves both the distributed resource management and leader election use cases our engineering teams face when building their systems. This framework is built on top of the AbstractCoordinator class, which exposes very powerful primitives that can be leveraged to build  distributed The powerful primitives exposed by the AbstractCoordinator class have made it relatively easy to build distributed resource management systems on top of Apache Kafka.

We think it's time for the AbstractCoordinator to have a companion public interface that is become part of Kafka's public API, so we can ensure backwards compatibility in future versions of the client libraries. This idea was mentioned on addressed by Gwen Shapira 's on her  talk at the strangeloop conference in 2018 '18 [https://www.youtube.com/watch?v=MmLezWRI3Ys]. Finally, advertising the Coordinator as one of the features Apache Kafka offers should be another goal, as the protocol is well designed and extensible, so it's applicable to many other use cases.

...