Motivation

Summary of existing controller

Current Kafka controller is a multi-threaded controller that emulates a state machine. It works in the following model.

Maintained state:

1. Partitions replicas on each machine.
2. Leaders of partitions.

State change input source:

• Listeners Registered to Zookeeper.
  1. AddPartitionsListener
  2. BrokerChangeListener
  3. DeleteTopicListener
  4. PartitionReassignedListener(Admin)
  5. PreferredReplicaElectionListener(Admin)
  6. ReassignedPartitionsIsrChangeListener
  7. TopicChangeListener
• Channels to brokers (controlled shutdown)
• Internal scheduled tasks (preferred leader election)

State change propagate model:

• P2P blocking channel from controller to each broker.
• Dedicated message queue for each controller-to-broker connection.
• No synchronization on message sent to different brokers.
• No callback for sending messages except topic deletion.

Fail Over/Back model:

• Zookeeper based leader election
• Zookeeper as persistent state store for fault tolerance.

Problems of existing controller

1. State change are executed by different listeners concurrently. Hence complicated synchronization is needed which is error prone and difficult to debug.
2. State propagation is not synchronized. Brokers might be in different state for undetermined time. This leads to unnecessary extra data loss.

New controller design

Design Principle

1. Abstract the output of each state change input source to an event.
2. Have a single execution thread to serially process events one at a time.
3. The zk listeners are responsible for only context updating but not event execution.
4. Use o.a.k.clients.NetworClient + callback for state change propagation.

Architecture