Page History

...

We’ll also add a BrokerEndPoint object, which will represent a method of connecting to a broker. BrokerEndPoint contains id, host , and port and protocol. This doesn’t get persisted. Components that need to connect to a broker will get a BrokerEndPoint from the broker based on the security protocol they wish to use. Basically, BrokerEndPoint replaces the old Broker in most protocols.

All these changes mean that we need to serialize and de-serialize Broker differently (to both ZK and wire) and we need to serialize / de-serialize BrokerEndPoints to wire.

...

1. ConsumerConfig: Adding a configuration to specify security.protocol (defaulting to plaintext). The new consumer (and producers) will just need to specify port, but the existing consumer takes the brokers directly from ZK and we’ll need to know which of the broker ports to use.

2. KafkaConfig: Instead of specifying port, advertised port and advertised host, we want to specify host-port-lists: comma separated pairs of protocol, host and port (ssl://192.1.1.8:9093, plaintext://10.1.1.5:9092).
   We also need to add replication.security.protocol configuration controlling which port and protocol the broker will use to connect to other brokers.

...

ConsumerMetadataResponse and TopicMetadataResponse return a list of Brokers that the consumer and producer can connect to. We will change these to return BrokerEndPoints instead, based on the security protocol the client is using .

This means that the matching Requests will need to include some information about the security protocol the client is using.

(determined by the port the client connected to). Note that this means the wire protocol does not change.

UpdateMetadataRequest - is used between brokers and needs to contain the entire broker information. We bump the protocol version of this request and make sure that the new broker can accept both versions of requests, and will only send the new request version if a configuration was modified to useNewVersion=true

LeaderAndIsrRequest - will use BrokerEndPoint instead of broker (based on replication protocol specified in configs). Wire protocol will not changeIt also means that we need to “bump up” the protocol version of those Requests and Responses, and make sure the handlers can handle both.

Tools that need to support multiple ports:

...