...

• The following client libraries will change:

  • We are adding org.apache.kafka.common.protocol.SecurityProtocol - an ENUM describing the possible security protocols the broker may support and the client may try to connect to.
    
    

• Quite a few changes from user configuration perspective:
  • Producers and consumers need to specify security.protocol - which protocol they'll use to connect to the server. Current valid values are PLAINTEXT and TRACE (for testing). Future values should include TLS and SASL.
  • Brokers that wish to support multiple protocols and multiple host/port pairs will need to specify a new configuration parameter: "listeners" and/or "advertised.listeners" - replacing host/port and advertised.host/advertised.port. Listener format is a comma-separated list of "protocol://host:port
  • Brokers need to specify replication.security.protocol - which protocol / port will be used to replicate data between brokers? this must be identical in all brokers.
  • Following an upgrade, brokers need to specify use.new.protocol = true, to start using the new wire protocol and to support new security protocols.
    
    

• Broker registration in Zookeeper will change to include all host/port pairs the broker supports. The new json is:

  {"version":2,
    "jmx_port":9999,
    "timestamp":"2233345666",
    "endpoints": [
      {"host":"myhost",
       "port":9092,
       "protocolType":"plain"},
     {"host":"myhost",
      "port":9093,
      "protocolType":"ssl"}]}

   

• The proposed client-server workflow is as follows:
  • Clients (producers and consumers) need to know about the broker ports in advance. They don't need to know about all brokers, but they need to know at least one host:port pair that speaks the protocol they
    want to use.
    The change from current behavior is that all host:port pairs in broker.list must be of the same protocol and match the security.protocol configuration parameter.
  • Client uses security.protocol configuration parameter to open a connection to one of the brokers and sends the good old MetadataRequest.
    The broker knows which port it got the connection on, therefore it knows which security protocol is expected (it needs to use the same protocol to accept the connection and respond), and therefore it can send a response that contains only the host:port pairs that are relevant to that protocol.
  • From the client side the MetadataResponse did not change - it contains a list of brokerId,host,port that the client can connect to.
    The fact that all those broker endpoints were chosen out of a larger collection to match the right protocol is irrelevant for the client.

Proposed Changes

See detailed description here: Multiple Listeners for Kafka Brokers

Compatibility, Deprecation, and Migration Plan

...