...

As such, we propose that Kafka brokers should be able to define multiple listeners for the same security protocol for binding (i.e. listeners) and sharing (i.e. advertised.listeners) so that internal, external and replication traffic can be separated if required.

Public Interfaces

A new broker config listener.security.protocol.map will be introduced so that we can map a protocol label to a security protocol. The config value should be in the CSV Map format that is currently used by the max.connections.per.ip.overrides config. The config value should follow map semantics: each key should only appear once, but values may appear multiple times. For example, the config could be defined in the following way to match the existing behaviour:

...

To ensure compatibility with existing configs, we propose the above as the default value for the new config. It's worth mentioning that the config value should be the same in every broker in the Kafka cluster for it to work as expected. This is also the case for a number of existing Kafka broker configs and since Kafka doesn't support cluster configs at this point, it seems acceptable.

The next step is to change the validation of advertised.listeners and listeners so that the protocol label has to be one of the keys in listener.security.protocol.map (only security protocols are allowed currently). For example, the following would configure a broker with two different host:port pairs mapped to the same security protocol in two cases:

...

It is an error to set both configs at security.inter.broker.protocol and inter.broker.protocol.label at the same time and . inter.broker.protocol.label will be null by default. This , which means that PLAINTEXT will be used by default (as is currently the case).

...

{
  "version":4,
  "jmx_port":9999,
  "timestamp":2233345666,
  "host":"localhost",
  “port”:9092,
  "rack":"rack1",
  "endpoints": [
    "CLIENT://cluster1.foo.com:9092",
    “REPLICATION://broker1.replication.local:9093”,
    “INTERNAL_PLAINTEXT://broker1.local:9094”,
	"INTERNAL_SASL://broker1.local:9095"
  ]    
}

The second and final one interface change is to the UpdateMetadataRequest protocol type. Version 2 would have a protocol_label field instead of security_protocol_type:

...