Status

Current state: Under DiscussionAdopted

Discussion thread: here

JIRA: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

{
	"version": 4,
	"jmx_port": 9999,
	"timestamp": 2233345666,
	"host": "localhost",
	"port": 9092,
	"rack": "rack1",
	"listener._security._protocol._map": {
		"PLAINTEXT": "PLAINTEXT",
		"SSL": "SSL",
		"SASL_PLAINTEXT": "SASL_PLAINTEXT",
		"SASL_SSL": "SASL_SSL"
	},
	"endpoints": [
		"CLIENT://cluster1.foo.com:9092",
		"REPLICATION: //broker1.replication.local:9093",
		"INTERNAL_PLAINTEXT: //broker1.local:9094",
		"INTERNAL_SASL://broker1.local:9095"
	]
}

Protocol

Version 2 3 of UpdateMetadataRequest will be introduced and the elements of the end_points array would also have a listener_name field.

UpdateMetadata Request (Version: 23) => controller_id controller_epoch [partition_states] [live_brokers] 
  controller_id => INT32
  controller_epoch => INT32
  partition_states => topic partition controller_epoch leader leader_epoch [isr] zk_version [replicas] 
    topic => STRING
    partition => INT32
    controller_epoch => INT32
    leader => INT32
    leader_epoch => INT32
    isr => INT32
    zk_version => INT32
    replicas => INT32
  live_brokers => id [end_points]
    id => INT32
    end_points => port host listener_name (new) security_protocol_type
      port => INT32
      host => STRING
      listener_name => String (new)
      security_protocol_type => INT16

...

The exception is ZooKeeper-based consumers. These consumers retrieve the broker registration information directly from ZooKeeper directly and would have to be updated to map from the listener name to the security protocoland will choose the first listener with PLAINTEXT as the security protocol (the only security protocol they support).

Compatibility, Deprecation, and Migration Plan

...

For users upgrading, they should only use listener names once all the brokers and ZooKeeper-based consumers have been upgraded to a version that supports listener names. ZooKeeper-based consumers will use the first listener with PLAINTEXT as the security protocol, so listener ordering is important in such cases.

Rejected Alternatives

1. Instead of adding the listener.security.protocol.map config, we could extend the protocol part of the listener definition to include both the listener name and security protocol. For example, CLIENT+SASL_PLAINTEXT://192.1.1.8:9092. This is appealing from a clarity perspective (the listeners are fully defined in a single config value), but it may lead to duplication between listeners and advertised.listeners. A way to avoid that issue (at the cost of loss of symmetry) would be for advertised.listeners to only include the listener name (we can infer the security protocol by looking at the listeners entry with the same name).
2. Assume that listener.security.protocol.map is the same in every broker. The slight benefit in terms of smaller broker registration JSON is not worth the additional operational complexity when it comes to changing the config values in a running cluster (two rolling upgrades would be needed in some simple cases).
3. Using hard-coded listener domains for internal and replication traffic. The config format is simpler and there's less scope for hard to understand configs. The main disadvantage is that it's a bit too specific and may need to be extended again as more sophisticated use cases appear. The current proposal is more general and it seems like a natural evolution of the existing system.

...