Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

During the 0.9.0.0 release cycle, support for multiple listeners per broker was introduced. Each listener is associated with a security protocol, ip/host and port. When combined with the advertised listeners mechanism, there is a fair amount of flexibility with one limitation: at most one listener per security protocol in each of the two configs (listeners and advertised.listeners).

...

As such, we propose that Kafka brokers should be able to define multiple listeners for the same security protocol for binding (i.e. listeners) and sharing (i.e. advertised.listeners) so that internal, external and replication traffic can be separated if required.

Public Interfaces

A new broker config listener.security.protocol.map will be introduced so that we can map a protocol label to a security protocol. The config value should be in the CSV Map format that is currently used by max.connections.per.ip.overrides. The config value should follow map semantics: each key should only appear once, but values may appear multiple times. For example, the config could be defined in the following way to match the existing behaviour:

...

Note that protocol labels only exist in the brokers, clients never see them.

Proposed Changes

We would have to change a number of places in the code that currently use SecurityProtocol as a key to use SecurityLabel instead. A few examples:

...

As stated previously, clients never see protocol labels and will make metadata requests exactly as before. The difference is that the list of endpoints they get back is restricted to the protocol label of the endpoint where they made the request. In the example above, if a client makes a metadata request to cluster1.foo.com:9092, it will be mapped to the 192.1.1.8:9092 interface. The broker will return SASL_PLAINTEXT://cluster1.foo.com:9092 in the metadata response. It returned the entry in advertised.listeners with the same protocol label, but it translated the protocol label to the associated security protocol before returning it to the client.

Compatibility, Deprecation, and Migration Plan

As mentioned previously, the default value of listener.security.protocol.map maps the existing security protocols to a label with the same name:

...

For users upgrading, they should only use protocol labels once all the brokers have been upgraded.

Future work

1. Different security protocol settings per listener. For example, one may want to configure SSL differently for internal versus external traffic.

Rejected Alternatives

1. Using hard-coded listener domains for internal and replication traffic. The config format is simpler and there's less scope for hard to understand configs. The main disadvantage is that it's a bit too specific and may need to be extended again as more sophisticated use cases appear. The current proposal is more general and it seems like a natural evolution of the existing system.

...