...

The following options will be added to SaslConfigs.java and can be configured as properties for Kafka clients and server:

1. sasl.mechanism (List<String>) Specifies the list of SASL mechanisms that are enabled. This may include any mechanism for which a security provider is available in the JVM. Default value is GSSAPI.
2. sasl.callback.handler.class  (Class). The fully qualified name of a class that implements the AuthCallbackHandler interface. This class should have a default constructor and implement the callback handlers required for the configured mechanisms. Default implementation supports callback handlers for GSSAPI and PLAIN.
3. sasl.login.class  (Class). The fully qualified name of a class that implements the Login interface. This class should have a default constructor.

...

The changes will be implemented under two JIRAs. KAFKA-3149 implements the changes to enable new mechanisms. KAFKA-2658 will be rebased after the review of this KIP to include a simple default implementation for SASL/PLAIN in Kafka. This section gives a summary of the changes and the rationale behind them.

...

1. LoginContext that specifies the login module class and properties for the login module, specified using JAAS configuration
2. SASL mechanism and other properties specific to the mechanism configured as Kafka client or server properties. Additional properties and selection policies handled by the SASL implementation may also be specified when the SaslClient or SaslServer is constructed.
3. Additional input required by the SASL implementation obtained using CallbackHandlers
   
4. SaslServer or SaslClient implementation for the configured mechanism. These are installed as security providers in the JVM. For custom mechanisms, providers can be initialized in the static initializer of the login module to ensure that the providers are installed before Kafka creates the SaslServer/SaslClient.

All the four types of configuration above need to be configured consistently for SASL authentication to operate correctly. 1) and 4) are JVM configuration options. 2) and 3) are currently hard-coded in Kafka. The proposed changes enable flexible configuration for 2) and 3) to enable any SASL mechanism to be supported in Kafka clients and servers.

...

Kafka server and client will have a new configuration option sasl.mechanisms to specify the list of enabled mechanisms. In order to plug in any SASL mechanism including custom mechanisms, mechanisms will be a list of String options rather than an enum with a restricted set of values.  GSSAPI will be used as the default mechanism without mechanism negotiation for interoperability of new clients with 0.9.0.0 brokers. If a list of mechanisms are is specified, one a mechanism is selected by the client after exchange of mechanisms supported mechanismsin the server, as described later.

SASL callback handler

...

Some organizations may have a requirement to use different authentication mechanisms within the same broker. For instance, this may be useful if internal and external clients connecting to the same broker use different authentication servers. Negotiation of SASL mechanisms is not included within in the SASL protocol. Typically, application protocols which support multiple mechanisms include a mechanism negotiation phase where the server advertises the list of enabled mechanisms and the client selects one of the available mechanisms and sends the selected mechanism to the server. The current Kafka authentication protocol for SASL does not perform any negotiation of mechanisms since only GSSAPI is supported.

Since GSSAPI clients send a non-empty token to the broker when a connection is established, an empty packet is sent to the broker to indicate that mechanism negotiation is required. Clients which do not specify the new configuration option sasl.mechanisms will use GSSAPI as the mechanism without negotiation to enable new clients to work with 0.9.0.0 brokers.

Client flow:

1. If sasl.mechanisms is specified as a non-empty list, send a packet with empty payload (just the 4-byte size field set to zero) to the server. Otherwise go to Step 4 with GSSAPI as the selected mechanism.
   • Packet Format: |Empty|
2. Wait for list of SASL mechanisms from the server
3. Select a mechanism from the mechanisms enabled in the server and send the selected mechanism to the server
   • Packet Format: | Version (Int16) | Mechanism (String) |
4. Create SaslClient with the selected mechanism
5. Perform SASL authentication with the selected mechanism

Server flow:

1. Wait for first authentication packet from client
2. If client payload size is not zero, skip mechanism negotiation and go to Step 5 and process this packet as the first GSSAPI client token
3. Send a list of enabled mechanisms to the client
   
   • Packet Format: | Version (Int16) | EnabledMechanisms (ArrayOf(String)) |
4. Wait for selected mechanism from client
5. Create SaslServer with the selected mechanism
6. Perform SASL authentication. If mechanism negotiation was skipped, process the initial packet that was received from the client first.
   
   

SASL/PLAIN implementation

SASL/PLAIN is a simple username/password authentication mechanism that is typically used with TLS for encryption to implement secure authentication. Unlike Kerberos, PLAIN does not require complex authentication infrastructure. Adding a default implementation for PLAIN in Kafka enables a simpler authentication mechanism for organizations which do not already use Kerberos. SASL/PLAIN protocol and its uses are described in https://tools.ietf.org/html/rfc4616..
The PR in KAFKA-2658 will be rebased on the extensible interface from this KIP for this implementation.

...

• Due to the complexity of setting up Kerberos, limited unit testing has been implemented for SASL in the clients project. Along with the implementation for SASL/PLAIN, comprehensive unit tests will be added for the existing SASL implementation as well as the new interfaces.
• End-to-end tests will be added in the core project along with the existing SASL/Kerberos tests for SASL/PLAIN, multi-mechanism configuration and to test an additional mechanism using the extension points.
  
• System tests will be added for SASL/PLAIN and for multi-mechanism support.
  

Compatibility, Deprecation, and Migration Plan

...

SASL mechanism can be modified with rolling restart using the following sequence:

1. Enable Add the new mechanism to the list of mechanisms enabled in the broker using rolling restart with change in and add the configuration properties required for the new mechanism. Perform rolling restart of the brokers.
2. Restart all clients with new SASL mechanism
3. Disable the old mechanism in the broker if necessary

...

This would provide additional flexibility, but would require users to implement more code. This would be more suitable if there is a requirement to implement authentication using protocols other than SASL.

Support multiple SASL mechanisms within a Kafka broker as

...

new security

...

protocols

Implementing multi-mechanism SASL as a new authentication protocol will simplify mechanism negotiation and rolling upgrade, leaving the existing protocol purely for GSSAPI. Two new security protocols SASL2_PLAINTEXT and SASL2_SSL can be added to specify the new SASL protocol including that includes mechanism negotiation. But this would involve maintaining and testing two variants of SASL.

...