...

1. Configurable SASL implementation to enable integration with existing authentication servers
2. Support for additional SASL mechanisms. This enables standard mechanisms that do not have a default implementation in Kafka (eg. DIGEST-MD5) as well as custom mechanisms
3. Configurable callback handlers to provide mechanism-specific input
4. SASL/PLAIN implementation to enable simple username/password authentication
5. Support for multiple SASL mechanisms within the same client or server. This will be useful, for example, in organizations where internal and external users require different authentication mechanisms.
6. Configurable login interface that manages the login process and the lifecyle of login context related resources.

Public Interfaces

Configuration options

The following options will be added to SaslConfigs.java and can be configured as properties for Kafka clients and server:

1. sasl.mechanism (StringList<String>) Specifies the SASL mechanismlist of SASL mechanisms that are enabled. This may be include any mechanism for which a security provider is available in the JVM. Default value is GSSAPI.
2. sasl.callback.handler.class  (Class). The fully qualified name of a class that implements the AuthCallbackHandler interface. This class should have a default constructor and implement the callback handlers required for the configured mechanism. Default implementation supports callback handlers for GSSAPI and PLAIN.
3. sasl.login.class  (Class). The fully qualified name of a class that implements the Login interface. This class should have a default constructor.

Callback handler interface

package org.apache.kafka.common.security.auth;
import org.apache.kafka.common.annotation.InterfaceStability;
import org.apache.kafka.common.Configurable;
import org.apache.kafka.common.KafkaException;
import javax.security.auth.callback.CallbackHandler;
/*
 * Callback handler for SASL-based authentication
 */
@InterfaceStability.Unstable
public interface AuthCallbackHandler extends CallbackHandler, Configurable { {
    
    /**
     * Configures this callback handler.
     * 
     * @param configs Configuration
     * @param subject Subject from login context
     * @param saslMechanism Negotiated SASL mechanism
     */
    void configure(Map<String, ?> configs, Subject subject, String saslMechanism) throws KafkaException;
    /**
     * Closes this instance.
     */
    void close() throws KafkaException;
}

...

Login interface

package org.apache.kafka.common.security.auth;
import java.util.Map;
import javax.security.auth.Subject;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;
/**
 * Login interface for authentication
 */
public interface Login {    
    /**
     * Configures this login instance.
     */
    void configure(Map<String, ?> configs, String loginContextName);
    /**
     * Performs login for each login module specified for the login context of this instance.
     */
    LoginContext login() throws LoginException;    
    /**
     * Returns the authenticated subject of this login context.
     */
    Subject subject();    
    /**
     * Returns the service name to be used for Sas
     */
    String serviceName();
    /**
     * Closes this instance.
     */
    void close();
}

Proposed Changes

The changes will be implemented under two JIRAs. KAFKA-xxx 3149 implements the changes to enable new mechanisms. KAFKA-2658 will be rebased after the review of this KIP to include a simple default implementation for SASL/PLAIN in Kafka. This section gives a summary of the changes and the rationale behind them.

...

Kafka server and client will have a new configuration option sasl.mechanismmechanisms to specify the mechanismlist of enabled mechanisms. In order to plug in any SASL mechanism including custom mechanisms, this property mechanisms will be a list of String parameter options rather than an enum with a restricted set of values.  Mechanism negotiation will be initiated by the client to enable support for multiple mechanisms within a broker without impacting existing 0.9.0.0 which use the default GSSAPI mechanism without negotiation.

SASL callback handler

Kafka server and client will have a new configuration option sasl.callback.handler.class to provide a callback handler class. Default callbacks are included in Kafka for the mechanisms which have an implementation in Kafka (GSSAPI and PLAIN). Default client callback handler obtains authentication id and password from the public and private credentials of the Subject respectively as String values. Configurable callbacks will enable other mechanisms to be used with Kafka without any changes to Kafka code.

...

The current implementation does not specify any properties when the SaslClient or SaslServer is constructed. To make the Kafka implementation flexible with pluggable mechanisms, all properties specified for Kafka client/server will be passed to the SaslClient/SaslServer. These include all properties specified by the user including properties not defined in Kafka, so that additional properties can be added without changes to Kafka.

Login interface

Kafka server and client will have a new configuration option sasl.login.class to manage the login process. This is required to plugin custom mechanisms which require login tokens to be refreshed periodically. Default login implementation will support GSSAPI and PLAIN and will be suitable for most of the standard mechanisms where mechanism-specific code can reside in the login module implementation. This interface is targeted at protocols like Kerberos which require a background thread to handle token refresh.

Support for multiple mechanisms in a broker

Some organizations may have a requirement to use different authentication mechanisms within the same broker. For instance, this may be useful if internal and external clients connecting to the same broker use different authentication servers. Negotiation of SASL mechanisms is not included within the SASL protocol. Typically, application protocols which support multiple mechanisms include a mechanism negotiation phase where the server sends the list of enabled mechanisms to the client and the client selects one of the available mechanisms and responds to the server. The current Kafka authentication protocol for SASL does not perform any negotiation of mechanisms since only GSSAPI is supported. To enable negotiation of mechanisms without breaking existing 0.9.0.0 clients, the following flow will be used:

1. Client sends a packet with empty payload (4-byte size set to zero). This avoids conflict with existing 0.9.0.0 clients which send the first non-empty client token for GSSAPI without mechanism negotiation.
   • Packet Format: |EMPTY|
2. Server responds with enabled mechanisms on seeing the empty packet. If the first packet is non-empty, server assumes GSSAPI protocol and omits the mechanism negotiation phase (0.9.0.0 client starts at Step 4, server moves to Step 6 with the first client token to process)
   • Packet Format: | Version (Int16) | EnabledMechanisms (ArrayOf(String)) |
3. Client chooses a mechanism from the mechanisms enabled in the server and sends the selected mechanism to the server
   • Packet Format: | Version (Int16) | Mechanism (String) |
4. Client creates SaslClient with the selected mechanism and starts SASL authentication with the selected mechanism
5. Server receives selected mechanism from client
6. Server creates SaslServer with the selected mechanism and starts SASL authentication

SASL/PLAIN implementation

...

• Due to the complexity of setting up Kerberos, limited unit testing has been implemented for SASL in the clients project. Along with the implementation for SASL/PLAIN, comprehensive unit tests will be added for the existing SASL implementation as well as the new interfaces.
• End-to-end tests will be added in the core project along with the existing SASL/Kerberos tests for SASL/PLAIN and to test an additional mechanism using the extension points.
  
• System tests will be added for SASL/PLAIN.

...

Existing clients will continue to use GSSAPI as the SASL mechanism and will not be impacted by the changes. Since default callback handlers can be used for SASL mechanisms that are implemented in Kafka, no configuration changes are required.

Rolling upgrade with change in SASL mechanism

Since this KIP does not address support for multiple SASL mechanisms within a broker, rolling upgrade to switch SASL mechanisms needs to be done in two stages. This involves a switch from SASL to non-SASL, followed another switch from non-SASL to SASL with the new mechanism.

Mechanism negotiation will be initiated by the client with a packet with empty payload. This avoids conflict with existing 0.9.0.0 clients which send the first non-empty client token for GSSAPI without mechanism negotiation. This ensures interoperability between 0.9.0.0 clients or broker with the updated protocol.

Rolling upgrade with change in SASL mechanism

SASL mechanism can be modified with rolling restart using the following sequence:

1. Enable the new mechanism in the broker using rolling restart with change in configuration properties required for the new mechanism
2. Restart all clients with new SASL mechanism
3. Disable the old mechanism in the broker if necessary
4. In the first stage, perform rolling restart of Kafka brokers to enable a non-SASL port (eg. SSL).
   1. Enable SSL port for the brokers and perform rolling restart of all brokers
   2. Restart all clients with SSL as protocol
   3. If using SASL for inter-broker communication, switch to SSL and perform another rolling restart of brokers.
5. In the second stage when SASL is no longer in use, perform rolling restart of Kafka brokers to update SASL mechanism with the updated JAAS configuration and properties.
6. Update SASL mechanism, JAAS configuration and other properties for the brokers and perform rolling restart of all brokers
7. Restart all clients with SASL_SSL/SASL_PLAINTEXT as required
If using SASL for inter-broker communication, switch to SASL_SSL/SASL_PLAINTEXT and perform rolling restart of brokers again.

Rejected Alternatives

Enable a small set of SASL mechanisms with a default implementation in Kafka

...

This would provide additional flexibility, but would require users to implement more code. This would be more suitable if there is a requirement to implement authentication using protocols other than SASL.

Support multiple SASL mechanisms within a Kafka broker

...

Typically SASL implementations negotiate SASL mechanism before the SASL authentication exchange. The server sends a list of mechanisms that it supports to the client when a connection is established and the client chooses one of those mechanisms that satisfies its requirements. The current wire protocol used by Kafka for SASL does not support negotiation of mechanisms. Since the flow that follows transport-layer handshake is mechanism-specific data and there is currently no versioning of authentication flows, introduction of mechanism negotiation will be a breaking change. If we anticipate that multi-mechanism support may be required in future, it may be better to include it now since it might be possible to design a flow that doesn't conflict with the current GSSAPI flow, enabling the choosing of mechanisms without breaking existing clients. The changes required to support this are

...

on different ports

This would require different SASL mechanisms to be defined as different security protocols to work with the current endpoint definitions. While this approach avoids the protocol changes required for negotiation of mechanisms, it makes it harder to add new mechanisms with every combination of transport layer protocol and SASL mechanism turning into a new security protocol. Negotiation of mechanisms makes the code slightly more complex, but provides a neater and more flexible external interface which is consistent with the way mechanisms are chosen in other SASL-enabled protocols.

...