You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 15 Next »

Status

Current state: Under Discussion

Discussion thread: kafka-dev

JIRA: Unable to render Jira issues macro, execution error. , Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka 0.9.0.0 added SASL-based authentication for clients and inter-broker communication. SASL is a framework that enables authentication and data security via replaceable mechanisms. But at the moment, SASL implementation in Kafka supports only SASL/GSSAPI using Kerberos and does not allow other SASL mechanisms to be plugged in. Enabling other SASL mechanisms will allow better integration with existing non-Kerberos authentication servers. In this KIP, we discuss a proposal for enabling other SASL mechanisms.

This KIP addresses the following extensions to the existing implementation:

  1. Configurable SASL implementation to enable integration with existing authentication servers
  2. Support for additional SASL mechanisms. This enables standard mechanisms that do not have a default implementation in Kafka (eg. DIGEST-MD5) as well as custom mechanisms
  3. Configurable callback handlers to provide mechanism-specific input
  4. SASL/PLAIN implementation to enable simple username/password authentication without complex infrastructure
  5. Support for multiple SASL mechanisms within the same client or server. This will be useful, for example, in organizations where internal and external users require different authentication mechanisms.
  6. Configurable login interface that manages the login process and the lifecyle of login context related resources to support custom mechanisms that require periodic ticket refresh

Public Interfaces

Configuration options

The following options will be added to SaslConfigs.java and can be configured as properties for Kafka clients and server:

  1. sasl.mechanisms (List<String>) Specifies the list of SASL mechanisms that are enabled. This may include any mechanism for which a security provider is available in the JVM. Default value is GSSAPI.
  2. sasl.callback.handler.class  (Class). The fully qualified name of a class that implements the AuthCallbackHandler interface. This class should have a default constructor and implement the callback handlers required for the configured mechanisms. Default implementation supports callback handlers for GSSAPI and PLAIN.
  3. sasl.login.class  (Class). The fully qualified name of a class that implements the Login interface. This class should have a default constructor.

Callback handler interface

AuthCallbackHandler
package org.apache.kafka.common.security.auth;
import org.apache.kafka.common.annotation.InterfaceStability;
import org.apache.kafka.common.KafkaException;
import javax.security.auth.callback.CallbackHandler;
/*
 * Callback handler for SASL-based authentication
 */
@InterfaceStability.Unstable
public interface AuthCallbackHandler extends CallbackHandler {
    
    /**
     * Configures this callback handler.
     * 
     * @param configs Configuration
     * @param subject Subject from login context
     * @param saslMechanism Negotiated SASL mechanism
     */
    void configure(Map<String, ?> configs, Subject subject, String saslMechanism) throws KafkaException;
    /**
     * Closes this instance.
     */
    void close() throws KafkaException;
}

Login interface

Login
package org.apache.kafka.common.security.auth;
import java.util.Map;
import javax.security.auth.Subject;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;
/**
 * Login interface for authentication
 */
public interface Login {    
    /**
     * Configures this login instance.
     */
    void configure(Map<String, ?> configs, String loginContextName);
    /**
     * Performs login for each login module specified for the login context of this instance.
     */
    LoginContext login() throws LoginException;    
    /**
     * Returns the authenticated subject of this login context.
     */
    Subject subject();    
    /**
     * Returns the service name
     */
    String serviceName();
    /**
     * Closes this instance.
     */
    void close();
}

 

Proposed Changes

The changes will be implemented under two JIRAs. KAFKA-3149 implements the changes to enable new mechanisms. KAFKA-2658 will be rebased after the review of this KIP to include a simple default implementation for SASL/PLAIN in Kafka. This section gives a summary of the changes and the rationale behind them.

SASL configuration

SASL in Kafka is configured using the standard JAAS configuration. SASL configuration consists of:

  1. LoginContext that specifies the login module class and properties for the login module, specified using JAAS configuration
  2. SASL mechanism and other properties specific to the mechanism configured as Kafka client or server properties. Additional properties and selection policies handled by the SASL implementation may also be specified when the SaslClient or SaslServer is constructed.
  3. Additional input required by the SASL implementation obtained using CallbackHandlers
  4. SaslServer or SaslClient implementation for the configured mechanism. These are installed as security providers in the JVM. For custom mechanisms, providers can be initialized in the static initializer of the login module to ensure that the providers are installed before Kafka creates the SaslServer/SaslClient.

All the four types of configuration above need to be configured consistently for SASL authentication to operate correctly. 1) and 4) are JVM configuration options. 2) and 3) are currently hard-coded in Kafka. The proposed changes enable flexible configuration for 2) and 3) to enable any SASL mechanism to be supported in Kafka clients and servers.

SASL mechanism

Kafka server and client will have a new configuration option sasl.mechanisms to specify the list of enabled mechanisms. In order to plug in any SASL mechanism including custom mechanisms, mechanisms will be specified as String rather than an enum with a restricted set of values.  GSSAPI will be used as the default mechanism without mechanism negotiation for interoperability of new clients with 0.9.0.0 brokers. If a list of mechanisms is specified, a mechanism is selected by the client after exchange of mechanisms supported in the server, as described later.

SASL callback handler

Kafka server and client will have a new configuration option sasl.callback.handler.class to provide a callback handler class. Default callbacks are included in Kafka for the mechanisms which have an implementation in Kafka (GSSAPI and PLAIN). Default client callback handler obtains authentication id and password from the public and private credentials of the Subject respectively as String values. Configurable callbacks will enable other mechanisms to be used with Kafka without any changes to Kafka code.

SASL properties

The current implementation does not specify any properties when the SaslClient or SaslServer is constructed. To make the Kafka implementation flexible with pluggable mechanisms, all properties specified for Kafka client/server will be passed to the SaslClient/SaslServer. These include all properties specified by the user including properties not defined in Kafka, so that additional properties can be added without changes to Kafka.

Login interface

Kafka server and client will have a new configuration option sasl.login.class to manage the login process. This is required to plugin custom mechanisms which require login tokens to be refreshed periodically. Default login implementation will support GSSAPI and PLAIN and will be suitable for most of the standard mechanisms where mechanism-specific code can reside in the login module implementation. This interface is targeted at protocols similar to Kerberos which require a background thread to handle ticket refresh.

Support for multiple mechanisms in a broker

Some organizations may have a requirement to use different authentication mechanisms within the same broker. For instance, this may be useful if internal and external clients connecting to the same broker use different authentication servers. Negotiation of SASL mechanisms is not included in the SASL protocol. Typically, application protocols which support multiple mechanisms include a mechanism negotiation phase where the server advertises the list of enabled mechanisms and the client selects one of the available mechanisms and sends the selected mechanism to the server. The current Kafka authentication protocol for SASL does not perform any negotiation of mechanisms since only GSSAPI is supported.

Since GSSAPI clients send a non-empty token to the broker when a connection is established, an empty packet is sent to the broker to indicate that mechanism negotiation is required. This enables new brokers to work with 0.9.0.0 clients which do not support mechanism exchange. Clients which do not specify the new configuration option sasl.mechanisms will use GSSAPI as the mechanism without negotiation to enable new clients to work with 0.9.0.0 brokers.

Client flow:

  1. If sasl.mechanisms contains any mechanism other than GSSAPI, send a packet with empty payload (just the 4-byte size field set to zero) to the server. Otherwise go to Step 4 with GSSAPI as the selected mechanism.
    • Packet Format: |Empty|
  2. Wait for list of SASL mechanisms from the server
  3. Select a mechanism from the mechanisms enabled in the server and send the selected mechanism to the server
    • Packet Format: | Version (Int16) | Mechanism (String) |
  4. Create SaslClient with the selected mechanism
  5. Perform SASL authentication with the selected mechanism

Server flow:

  1. Wait for first authentication packet from client
  2. If client payload size is not zero, skip mechanism negotiation and go to Step 5 and process this packet as the first GSSAPI client token
  3. Send a list of enabled mechanisms to the client
    • Packet Format: | Version (Int16) | EnabledMechanisms (ArrayOf(String)) |
  4. Wait for selected mechanism from client
  5. Create SaslServer with the selected mechanism
  6. Perform SASL authentication. If mechanism negotiation was skipped, process the initial packet that was received from the client first.

SASL/PLAIN implementation

SASL/PLAIN is a simple username/password authentication mechanism that is typically used with TLS for encryption to implement secure authentication. Unlike Kerberos, PLAIN does not require complex authentication infrastructure. Adding a default implementation for PLAIN in Kafka enables a simpler authentication mechanism for organizations which do not already use Kerberos. SASL/PLAIN protocol and its uses are described in https://tools.ietf.org/html/rfc4616..
The PR in KAFKA-2658 will be rebased on the extensible interface from this KIP for this implementation.

Testing

  • Due to the complexity of setting up Kerberos, limited unit testing has been implemented for SASL in the clients project. Along with the implementation for SASL/PLAIN, comprehensive unit tests will be added for the existing SASL implementation as well as the new interfaces.
  • End-to-end tests will be added in the core project along with the existing SASL/Kerberos tests for SASL/PLAIN, multi-mechanism configuration and to test an additional mechanism using the extension points.
  • System tests will be added for SASL/PLAIN and for multi-mechanism support.

Compatibility, Deprecation, and Migration Plan

Impact on existing clients

Existing clients will continue to use GSSAPI as the SASL mechanism and will not be impacted by the changes. Since default callback handlers can be used for SASL mechanisms that are implemented in Kafka, no configuration changes are required.

Rolling upgrade from 0.9.0.0

Rolling upgrade with GSSAPI as the SASL mechanism can be performed using a simple rolling restart with no change in properties. By default, if sasl.mechanisms property is not specified, GSSAPI will be used without any negotiation of mechanisms. If the mechanism is to be changed, this rolling restart can be followed by the addition of the new mechanism as described below.

Rolling upgrade with change in SASL mechanism

SASL mechanism can be modified with rolling restart using the following sequence:

  1. Add the new mechanism to the list of mechanisms enabled in the broker and add the configuration properties required for the new mechanism. Perform rolling restart of the brokers.
  2. Restart all clients with new SASL mechanism
  3. Disable the old mechanism in the broker if necessary

Rejected Alternatives

Enable a small set of SASL mechanisms with a default implementation in Kafka

Since the security requirements and infrastructure used in different organizations vary, default implementation of login modules and security providers and unlikely to be sufficient for all users. Unlike Kerberos, where most users are likely to use the Kerberos module provided in the JDK, other mechanisms are likely to be customized by users to enable integration with existing authentication providers. The proposed implementation removes the restriction that a SASL mechanism without a default implementation in Kafka cannot be used at all.

Make Authenticator configurable

This would provide additional flexibility, but would require users to implement more code. This would be more suitable if there is a requirement to implement authentication using protocols other than SASL.

Support multiple SASL mechanisms within a Kafka broker as new security protocols

Implementing multi-mechanism SASL as a new authentication protocol will simplify mechanism negotiation, leaving the existing protocol purely for GSSAPI. Two new security protocols SASL2_PLAINTEXT and SASL2_SSL can be added to specify the new SASL protocol that includes mechanism negotiation. But this would involve maintaining and testing two variants of SASL.

Support multiple SASL mechanisms within a Kafka broker on different ports

To avoid mechanism negotiation altogether, each SASL mechanism could be defined on a different port. With the current endpoint definitions, this would require each combination of transport layer and SASL mechanism to be defined as a new security protocol. This makes it harder to introduce new mechanisms without changing Kafka code. To define custom SASL protocols, the current security protocol enumeration needs to be made extensible as well.

 

 


  • No labels