Status

Current state: DISCUSS ACCEPTED

Voting thread: https://www.mail-archive.com/dev@kafka.apache.org/msg104580.html

Discussion thread: https://www.mail-archive.com/dev@kafka.apache.org/msg101011.html

JIRA:

PR: https://github.com/apache/kafka/pull/8338

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

This KIP proposes the work on top of what has been done already for SSL configuration like KIP-226 - Dynamic Broker Configuration and KIP-492: Add java security providers in Kafka Security config.

Our primary goal is: Instead of keep adding various ssl configurations to customize smaller portions of SSLContext/Engine, we should have a single configurable/pluggable SSLContext/Engine functionality.

Pluggable SSLContext or SSLEngine?

Java's Security Providers architecture already enables using custom SSLEngine by using the right JSSE compatible Provider like BouncyCastleJsseProvider. We could use any SSLEngine of the  TLS implementations as far as we have JSSE compatible provider available. We can use Kafka's 'security.providers' configuration to use a custom provider.

Making javax.net.ssl.SSLContext setup pluggable provides flexibility for providing Key Material, Secure Random implementation and configure Key/Trust managers in a custom way. Example: Apache HttpComponents and Netty SslContextBuilder. 

However, Kafka configures the SSLEngine for Client and Server both modes. Hence according to existing code it would be useful to make SslEngineBuilder pluggable. That will provide us a way to configure SSLContext object in a flexible way and at the same time will allow creation of SSLEngine with Client/Server mode.

Public Interfaces

New configuration

ssl.engine.factory.class - This configuration will take class of the below interface's type and will be used to create javax.net.ssl.SSLEngine object.

Default value will be as mentioned below.

...

    public static final String SSL_ENGINE_FACTORY_CLASS_CONFIG = "ssl.engine.factory.class";
    public static final String DEFAULT_SSL_ENGINE_FACTORY_CLASS = org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.class.getCanonicalName();
    public static final String SSL_ENGINE_FACTORY_CLASS_DOC = "The class of type org.apache.kafka.common.security.auth.SslEngineFactory to provide SSLEngine objects. Default value is " + DEFAULT_SSL_ENGINE_FACTORY_CLASS;

Interface for SslEngineFactory

Below is the interface suggested for this.

In the interface we provide a way to customize SSLContext and SSLEngine both because SSLContext ideally has all the data points to be customized (see JSSE reference diagram) but for Kafka SSLEngine needs to be further configured based on Client or Server Mode as it is done currently.

package org.apache.kafka.common.security.sslauth;

import org.apache.kafka.common.network.ModeConfigurable;

import javax.net.ssl.SSLContextSSLEngine;
import javax.net.ssl.SSLEnginejava.io.Closeable;
import java.security.KeyStore;
import java.util.Map;
import java.util.Set;

public interface SslEngineFactory {

	/**
     * Creates SSLContext by providing required/**
 * Plugin interface for allowing creation of SSLEngine object in a custom way.
 * Example: You want to use custom way to load your key material and {@code java.security.SecureRandom}
     *
     * @return	The SSLContext.
     */
	SSLContext createSSLContext();trust material needed for SSLContext.
 * However, keep in mind that this is complementary to the existing Java Security Provider's mechanism and not a competing
 * solution.
 */
public interface SslEngineFactory extends Configurable, Closeable {

    /**
     * CreatesCreate a new SSLEngine object.
 to be used by *the client.
     *
 @param mode   * @param peerHost Whether to use client or server mode.
     * @param peerHost  The peer host to use. This is used in client mode if endpoint validation is enabled.
     * @param peerPort               The peer port to use. This is a hint and not used for validation.
     * @param endpointIdentification Endpoint identification algorithm for client mode.
     * @return          The new SSLEngine.
     */
    SSLEngine createSSLEnginecreateClientSslEngine(Mode mode, String peerHost, int peerPort, String endpointIdentification);

    /**
     * Returns the currently used configurations by this engine Create a new SSLEngine object to be used by the server.
     *
     * @param peerHost               The peer host to use. This is used in client mode if endpoint validation is enabled.
     * @param peerPort               The peer port to use. This is a hint and not used for validation.
     * @return The new SSLEngine.
     */
    SSLEngine createServerSslEngine(String Map<StringpeerHost, Object>int currentConfigs(peerPort);

 	/**
   /**  * Returns true if SSLEngine needs to be rebuilt. This method will be called when reconfiguration is triggered on
     * {@link org.apache.kafka.common.security.ssl.SslFactory}. Based on the <i>nextConfigs</i>, this method will
     * Returns the reconfigurable configs used by this engine.decide whether underlying SSLEngine object needs to be rebuilt. If this method returns true, the
     * @return{@link org.apache.kafka.common.security.ssl.SslFactory} will re-create instance of this object and run other
     * checks before deciding to use the new object for the <i>new incoming connection</i> requests.The existing connections
     * are  Set<String> reconfigurableConfigs();
not impacted by this and will not see any changes done as part of reconfiguration.
     *
    / ** <pre>
     * Returns true if this engine needs to be rebuilt     Example: If the implementation depends on the file based key material it can check if the file is updated
     *     compared to the previous/last-loaded timestamp and return true.
     * </pre>
     *
     * @param nextConfigs       The configuration we want to use.
     * @return                  True only if the thisunderlying SSLEngine builderobject should be rebuilt.
     */
    boolean shouldBeRebuilt(Map<String, Object> nextConfigs);

    /**
     * Returns the names of configs that may be reconfigured.
     */
    Set<String> reconfigurableConfigs();

    /**
     * Returns keystore.
     * @return
     */
    KeyStore keystore();

    /**
     * Returns truststore.
     * @return
     */
    KeyStore truststore();
}

Proposed Changes

Currently SslFactory.java uses SslEngineBuilder.java. Instead of that we will modify SslFactory.java to load a class configured via the new configuration 'ssl.engine.factory.class' and delegate the SSLEngine creation call to the implementation. 

...

Default implementation would be along the lines (but not necessarily the same) of - sample code which is copied from current SslEngineBuilder.java.be very similar to the current SslEngineBuilder.java.

Which classes will be deleted?

• SslEngineBuilder.java (functionality will be moved to DefaultSslEngineFactory.java)

Which classes will be added?

• SslEngineFactory.java Interface
• DefaultSslEngineFactory.java (mostly having code from existing SslEngineBuilder)

Which classes will be modified primarily?

• SslFactory.java

How does configs get to the implementation class?

The configuration of Map will be passed to the implementation class via the configure() method. See below example,

public DefaultSslEngineFactory implements SslEngineFactory {
...
...
	/* Default empty argument constructor */

	/* implement configure() method */
    @Override
    public void configure(Map<String, ?> configs) {
    ...
    }
...
...
}

These configuration will be passed from SslFactory to the implementation of the SslEngineFactory interface via reflection like below

public class SslFactory implement Reconfigurable {
...
...
     private SslEngineFactory instantiateSslEngineFactory(Map<String, Object> configs) {
        @SuppressWarnings("unchecked")
        Class<? extends SslEngineFactory> sslEngineFactoryClass =
                (Class<? extends SslEngineFactory>) configs.get(SslConfigs.SSL_ENGINE_FACTORY_CLASS_CONFIG);
        SslEngineFactory sslEngineFactory = Utils.newInstance(sslEngineFactoryClass);
        sslEngineFactory.configure(configs);
        this.sslEngineFactoryConfig = configs;
        return sslEngineFactory;
    }
...
}

Support for reconfiguration of custom configs

By custom configs we mean the configs used by the SslEngineFactory's implementation. Those configs does not have to be part of definition of Kafka configs since only the implementation class knows what are those. Kafka already supports custom configs so this should not be a new challenge. 

Compatibility, Deprecation, and Migration Plan

...

Not applicable since old code behavior will be kept with default implementation of DefaultSslEngineFactory and modification to SslFactory class.

Rejected Alternatives

Make SSLEngine pluggable

As mentioned noted in the motivation there are/were several attempts to make various ssl configurations pluggable over time focusing on specific aspect of the SSL configuration. However, this KIP proposes to allow customization at SSLContext/SSLEgnine level hence there are no alternatives applicable in our opinionsection- Java's Security Providers architecture already enables using custom SSLEngine by using the right JSSE compatible Provider hence Kafka should not have to derive another way to make SSLEngine pluggable.

Making SslFactory the pluggable interface (KIP-383)

This is because currently SslFactory does certain validations which we want to keep separate and mandate those checks across any possible implementation of pluggable ssl context class. Also, once we start writing the reconfigurable classes we realize that we need two classes - 1) SslEngineFactory implementation and 2) Container of the factory implementation. We believe that keeping SslFactory as Reconfigurable object and help reconfigure the underlying SslEngineFactory will simplify the implementations of SslEngineFactory.

Also, we rejected to make SslEngineFactory extend the Reconfigurable interface due to following reason,

There will be good amount of state in the SslEngineFactory's implementation (as it will be similar to the current SslEngineBuilder class). We believe that making SSLContext creation and SSLEngine object's configuration pluggable is worth to allow SSL experts to write their own implementation having the SSL domain knowledge and keep them free of knowing much about Kafka's reconfigurability - example: Apache HttpComponents. We prefer SslFactory class to do what it is doing right now and keep the responsibility of re-creating underlying SslEngineFactory object based on the configurations specified by the SslContextFactory's implementation.

Creating builder for SSLContext

We could create a builder for SSLContext object and have a mechanism to configure SSLEngine object for client/server mode non-pluggable. However, creating a builder interface with options to build SSLContext will need to have method(s) to allow keys/trusted-certs. It will also require us to have 'key-password' as input for the keystore. In the current Kafka implementation it requires the password to be configured in the plaintext via 'ssl.key.password', 'ssl.keystore.password' and 'ssl.truststore.password'. If we need to customize how the password is loaded, due to security reasons, this approach will not work since some other mechanism for making password pluggable (See KIP-76 Enable getting password from executable rather than passing as plaintext in config files AND KIP-486: Support custom way to load KeyStore and TrustStore) need to be devised which will add more ssl related configurations to Kafka.