...

Our primary goal is: Instead of keep adding various ssl configurations to customize smaller portions of SSLContext/Engine, we should have a single configurable/pluggable SSLContext/Engine functionality.

Pluggable SSLContext or SSLEngine?

Java's Security Providers architecture already enables using custom SSLEngine by using the right JSSE compatible Provider like BouncyCastleJsseProvider. We could use any SSLEngine of the  TLS implementations as far as we have JSSE compatible provider available. We can use Kafka's 'security.providers' configuration to use a custom provider.

Making javax.net.ssl.SSLContext setup pluggable provides flexibility for providing Key Material, Secure Random implementation and configure Key/Trust managers in a custom way. Example: Apache HttpComponents and Netty SslContextBuilder. 

Hence we feel we should make SSLContext initialization pluggable.

Public Interfaces

New configuration

ssl.enginecontext.factory.class - This configuration will take class of the below interface's type and will be used to create javax.net.ssl.SSLEngine SSLContext object.

Default value will be as mentioned below.

final static String SSL_CONTEXT_ENGINEFACTORYFACTORY_CLASS_CONFIG = "ssl.enginecontext.factory.class";
final static String DEFAULT_SSL_CONTEXT_ENGINEFACTORYFACTORY_CLASS = "org.apache.kafka.common.security.ssl.DefaultSslEngineFactoryDefaultSslContextFactory";

Interface for

...

SSLContextFactory

Below is the interface suggested for this.

package org.apache.kafka.common.security.ssl;

import org.apache.kafka.common.network.Mode;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLEngine;
import java.util.Map;
import java.util.Set;

public interface SslEngineFactorySslContextFactory {

    /**
     * CreatesReturns aSSLContext newloaded SSLEngineby object.
     *
     * @param mode      Whether to use client or server modethis factory.
     * @param peerHost  The peer host to use. This is used in client mode if endpoint validation is enabled.
     * @param peerPort  The peer port to use. This is a hint and not used for validation.
     * @param endpointIdentification Endpoint identification algorithm for client mode.
     * @return          The new SSLEngine.
     */
    SSLEngineSSLContext createSSLEngine(Mode mode, String peerHost, int peerPort, String endpointIdentificationgetSSLContext();

    /**
     * Returns the currently used configurations by this engineobject.
     * @return
     */
    Map<String, Object> currentConfigsconfigs();

    /**
     * Returns the reconfigurable configs used by this engineobject.
     * @return
     */
    Set<String> reconfigurableConfigs();

    /**
     * Returns true if thisthe engineSSLContext needs to be rebuilt.
     *
     * @param nextConfigs       The configuration we want to use.
     * @return                  True only if thisthe builderSSLContext should be rebuilt.
     */
    boolean shouldRebuiltFor(Map<String, Object> nextConfigs);
}

...

• SslEngineBuilder.java (functionality of SSLContext loading will be moved to DefaultSslEngineFactoryto DefaultSslContextFactory.java and createEngine() method will move to SslFactory.java)

Which classes will be added?

• SslEngineFactorySslContextFactory.java Interface
• DefaultSslEngineFactoryDefaultSslContextFactory.java (mostly having code from existing SslEngineBuilder)

Which classes will be modified primarily?

• SslFactory.java java
  • Will host the createEngine() method from current SslEngineBuilder and will have mechanism to load the SslContextFactory implementation

How does configs get to the implementation class?

The configuration of Map will be passed to the implementation class via the constructor. See below example,Detailed code link: My fork

public DefaultSslEngineFactoryDefaultSslContextFactory(Map<String, ?> configs) {
...
...
}

...

These configuration will be passed from SslFactory to the implementation of the SslEngineFactory SslContextFactory interface via reflection like below

public class SslFactory implement Reconfigurable {
...
...
     sslEngineFactoryClasssslContextFactoryClass.getDeclaredConstructor(Map.class).newInstance(configs);
...
}

...

By custom configs we mean the configs used by the SslEngineFactorySslContextFactory's implementation. Those configs does not have to be part of definition of Kafka configs since only the implementation class knows what are those. Kafka already supports custom configs so this should not be a new challenge. 

...

Currently reconfigurations are pushed from Kafka code base to the reconfigurable classes. However, depending upon SslEngineFactorySslContextFactory's implementation we could have some events/changes detected by the implementation first and we would need to trigger reconfiguration on SslFactory in order to get re-initialized!

Probably this could be achieved by passing listener to those implementation changes but this needs to be further explored.

Sequence Diagram of SslFactory instantiating the SslEngineFactory implementation

Image Removed

Sequence diagram for the Reconfiguration flow

Image Removed

Compatibility, Deprecation, and Migration Plan

...

Not applicable since old code behavior will be kept with default implementation of DefaultSslEngineFactoryDefaultSslContextFactory and modification to SslFactory class.

Rejected Alternatives

Make SSLEngine pluggable

As mentioned noted in the motivation there are/were several attempts to make various ssl configurations pluggable over time focusing on specific aspect of the SSL configuration. However, this KIP proposes to allow customization at SSLContext/SSLEgnine level hence there are no alternatives applicable in our opinion. However there are couple of implementation alternatives that we rejected and document as below.

...

section- Java's Security Providers architecture already enables using custom SSLEngine by using the right JSSE compatible Provider hence Kafka should not have to derive another way to make SSLEngine pluggable.

Making SslFactory the pluggable interface (KIP-383)

This is because currently SslFactory does certain validations which we want to keep separate and mandate those checks across any possible implementation of pluggable ssl context class. Also, once we start writing the reconfigurable classes we realize that we need two classes - 1) Engine factory SslContextFactory implementation and 2) Container of the factory implementation. We believe that keeping SslFactory as Reconfigurable object and help reconfigure the underlying SslEngineFactory SslContextFactory will simplify the implementations of SslEngineFactorySslContextFactory.

...

Also, we rejected to make SslContextFactory extend the Reconfigurable interface due to following reason,

There will be good amount of state in the SslEngineFactorySslContextFactory's implementation (as it will be similar to the current SslEngineBuilder class). We believe that making SSLEngine SSLContext creation pluggable is worth to allow SSL experts to write their own implementation having the SSL domain knowledge and keep them free of knowing much about Kafka's reconfigurability - example: Netty OpenSslEngine Apache HttpComponents. We prefer SslFactory class to do what it is doing right now and keep the responsibility of re-creating underlying SslEngineFactory SslContextFactory object based on the configurations specified by the SslEngineFactory's implementationSslContextFactory's implementation.

Creating builder for SSLContext

Creating a builder interface with options to build SSLContext will need to have method(s) to allow keys/trusted-certs. It will also require us to have 'key-password' as input for the keystore. In the current Kafka implementation it requires the password to be configured in the plaintext via 'ssl.key.password', 'ssl.keystore.password' and 'ssl.truststore.password'. If we need to customize how the password is loaded, due to security reasons, this approach will not work since some other mechanism for making password pluggable (See KIP-76 Enable getting password from executable rather than passing as plaintext in config files AND KIP-486: Support custom way to load KeyStore and TrustStore) need to be devised which will add more ssl related configurations to Kafka.