...

This KIP will introduce a new common configuration option ssl.sslfactory.class to be added to SslConfigs. This will automatically add the new option to AdminClientConfig, ConsumerConfig and ProducerConfig.

A new public interface PluggableSslFactory named SslFactory will be created. An implementation of PluggableSslFactory will use the new config keys ssl.mode and ssl.verify.keystore.using.truststore internally so they must be documented but they  will not appear as ConfigDefs in SslConfigs.

The non-default constructors of SslFactory will be deprecated and a new default constructor will be added.

Proposed Changes

A new public interface will be defined:

public interface PluggableSslFactory extends Reconfigurable {
public SSLEngine createSslEngine(String peerHost, int peerPort);
}

...

A default implementation named DefaultSslFactory will be included with Kafka. The implementation is private, but the class name will leak as the default value of ssl.sslfactory.class

An application developer can implement SslFactory and set ssl.sslfactory.class to the fully qualified class name to instruct Kafka to use this implementation instead of the default implementation.

Proposed Changes

A new configuration option will be added to SslConfigs:

public static final String SSL_SSLFACTORY_CLASS_CONFIG = "ssl.sslfactory.class";
public static final String SSL_SSLFACTORY_CLASS_DOC = "...";
public static final String DEFAULT_SSL_SSLFACTORY_CLASS = "org.apache.kafka.common.security.ssl.SslFactoryDefaultSslFactory";

config.define(SslConfigs.SSL_SSLFACTORY_CLASS_CONFIG, ConfigDef.Type.CLASS, SslConfigs.DEFAULT_SSL_SSLFACTORY_CLASS, ConfigDef.Importance.LOW, SslConfigs.SSL_SSLFACTORY_CLASS_DOC)

Currently, SslFactory is a class that is not part of the public API. This class will be renamed DefaultSslFactory to make the name SslFactory available for the new interface.

A new public interface named SslFactory will be created.

public interface SslFactory extends Reconfigurable {

   /* valid values are Mode.CLIENT or Mode.SERVER */

   public static final String SSL_MODE_CONFIG = "ssl.mode";

   /* valid values are Boolean.TRUE or Boolean.FALSE */

   public static final String SSL_VERIFY_KEYSTORE_USING_TRUSTSTORE_CONFIG = "ssl.verify.keystore.using.truststore";

   public SSLEngine createSslEngine(String peerHost, int peerPort);

}

The source code of SslFactory will contain javadoc comments to describe the interface and each declaration.

An implementation of SslFactory will be instantiated with a default constructor. The non-default constructors in DefaultSslFactory will be removed. The member clientAuthConfigOverride will also be removed. The default constructor of DefaultSslFactory does nothing and therefore can be omitted too.

DefaultSslFactory can no longer rely on constructor arguments. Instead it will receive these values as regular config options in configure((Map<String, ?> configs). This KIP introduces two new config keys ssl.mode and ssl.verify.keystore.using.truststore. The configure() method in DefaultSslFactory will be modified to initialize the mode from the ssl.mode config and the boolean keystoreVerifiableUsingTruststore from the ssl.verify.keystore.using.truststore config. The variable clientAuthConfig will always be initialized from the ssl.client.auth config. It is the caller's responsibility to override ssl.client.auth before calling configure() because it can no longer be overridden by a constructor argument.

In SaslChannelBuilder and SslChannelBuilder, the type of the member sslFactory will still be changed to PluggableSslFactorySslFactory but now it refers to the interface. Instead of calling the SslFactory constructor directly, we will call a new method createSslFactory(). This method will be duplicated in both channel builders.

createSslFactory() will alter (a copy of?) the configs to contain the values for what used to be passed as arguments in the SslFactory constructor. The SSL mode will be stored with the key ssl.mode; if not null clientAuthConfigOverride will be stored with the key ssl.client.auth overwriting the value already there; keystoreVerifiableUsingTruststore will be stored with the key ssl.verify.keystore.using.truststore; The new keys ssl.mode and ssl.verify.keystore.using.truststore are private. We will define them as constants in SslConfigs but they will not have associated ConfigDefs. .

createSslFactory() will call the default constructor of the pluggable SSL Interface by calling Class.newInstance() followed by a call to configure() to pass the configs.The class comment of the PluggableSslFactory interface will document the necessary default constructor and the private keys ssl.mode and ssl.verify.keystore.using.truststore

EchoServer calls SslFactory directly and will no longer compile. This is an internal test so backwards compatibility does not apply. We will update EchoServer to call DefaultSslFactory directly instead.

Compatibility, Deprecation, and Migration Plan

This Because SslFactory was not a public API, this KIP is considered backwards compatible. Most applications will create the SslFactory through the SslChannelBuilder or the SaslChannelBuilder which will hide the changesApplications that called SslFactory directly will have to be updated but that's the risk taken when calling private APIs.

The default value for the new config ssl.sslfactory.class is org.apache.kafka.common.security.ssl.SslFactory DefaultSslFactory which selects the existing implementation.

The non-default constructors of SslFactory will be preserved and marked as deprecated. The constructor arguments will override the configs, just like clientAuthConfigOverride did, but now for all three arguments.

The new config keys ssl.mode and ssl.verify.keystore.using.truststore are private. They should not be configurable by the user.

SslFactory implementation is instantiated by the SslChannelBuilder or the SaslChannelBuilder which will hide all the changes.

EchoServer calls SslFactory directly, but that's an internal test. We will update EchoServer as part of this KIP.The EchoServer used in the tests could be updated to use PluggableSslFactory. It is suggested to keep it the way it is to test backwards compatibility. 

Rejected Alternatives

Kafka could define a new configuration option to hold an instance of SSLSocketFactory. This is similar to many Java libraries that accept an instance of SSLSocketFactory. This was rejected because Kafka tries to be language neutral. It was thought it would make it more difficult to support C and Python.

Ideally, the interface should be called SslFactory and the built-in implementation should be called DefaultSslFactory. This was rejected to improve backwards compatibility for applications that call the SslFactory directly.

Kafka's naming convention does not use a special tag for interfaces. Accordingly, these interface names were rejected ISslFactory, SslFactoryIFace, SslFactoryInterface.

It was possible to preserve the name SslFactory for the default implementation and use the name PluggableSslFactory for the interface. Since SslFactory was not part of the public API, it was decided we could rename the class and reuse the nicer name for the interface.

The constants SSL_MODE_CONFIG and SSL_VERIFY_KEYSTORE_USING_TRUSTSTORE_CONFIG are not user configurable and therefore must not appear in SslConfigs as ConfigDefs. Declaring the constants in SslConfigs without a ConfigDef would just cause confusion.

The only call to SslFactory.sslContext() is in EchoServer which is part of the client tests. We felt this was not a strong enough motivation to add sslContext() to the PluggableSslFactory interface, especially if EchoServer continues to call SslFactory directlydoes not count because it is an internal test. We decided to leave the sslContext() method out of the interface.

Kafka is not consistent to name configs that hold class names. Compare [partitioner.class, interceptor.classes, principal.builder.class, sasl.server.callback.handler.class, sasl.client.callback.handler.class, sasl.login.class] versus [key.deserializer, value.deserializer, key.serializer, value.serializer]. It appears the serializer/deserializer configs are a special case. Therefore the name ssl.sslfactory.class was selected instead of ssl.sslfactory

We could require the Pluggable SSL Factory implementation to have the same 3-argument constructor as the old SslFactory. We find the arguments clientAuthConfigOverride and keystoreVerifiableUsingTruststore are somewhat arbitrary for a general pluggable interface. We prefer to use a default constructor for the pluggable SSL Factory implementation. This does not alter backwards compatibility because it is easy to keep the non-default constructors in SslFactory and have those values override whatever is in the configs.

We need the ability to send custom configuration options to the PluggableSslFactory implementation. Should this This will be covered in another KIP?.