Status

Current state: "Under Discussion"Discarded, replaced by KIP-519: Make SSL context/engine configuration extensible

Discussion thread: here

JIRA: here

...

Of course it would be easy to simply change the Kafka source code and ship a customized distribution. The idea is you should be able to replace the implementation through configuration, without rebuilding a custom Kafka distribution or resorting to classpath tricks to shadow Kafka classes.

SslFactory handles two different tasks: creating the SSLContext and reconfiguration. There is no need to make reconfiguration pluggable. This functionality should remain in SslFactory to be shared by all pluggable implementations. SslFactory only needs to delegate the creation of the SSLContext to the pluggable interface.

Since SslFactory will handle reconfiguration, the idea is to make the configuration immutable in the pluggable factory. SslFactory would create a new pluggable factory every time the configuration changes. The pluggable factory creates its SSLContext when it is configured and never changes it. It turns out SslFactory does not really need the SSLContext, so it can use the new pluggable factory as an SSLEngine factory instead.

Public Interfaces

This KIP will introduce a new common configuration option ssl.sslfactorysslengine.factory.class to be added to SslConfigs. This will automatically add the new option to AdminClientConfig, ConsumerConfig and ProducerConfig.

A new public interface PluggableSslFactory named SslEngineFactory will be created. An implementation of PluggableSslFactory will use the new config keys ssl.mode and ssl.verify.keystore.using.truststore internally so they must be documented but they  will not appear as ConfigDefs in SslConfigs.The non-default constructors of SslFactory will be deprecated and a new default constructor will be addedA default implementation named DefaultSslEngineFactory with the existing behavior will be included with Kafka. The implementation is private, but the class name will leak as the default value of ssl.sslengine.factory.class

An application developer can implement SslEngineFactory and set ssl.sslengine.factory.class to the fully qualified class name to instruct Kafka to use this implementation instead of the default implementation.

Proposed Changes

A new public interface will be defined:

public interface PluggableSslFactory extends Reconfigurable {
public SSLEngine createSslEngine(String peerHost, int peerPort);
}

...

configuration

...

option

...

will

...

be

...

added

...

to

...

SslConfigs:

  public static final String SSL_SSLFACTORYSSLENGINE_FACTORY_CLASS_CONFIG = "ssl.sslengine.sslfactoryfactory.class";
  public static final String SSL_SSLFACTORYSSLENGINE_FACTORY_CLASS_DOC = "...";
  public static final String DEFAULT_SSL_SSLFACTORYSSLENGINE_FACTORY_CLASS = "org.apache.kafka.common.security.ssl.SslFactoryDefaultSslEngineFactory";

  config.define(SslConfigs.SSL_SSLFACTORYSSLENGINE_FACTORY_CLASS_CONFIG, ConfigDef.Type.CLASS, SslConfigs.DEFAULT_SSL_SSLENGINE_SSLFACTORYFACTORY_CLASS, ConfigDef.Importance.LOW, SslConfigs.SSL_SSLENGINE_SSLFACTORYFACTORY_CLASS_DOC)

A new public interface named SslEngineFactory will be created.

  public interface SslEngineFactory {

      void configure(Map<String, ?> configs, Mode mode)

In SaslChannelBuilder and SslChannelBuilder, the type of the member sslFactory will be changed to PluggableSslFactory. Instead of calling the SslFactory constructor directly, we will call a new method createSslFactory(). This method will be duplicated in both channel builders.

createSslFactory() will alter (a copy of?) the configs to contain the values for what used to be passed as arguments in the SslFactory constructor. The SSL mode will be stored with the key ssl.mode; if not null clientAuthConfigOverride will be stored with the key ssl.client.auth overwriting the value already there; keystoreVerifiableUsingTruststore will be stored with the key ssl.verify.keystore.using.truststore; The new keys ssl.mode and ssl.verify.keystore.using.truststore are private. We will define them as constants in SslConfigs but they will not have associated ConfigDefs. createSslFactory() will call the default constructor of the pluggable SSL Interface by calling Class.newInstance() followed by a call to configure() to pass the configs.

;

      SSLEngine createSslEngine(String peerHost, int peerPort);

  }

SslFactory finds the class name of the SslEngineFactory implementation in the ssl.sslengine.factory.class config and instantiates it using the default constructor by calling Class.newInstance(). The configs and mode are then passed to configure. This creates the SSLContext which remains private to the SslEngineFactory implementation. SslFactory can now call createSslEngine(peerHost, peerPort) as often as needed.

This assumes SslFactory copies the configs and overwrites ssl.client.auth when there is a clientAuthConfigOverride.

When reconfiguring, only the keystore and truststore configs are allowed to change. This is the existing behavior and this KIP simply preserves it. SslFactory must keep a copy of the previous configs to know the original values for the other configs. This copy also guarantees we don't overwrite anything in the caller's configs.

SslFactory must inspect the keystore and truststore to detect a future change in configuration. In particular, it needs to keep the lastModified times, plus the CertificateEntries of the keystore. The configuration might change during the small interval between the inspection and loading the files in the SslEngineFactory implementation. By always inspecting the files before calling configure, we make sure the SslEngineFactory implementation will always have the same or newer configuration and reconfiguration will never miss an update later. 

It is allowed to keep ssl.keystore.location and/or ssl.truststore.location empty. This means the discovery of the keystore and truststore is fully delegated to the SslEngineFactory implementation. This turns off reconfiguration when both the old and new configuration delegates to the SslEngineFactory implementation.

SslFactory.sslContext() will be removed because the SslContext is now private to the SslEngineFactory implementation.The class comment of the PluggableSslFactory interface will document the necessary default constructor and the private keys ssl.mode and ssl.verify.keystore.using.truststore

Compatibility, Deprecation, and Migration Plan

This KIP is considered backwards compatible. Most applications will create the SslFactory through the SslChannelBuilder or the SaslChannelBuilder which will hide the changesSslFactory is not part of the Kafka public API. The SslFactory API remains the same anyway except for the removal of sslContext().

EchoServer is the only caller of SslFactory.sslContext(), This is an internal test so backwards compatibility does not apply. We will update EchoServer to call DefaultSslFactory directly instead.

The default value for the new config ssl.sslfactorysslengine.factory.class is org.apache.kafka.common.security.ssl.SslFactory DefaultSslEngineFactory which selects implements the existing implementation.

The non-default constructors of SslFactory will be preserved and marked as deprecated. The constructor arguments will override the configs, just like clientAuthConfigOverride did, but now for all three arguments.

The new config keys ssl.mode and ssl.verify.keystore.using.truststore are private. They should not be configurable by the user.

The EchoServer used in the tests could be updated to use PluggableSslFactory. It is suggested to keep it the way it is to test backwards compatibility. 

behavior.

Rejected Alternatives

Kafka could define a new configuration option to hold an instance of SSLSocketFactory. This is similar to many Java libraries that accept an instance of SSLSocketFactory. This was rejected because Kafka tries to be language neutral. It was thought it would make it more difficult to support C and Python.Ideally, the interface should be called SslFactory and the built-in implementation should be called DefaultSslFactory. This was rejected to improve backwards compatibility for applications that call the SslFactory directly.

Kafka's naming convention does not use a special tag for interfaces. Accordingly, these interface names were rejected ISslFactoryISslEngineFactory, SslFactoryIFaceSslEngineFactoryIFace, SslFactoryInterface.SslEngineFactoryInterface.

We could expose the SSLContext with an accessor in the SslEngineFactory interface, but it was only used by an internal test, so we decided to leave it outThe only call to SslFactory.sslContext() is in EchoServer which is part of the client tests. We felt this was not a strong enough motivation to add sslContext() to the PluggableSslFactory interface, especially if EchoServer continues to call SslFactory directly.

Kafka is not consistent to name configs that hold class names. Compare [partitioner.class, interceptor.classes, principal.builder.class, sasl.server.callback.handler.class, sasl.client.callback.handler.class, sasl.login.class] versus [key.deserializer, value.deserializer, key.serializer, value.serializer]. It appears the serializer/deserializer configs are a special case. Therefore the name ssl.sslengine.sslfactoryfactory.class was selected instead of ssl.sslengine.sslfactoryfactory

We could require the Pluggable SSL Factory implementation to have the same 3-argument constructor as SslFactory. We find the arguments clientAuthConfigOverride and keystoreVerifiableUsingTruststore are somewhat arbitrary for a general pluggable interface. We prefer to use a default constructor for the pluggable SSL Factory implementation. This does not alter backwards compatibility because it is easy to keep the non-default constructors in SslFactory and have those values override whatever is in the configsadd clientAuthConfigOverride as a third argument for SslEngineFactory.configure(). This looked like an internal feature of Kafka that leaked into the more general interface. We thought overwriting ssl.client.auth in a copy of the configs was cleaner.

We could pass the keystore and truststore as arguments to the SslEngineFactory since SslFactory already loaded them. This felt awkward since the keystore and truststore configs would still be present in the configs passed to configure(). It also felt clumsy when the SslEngineFactory uses its own keystore and truststore, for example, when reusing an SSLContext from the rest of the application.

We need the ability to send custom configuration options to the PluggableSslFactory implementation. Should this This could be covered in another KIP?.