Status

Current state: Under Discussion

Discussion thread: https://www.mail-archive.com/dev@kafka.apache.org/msg99126.html

JIRA: KAFKA-8621

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Current Kafka versions supports file based KeyStore and TrustStore via ssl.keystore.location and ssl.truststore.location configurations along with required passwords configurations.

This configuration requirement creates challenges for larger Kafka deployment with following setup,

• Having a company internal CA authority issuing the certificates
• Thousands of brokers
• 10x/100x number of Kafka Client boxes
• Requirement to use client auth
• Need to avoid storing key/trust store files on file system for stronger security
• Polyglot client base

The challenges are,

• Deploying the key/trust store files to the thousands of brokers
• Deploying the key/trust store files to 10x/100x Kafka Client boxes
• Keep all those key/trust store files and their passwords secure
• Operationally manage key rotations
• Lack of unified way to distribute KeyStore and TrustStores for different languages

Primarily the requirement of having the KeyStore and TrustStore locations on the file system manifests itself into many of the challenges listed above.

Assuming we have a custom Key Manager which can provide secure API to provision/access the keys we must find an alternative to the file based KeyStore and TrustStore.

The primary motivation here is to essentially provide an optional way to allow custom way to load KeyStore and TrustStore instead of relying on the file system. Of course, that needs to be accompanied by having an ability to load required passwords (for KeyStore, TrustStore and Keys) accordingly.

Public Interfaces

We will introduce an optional way to load KeyStore and TrustStore along with their required passwords as applicable.

This will be done via

1. Introducing two new configurations- ssl.keystore.loader and ssl.truststore.loader.
2. For each of the new configuration, we will have public interfaces as described below,

package org.apache.kafka.common.security.ssl;

import java.security.KeyStore;

public interface KeyStoreLoader {
	/**
	 * This loads the keystore. The keystore password will be fetched by whatever mechanism the implementation of this class chooses.
	 * Example: It could use current ssl.keystore.password configuration if it chooses.
	 * @return KeyStore object
	 */
	public KeyStore load();

	/**
	 * This returns the key's password.
	 */
	public String getKeyPassword();

	/**
	 * This method checks if the given keystore has been modified based on some criteria, typically last-modified timestamp. The definition of 'modified' is left to the 
	 * implementation.
	 * @return true - If the keystore was modified as defined by the implementation; false otherwise
	 */
	public boolean modified();
}

package org.apache.kafka.common.security.ssl;

import java.security.KeyStore;

public interface TrustStoreLoader {
	
	/**
	 * This loads the truststore. The truststore password will be fetched by whatever mechanism the implementation of this class chooses.
	 * Example: It could use current ssl.truststore.password configuration if it chooses.
	 * @return KeyStore object
	 */
	public KeyStore load();

	/**
	 * This method checks if the given truststore has been modified based on some criteria, typically last-modified timestamp. The definition of 'modified' is left to the 
	 * implementation.
	 * @return true - If the truststore was modified as defined by the implementation; false otherwise
	 */
	public boolean modified();
}

Why we do not specify key/trust store password as input method arguments in the interfaces?

We are not specifying the key/trust store passwords in the KeyStoreLoader/TrustStoreLoader load() method. This is because we want to avoid the dependency in the caller class to load the password. This implementation leaves it open to the Loader implementation to read required configuration or use other mechanism for fetching the password. Typically if you have a Key Manager solution you might be using some sort of 'auth-token' in order to access the Key Manager's API and might not require key/trust store password (you will still need password for unlocking the keys though).

Proposed Changes

Kafka Client library and Kafka Broker both uses SslEngineBuilder class to load KeyStore and TrustStore from the file based configurations.

1. As documented in public interfaces section, we will introduce two interfaces to allow pluggable implementation to provide key/trust stores loading
2. We will make changes to the SslEngineBuilder#createSSLContext() method to invoke the key/trust store loading from new ssl configurations we introduce.
   1. Pseudocode changes in the SslEngineBuilder#createSSLContext() looks like below

      if ( keystore != null ) {
       // load keystore in existing way
      } else if ( 'ssl.keystore.loader' specified ) {
       // load keystore by invoking the pluggable implementation class for the config
      }
      ...
      ...
      ...
      if ( truststore != null ) {
       // load truststore in existing way
      } else if ( 'ssl.truststore.loader' specified ) {
       // load truststore by invoking the pluggable implementation class for the config
      }

3. We will make changes to the SslEngineBuilder#shouldBeRebuilt() method appropriately

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?

Existing users using file based key/trust stores are not going to be impacted at all.

• If we are changing behavior how will we phase out the older behavior?

Older behavior does not change so no need to phase it out.

• If we need special migration tools, describe them here.

No special migration tools needed.

• When will we remove the existing behavior?

We will keep the existing behavior and add optional new behavior.

Rejected Alternatives

Using existing ssl.provider config

We could think of writing a custom java.security.Provider implementation to load the required KeyStore and TrustStore in a custom way. However, it would not be the correct way to use the java.security.Provider as per its responsibilities documented in java docs.

This was not the correct way.

Provide a way to delegate SSLContext creation

We could create a new configuration like ssl.context.loader/ssl.context.initializer and use the implementation class to obtain the object of javax.net.ssl.SSLContext instead of using SslEngineBuilder#createSSLContext(). 

This is more work than actually needed and we looked at Keys/Secrets' Managers like Hashicorp's Vault as a sample integration and realized that the Vault API provides us way to get the Keys/Secrets/Certs but not the SSLContext object we need. This will be true for most Key Managers since it's primary responsibility is to manage keys/secrets/certs but not  the SSLContext.

Also, if we do provide a way to create SSLContext in a custom way, we must still honor the "provider" value used in SslEngineBuilder#createSSLContext().

Overall, we didn't find enough justification to follow this path.

Generated the required SSL configuration values from the Key Manager API

If we have a Key Manager solution which provides APIs like Hashicorp's Vault we need to find a way to generate the required ssl configurations for Kafka (key/trust stores files, passwords etc) from the same.

We could,

• Build a mechanism to download the required keys/certs from the Key Manager API and create required key/trust store files
• Standardize the path to the key/trust stores files on the disk
• Protect the key/trust stores files & Kafka Broker/Client files with appropriate permissions
• Generate the required configurations for Kafka Client boxes and Kafka Brokers

... AND we will not need any customization we are talking about here.

However, this approach does not scale for managing deployments across many brokers and client boxes and it does not solve for any of the challenges mentioned in the "Motivation" section.

Hence this approach was rejected.