Status

Current state: Under Discussion

Discussion thread: here

JIRA:  KAFKA-10731 - Add Support for SSL hot reload Open

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka clients authenticate with brokers using TLS. The SSL credentials (keystore and truststore) are loaded once at startup and never refreshed. When certificates expire or are rotated, the only way to apply new credentials today is to restart every client. In environments that use short-lived certificates this forces frequent restarts, adds operational complexity, and increases the risk of downtime.

This KIP adds opt-in SSL hot reload to Kafka clients. When enabled, the client monitors its keystore and truststore files and automatically reconfigures its SslFactory when a change is detected, with no restart required.

Prior work: KIP-1119

KIP-1119 ("Add support for SSL auto reload") addressed the same problem. That work was not merged, for two reasons that shaped the design choices made in this KIP:

1. Reliability of Java's WatchService in Kubernetes

KIP-1119 used WatchService to detect file changes. WatchService relies on the Linux inotify kernel subsystem. However, inotify does not emit events for bind mounts, which is how Kubernetes mounts ConfigMaps and Secrets into pods. This means WatchService would silently miss certificate rotations in the most common Kubernetes deployment pattern, the exact environment where automatic reload is most needed.

2. Transient inconsistency during rotation

A certificate rotation is not atomic. An operator (or automation tool like cert-manager) must write the new keystore and the new truststore in two separate file operations. There is always a window (sometimes several seconds) between the two writes. If the reload fires after the first write but before the second, the client loads a mismatched pair: a new keystore with an old truststore (or vice versa). Each file is individually valid, so no exception is thrown, but the pair is semantically inconsistent. New TLS handshakes can fail until the second file is also written and the client reloads again.

Public Interfaces

New Configuration Properties

All three keys are optional and off by default. Existing configurations require no changes.

New Internal Classes

These classes are in org.apache.kafka.common.security.ssl and are not exposed in any public API surface.

Proposed Changes

Why polling instead of WatchService

The WatchService approach was rejected for this KIP for the same reason it blocked KIP-1119: inotify does not fire on bind-mounted files in Kubernetes, which is the primary deployment environment for this feature. Polling with Files.getLastModifiedTime() is less elegant but works uniformly across bare-metal, VMs, Docker, and Kubernetes, regardless of how the volume is mounted.

The polling interval defaults to 60 seconds. For production workloads with automated rotation tools this is more than sufficient; for testing or fast-rotating certificates it can be reduced via ssl.hotreload.poll.interval.seconds.

Why debounce

A certificate rotation involves writing two files (keystore and truststore) in two separate operations. Without a grace period, the poller may detect the first write and reload immediately, loading an inconsistent pair. The debounce mechanism delays the reload until the filesystem has been quiet for a configurable window.

The logic is:

1. The poller detects a file change.
2. Instead of notifying listeners immediately, it schedules a one-shot "notify" task to run after ssl.hotreload.debounce.seconds.
3. If another file change is detected before the task fires, the task is cancelled and a new one is scheduled.
4. When the task finally fires, it notifies all listeners. By this time both files have settled.

Setting ssl.hotreload.debounce.seconds=0 disables this behaviour entirely and restores immediate-fire semantics. The default of 5 seconds covers typical rotation tooling (cert-manager, Vault Secrets Operator,..) without introducing a noticeable delay.

The SslMaterialPollerRegistry

A single Kafka application can create many SslFactory instances (one per consumer, one per producer, etc.). The naive approach would create one polling thread per SslFactory. This is wasteful in the common case where all factories share the same SSL configuration.

Two extremes were considered:

• One global singleton poller: efficient, but invalid if the application connects to multiple clusters using different certificates.
• One poller per factory: safe but redundant.

This KIP takes a middle path. SslMaterialPollerRegistry is a process-wide registry keyed by the tuple (keystorePath, truststorePath, pollIntervalSeconds, debounceSeconds). Two factories with identical keys share one poller and one polling thread. Each factory registers a listener (its reconfigure callback) with that shared poller. When a factory is closed it deregisters its listener; the poller is stopped and removed from the registry when its last listener is removed.

This means:

• Applications with a single SSL configuration (the common case) use exactly one polling thread, regardless of how many consumers and producers they create.
• Applications that connect to multiple clusters with different certificates get independent pollers, one per distinct configuration. Isolation is preserved: a file change for cluster A does not trigger a reload for cluster B.

Changes to SslFactory

• Add field Runnable onSslMaterialChange to hold the listener reference (required for deregistration, which uses reference equality).
• In configure(): if hot reload is enabled, deregister any previous listener and call SslMaterialPollerRegistry.getInstance().register(configs, this.onSslMaterialChange).
• In close(): call SslMaterialPollerRegistry.getInstance().deregister(configs, this.onSslMaterialChange) to clean up. This ensures no thread leak when a producer or consumer is closed.

Compatibility, Deprecation, and Migration Plan

• The feature is opt-in and disabled by default. No existing behaviour changes unless ssl.hotreload.enable=true is set.
• The feature applies to clients only: producers, consumers, and admin clients. Broker-side hot reload is already possible via dynamic configuration and is out of scope for this KIP.

Test Plan

• Verify that changing a truststore file (simulated by updating its modification time) triggers SslFactory reconfiguration.
• Verify that when two factories watch different files, changing one file only reloads the corresponding factory.
• Verify that hot reload disabled (ssl.hotreload.enable=false) produces no reconfiguration even when files change.
• Verify that the debounce timer resets when a second file change arrives within the window.
• Verify that two changes in rapid succession produce exactly one listener invocation, not two.
• Verify that setting debounce=0 calls the listener immediately.
• Verify that two factories with the same SSL config share one poller (one polling thread).
• Verify that two factories with different SSL configs get independent pollers.
• Verify that closing the last factory for a given config removes the poller from the registry.

This was materialized in 8 new tests. in SslFactoryTest.

Rejected Alternatives

Using WatchService

Rejected because inotify does not emit events for bind-mounted files. Kubernetes mounts ConfigMaps and Secrets as bind mounts. This is the most common deployment environment for this feature, so a mechanism that silently fails there is not acceptable.

Very good article I found when I was surprised the no event was emitted in Kubernetes setup: https://blog.arkey.fr/2019/09/13/watchservice-and-bind-mount/

One SslMaterialPoller per SslFactory (no registry)

Simple to implement and fully isolated, but creates one polling thread per producer and consumer even when they all watch the same two files. In applications with many clients this is wasteful. Rejected in favour of the registry approach.

One global singleton poller

Efficient for the common case (that was the choice for KIP-1119), but incorrect when a single application connects to multiple clusters with different SSL credentials. A global poller can only watch one set of files; it cannot serve multiple distinct configurations. Rejected in favour of the registry approach.