...

Currently, the SocketServer does not support invalid static configurations, as it will crash Kafka with an uncaught exception if the static configuration is invalid during server startup, even if previous dynamic reconfigurations make the configuration valid. It would be better to support invalid static SocketServer configurations so long as there were dynamically set changes that made them valid.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

SocketServer dynamic config changes should be written to a file. This file is watched to apply configuration changes to in-memory config state.

During startup, the server will:

...

Based on the approach chosen below, enabling this functionality may require an additional file on disk.

Proposed Changes

There are two general approaches: 

1. Load dynamic configuration changes via the local metadata log snapshot and apply them before constructing the SocketServer (current preferred approach).
2. Write SocketServer dynamic configuration changes to a file. Load this file and apply the changes before constructing the SocketServer.

Points of discussion on the snapshot approach:

• Does not have to deal with any cases where the file is not readable/writeable.
• Reuses state we already persist to disk.
• May not have the most up-to-date information depending on how old the snapshot is. One approach to remediate this is to have the broker take a snapshot while in controlled shutdown.

Points of discussion on the file approach:

• Have to deal with cases where file is not readable/writeable. Should the dynamic configuration updates go through but potentially be lost if the broker crashes?

...

Compatibility, Deprecation, and Migration Plan

...