...

Discussion thread:

JIRA: 

Motivation

Currently, all cluster configuration is managed via property files read by the brokers on startup with the exception of topic configuration. There is no In Kafka, there is no general mechanism to change broker entity configuration without having to do doing a rolling restart of the entire cluster. Additionally, there is no way to model configuration per-client i.e. quotas, permissions etc.This Currently, only topic configs can be changed dynamically. This proposal attempts to build a unified mechanism for modeling configuration across various entities like within Kafka i.e. topics, clients and brokers using Zookeeperetc.

Public Interfaces

• We will add a new tool called ConfigChangeCommand that can manage all config changes in ZK. New methods will be added in AdminUtils to change configuration. This will be similar to the TopicCommand tool already present. Going forward, we will also deprecate the TopicCommand tool and have all config changes be driven through ConfigChangeCommand.
• There will be new zookeeper paths under "config" but they are not considered to be public interfaces.

Proposed Changes

This proposal will reuse pieces of the TopicQuotaManager. Similar to topic configs, we can have quota managers for Brokers and Clients (producer and consumer). These classes will listen to Zk notifications from a certain path and apply the notifications within the brokers on a per-topic, broker or clientId basis. These approaches can share significant amounts of code amongst themselves. The only thing that is different is the entity that is being modeled and how the config changes are appliedgeneralize all the code in TopicConfigManager to make it possible to handle any kind of config change via Zookeeper. Initially, we will have 2 specific implementations:

• TopicConfigManager - Same as current but using the new general classes
• ClientConfigManager - The purpose of this is to manage per-client configs. This is required for quotas and potentially for managing permissions/authorization in the future.

ZNode Structure:

There should be 4 paths within config
/config/producersclients/<client_id>
/config/consumers/<client_id>
/config/topics/<topic_name>
/config/brokers/<broker_id>

Internally, the znodes are comma-separated key-value pairs where key represents the configuration to change.
{"version": x, "config" : {X1=Y1, X2=Y2..}

...

Upon startup, all brokers will load all the configs from zookeeper. In order to receive notifications, it's undesirable to watch every path under the root config /directory. We can model change notifications the same way as they are currently handled for topic based configs. Here is the workflow for changing or adding a config of any type:

• Create a znode (or modify existing) under the required path with the configs that you want to update. Example, if you want to change quotas for producer "Adi", add a path under /config/producersconfig/Adi as shown below.
• Create a sequential znode under "config/changes/config_change_XX". This will send a notification to all the watchers. The data within the change node should indicate what has changed i.e. topic config + topic name, client config + clientId or broker config + brokerId.
• The brokers process the changed configs.

Config Precedence

These configs Configs in Zookeeper will override take precedence over any configuration read from the properties file. We also do not There is currently no plan to move away from file based configuration.

Modeling Default values

In addition to overrides, we also need a mechanism to model default configuration . For example: say we have default quotas for all clients and we need to selectively override them on a per-client basis. So far, we don't have a way to change the the default values. We can do this by having a special path to store defaults.

The properties in this znode are applicable to all clients, topics and brokers respectively. Changing something here should affect all entities unless overridden explicitly.
 
/config/producers/__default
/config/consumers/__default
/config/topics/__default
/config/brokers/__default
 
Let's extend the example of quotas. Assume that we have a default quota of 5Mbytes per second per producer and we want to change it to 10M for the producer Adi. The default znode will look like this:
/config/producers/__default
{"version": 0, "config" : {quota=5M}

Overridden config:
/config/producers/Adi
{"version": 0, "config" : {quota=10M}

Broker Configuration

For Broker configs, the key can be the KafkaConfig property that we wish to override. All configs represented here should override the configs from the property file.

/config/brokers/0
{"version": 0, "config" : {log.retention.check.interval.ms=100000}

 This KIP provides a mechanism to change any broker config dynamically but identifying which configs can be changed and how the config changes should be applied within the brokers is out of scope. 

ConfigDef Changes

We will need to add a new property to the ConfigKey to indicate if a config is updatable or not. This property can be used to generate documentation so it becomes very easy to discover which properties can be dynamically changed. If a broker property is set in zookeeper and it is not a config marked "isDynamic", that property can simply be ignored. This is similar to configuring Kafka with a property it does not understand.

...

for service configs.

Applying Configs within Broker

The ZK listener can have a reference to KafkaConfig or the properties object which was used to start the KafkaServer. Anytime a change notification is received, we should do the following:

...