Status

Current state: WIP

Discussion thread:

JIRA: 

Motivation

In Kafka, there is no general mechanism to change entity configuration without doing a rolling restart of the entire cluster. Currently, only topic configs can be changed dynamically. This proposal attempts to build a unified mechanism for modeling configuration across various entities within Kafka i.e. topics, clients etc.

Public Interfaces

• We will add a new tool called ConfigChangeCommand that can manage all config changes in ZK. New methods will be added in AdminUtils to change configuration. This will be similar to the TopicCommand tool already present. Going forward, we will also deprecate the TopicCommand tool and have all config changes be driven through ConfigChangeCommand.
• There will be new zookeeper paths under "config" but they are not considered to be public interfaces.

Proposed Changes

This proposal will generalize all the code in TopicConfigManager to make it possible to handle any kind of config change via Zookeeper. Initially, we will have 2 specific implementations:

• TopicConfigManager - Same as current but using the new general classes
• ClientConfigManager - The purpose of this is to manage per-client configs. This is required for quotas and potentially for managing permissions/authorization in the future.

ZNode Structure:

There should be 4 paths within config
/config/clients/<client_id>
/config/topics/<topic_name>
/config/brokers/<broker_id>

Internally, the znodes are comma-separated key-value pairs where key represents the configuration to change.
{"version": x, "config" : {X1=Y1, X2=Y2..}

Upon startup, all brokers will load all the configs from zookeeper. In order to receive notifications, it's undesirable to watch every path under the root config directory. We can model change notifications the same way as they are currently handled for topic based configs. Here is the workflow for changing or adding a config of any type:

• Create a znode (or modify existing) under the required path with the configs that you want to update. Example, if you want to change quotas for producer "Adi", add a path under /config/config/Adi as shown below.
• Create a sequential znode under "config/changes/config_change_XX". This will send a notification to all the watchers. The data within the change node should indicate what has changed i.e. topic config + topic name, client config + clientId.
• The brokers process the changed configs.

Config Precedence

Configs in Zookeeper take precedence over any configuration read from the properties file. There is currently no plan to move away from file based configuration for service configs.

Applying Configs within Broker

The ZK listener can have a reference to KafkaConfig or the properties object which was used to start the KafkaServer. Anytime a change notification is received, we should do the following:

1. Parse received notification into a properties object.
2. Within KafkaConfig, for each newly changed property, verify that it can be changed dynamically. For this, ConfigDef needs to expose a getConfigKey() method that can be used to check the isDynamic flag. If a property cannot be updated, log a warning and move on.

All configs should always be accessed via a reference to KafkaConfig. For this, all subsystems within the broker need to be configured with a config object and not individual config values.

// Sample code only
class BrokerConfigManager {
	// current kafka config
	val kafkaConfig : KafkaConfig = ... ;
	.....
	def processConfigChanges(notifications: Seq[String]) {
		val updatedProperties = AdminUtils.fetchConfig(zkClient, configPath)
		kafkaConfig.updateProperties(updatedProperties);		
	}	
}
 
class AdminUtils {
 def fetchConfig(zkClient: ZkClient, path: String): Properties = {
	val str: String = zkClient.readData(path, true)
	val props = new Properties()
	// Read the znode, parse the JSON and return a properties object
	props
}

Config Change Notification

Currently, the change notification znode only contains the topic name. We need to add more information to distinguish whether this config change is for a topic, client or broker config. We need to distinguish producers and consumers separately because it is possible for a producer and consumer to have the same clientId. Version numbering should also be added.

The notification data can be:
{"version" : 1, "entity":"topic/producer/consumer/broker", "value" : "topic_name/client_id/broker_id"}

Compatibility, Deprecation, and Migration Plan

• TopicConfigManager has a config_change_XX sequential znode under configs/. The format of data within the config_change node is going to change hence it is important to not make any config changes using AdminTools until the cluster is fully upgraded. 

Rejected Alternatives