...

This proposal will generalize all the code in TopicConfigManager to make it possible to handle any kind of config change via Zookeeper. Initially, we will have 2 specific implementationsentities:

• TopicConfigManager - Same as current but using the new general classesgeneralized code.
• ClientConfigManager - The purpose of this is to This will manage per-client configs. This is required for quotas and potentially for managing permissions/authorization in the future.

ZNode Structure:

There shouldwill be 3 paths within config
/config/clients/<client_id>
/config/topics/<topic_name>
/config/brokers/<broker_id>

Internally, the znodes are comma-separated key-value pairs where key represents the configuration property to change.
{"version": x, "config" : {X1=Y1, X2=Y2..}}

...

• Create a znode (or modify existing) under the required path with the configs that you want to update. Example, if you want to change quotas for producer "Adi", add a path under /config/configclients/Adi as shown below.
• Create a sequential znode under "config/changes/config_change_XX". This will send a notification to all the watchers. The data within the change node should indicate what has changed i.e. topic config + topic name, client config + clientId.
• The brokers process all the configs for the entity that has changed configs. 

Config Precedence

Configs in Zookeeper take precedence over any configuration read from the properties file. There is currently no plan to move away from file based configuration for service configs.

...

The first usecase for client configs it is to process quota changes per-client. This class will process all notifications and change clientId quotas accordingly. This also requires some Some changes to the metrics package changes are also required to allow modification of quotas. Producer and consumer quotas are distinguished by having different keys properties since it's possible for a producer and consumer to have the same client id.

...

{"version" : 1, "config" : {"producer-byte-rate"=X1, "consumer-byte-rate"=X2}}

...

Config Change Notification

...

• TopicConfigManager has a config_change_XX sequential znode under configs/. The format of data within the config_change node is going to change hence it is important to not make any config changes using AdminTools TopicCommand until the cluster is fully upgraded. 

Rejected Alternatives

Dynamic Service Configs: After plenty of discussion, we've decided to not allow broker(service) configuration be dynamic. There were a couple of approaches:

• Service Config in ZK
• Reload config file via SIGHUP

See the attached discussion thread for more details on this decision.Broker Configs: