Page History

...

This proposal will reuse pieces of the TopicQuotaManager. Similar to topic configs, the proposal is to add a BrokerQuotaManager and ClientConfigManagerwe can have quota managers for Brokers and Clients (producer and consumer). These classes will listen to Zk notifications from a certain path and apply the notifications within the brokers on a per-topic, broker or clientId basis. These approaches can share significant amounts of code amongst themselves. The only thing that is different is the entity that is being modeled and how the config changes are applied.

...

• Create a znode (or modify existing) under the required path with the configs that you want to update. Example, if you want to change quotas for client producer "Adi", add a path under /config/clientsproducers/Adi as show below.
• Create a sequential znode under "config/changes/config_change_XX". This will send a notification to all the watchers. The data within the change node should indicate what has changed i.e. topic config + topic name, client config + clientId or broker config + brokerId.
• The brokers process the changed configs.

Tooling will be provided within AdminUtils to perform all of these tasks.

Modeling Default values

In addition to topic/client/broker level overrides, we also need a mechanism to model default configuration. For example: say we have default quotas for all clients and we need to selectively override them on a per-client basis. The above approach doesn't provide a way to specify the default values. We can treat the defaults as a special path ca

The properties in this znode are applicable to all clients, topics and brokers respectively. Changing something here should affect all entities unless overridden explicitly.
 
/config/producers/__default
/config/consumers/__default
/config/topics/__default
/config/brokers/__default
 
Let's extend the example of quotas. Assume that we have a default quota of 5Mbytes per second per producer and we want to change it to 10M for the producer Adi. The default znode will look like this:
/config/producers/__default
{"version": 0, "config" : {quota=5M}

/config/producers/Adi
{"version": 0, "config" : {quota=10M}

Broker Configuration

This KIP only provides an approach to represent broker configuration within zookeeper. It does not discuss how these configs are actually applied within brokers. Here For Broker configs, the key can be the KafkaConfig property that we wish to override. All configs represented here should override the configs from the property file.

/config/brokers/0
{"version": 0, "config" : {log.retention.check.interval.ms=100000}

 This KIP provides a mechanism to change any broker config dynamically but identifying which configs can be changed and how the config changes should be applied within the brokers is out of scope. 

Config Change Notification

Currently, the change notification znode only contains the topic name. We need to add more information to distinguish whether this config change is for a topic, client or broker config. We need to distinguish producers and consumers separately because it is possible for a producer and consumer to have the same clientId. 

The notification data can be:
{"entity": "topic/producer/clientconsumer/broker", "value" : "topic_name/client_id/broker_id"}

...