Page History

...

The configuration itself will be maintained in zookeeper. Currently I believe zookeeper has a structure like the following:

get /brokers/topics/log-cleaner-test-1485952437-0  /brokers/
  topics/
    my-topic => {"0":["0", "1"]}
    your-topic => {...}
{ "0": ["0"] }

This structure would be changed to something like the following:

/brokers/
  topics/
    my-topic/
      replicas => {0:[0,1], ...}
      config => {"log.retention.bytes": 12345,...}

(not sure if this is the best layout...?)

All nodes would watch these config nodes and when one changed all brokers would update the appropriate in-memory log instances with the new config.

Creating, altering, and deleting topics

Currently we have a CreateTopicCommand object that can be used to create topics. There is no corresponding delete. If this change were made to config we would also need the ability to change configs too.

I propose adding a new APIs to the protocol:

ModifyTopicRequest => Operation TopicName [PropName PropValue]
Operation => "CREATE" | "DELETE" | "ALTER"
TopicName => String
PropName => String
PropValue => String


ModifyTopicResponse => ErrorCode

Note that there is no attempt to encode the properties as fields in the protocol since it is assumed these may evolve quickly, instead they are just key-value pairs.

The existing tool would just call this API. The existing logic would move into KafkaApis.

Open Questions

1. I am not so sure about the zookeeper layout, need help thinking that through...
2. One nuance of config is how defaults are handled. One approach would be to snapshot the full set of defaults into zookeeper at topic creation time for any setting not specified. The other approach would be to only save in zookeeper what the user gives and dynamically inherit the defaults. The difference is that in the later case it is easier to make a global config change and have it apply to all topics that haven't specified otherwise. This would also result in less data stored in zookeeper. 
3. Should we maybe just move all config into zookeeper?