Status

Current state: Under Discussion

Discussion thread: 

JIRA: 

Released: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

We have 2 methods in AdminClient  for updating config, alterConfigs  and incrementalAlterConfigs , the former has many restrictions and has been deprecated from 2.3.0. However, It's still used in ConfigCommand  to update broker config and is resulting in a bug Unable to render Jira issues macro, execution error. , so I'm inclined to move it to incrementalAlterConfigs  and provide a flag to still use alterConfigs  for new client to interact with old servers. There are some other reasons for this change:

1. alterConfigs has been deprecated and will be removed in a future release;
2. we are using incrementalAlterConfigs to change user/topic/client-metrics configs, it would be benefit to unify broker configs;
3. incrementalAlterConfigs is more convenient especially for updating configs of list data type, such as "leader.replication.throttled.replicas" 

4. We are forced to pass all sensitive configs to update broker configs when using alterConfigs  with the current cli tool because sensitive config values are never returned to the client.

Note that I'm only changing the way we updating broker configs, user/topic/client-metrics configs are already being updated using incrementalAlterConfigs

Public Interfaces

kafka-configs.sh

1. Adding new options --enable-incremental , which is used for client to connect to new servers after 2.3.0 when updating broker configs. 

Proposed Changes

1. --enable-incremental is only makes sense when used together with (--alter) and (--broker <broker-id>)/(--entity-types brokers), or it will be ignored.
2. --enable-incremental is only makes sense when used together with (--bootstrap-server)/(bootstrap-controller), we are leaving zookeeper case unchanged.
3. use AdminClient.incrementalAlterConfigs if --enable-incremental flag is specified when alter broker configs, which will fail if the broker is before 2.3.0.
4. -enable-incremental is deprecated and will be removed in the future together with AdminClient.alterConfigs, and  AdminClient.incrementalAlterConfigs will be the only choice 

here is an example of changing the broker configs twice:

1. set log.cleaner.threads=2
2. set background.threads=1

in old version, or in the new version without --enable-incremental,  here are what happened:

1. Use kafka-configs.sh to set log.cleaner.threads=2, the client will fetch all broker configs and get a empty properties {}, merge it with the delta and get {log.cleaner.threads=2}, send it to server using AdminClient.alterConfigs(), the server will persist it to metadata storage.
2. Use kafka-configs.sh to set background.threads=1, the client will fetch all broker configs and get {log.cleaner.threads=2}, merge it with the delta and get {log.cleaner.threads=2, background.threads=1 }, send it to server using AdminClient.alterConfigs(), the server will persist it to metadata storage.

in new case with --enable-incremental, or when we removed the deprecated AdminClient.alterConfigs, here are what happened:

1. Use kafka-configs.sh to set log.cleaner.threads=2, the client will send it to server using AdminClient.incrementalAlterConfigs(), the broker will merge the old snapshot( {} ) with delta and save the new snapshot( {log.cleaner.threads=2} ) to metadata storage.
2. Use kafka-configs.sh to set background.threads=1, the client will send it to server using AdminClient.incrementalAlterConfigs(), the broker will merge the old snapshot( {log.cleaner.threads=2} ) with delta and got the new snapshot ({log.cleaner.threads=2, background.threads=1 }) to metadata storage.

The result is expected to be the same for both way, except that we can avoid some issues for AdminClient.alterConfigs().

Compatibility, Deprecation, and Migration Plan

There will be a backward compatibility problem here since incrementalAlterConfigs  is added from 2.3.0, we add a --disable-incremental option for older client to do smooth migration.

Test Plan

new client tool with older server can be tested locally and using test cases.

Documentation Plan

There are some api changes and we should document about kafka-configs.sh change list

Rejected Alternatives

1. We fallback to using alterConfigs automatically if incrementalAlterConfigs is not supported. We need to unify the semantics if we want to do this heuristically, so we choose to give this privilege to users by adding --disable-incremental.
2. we keep existing scripts unchanged and add an "--enable-incremental" flag for new servers in Kafka 3.X, and remove it in Kafka 4.X. In this way we move the incompatible process to 4.X, I'm inclined to do it as soon as possible since it's inevitable.
3. We just forward all invocations of alterConfigs to incrementalAlterConfigs. Similar to first one, we need to unify the semantics, it's not recommend since alterConfigs is deprecated and we just leave it as it was.