...
MirrorMaker should not clear topic configurations that are excluded from replication.
In addition, source and target topic configurations could have different default values. Currently, Mirrormaker overwrites configurations on target topic even if the configurations on the source topic have not been changed but happened to have different default values. However, user may only want to sync configurations that they have changed but leave the ones with default values.
Public Interfaces
This KIP proposes to migrate to incrementalAlterConfigs
API for syncing topic configurations in MirrorMaker. The incrementalAlterConfigs
API has been around for several years since it was introduced in Kafka 2.3.0 and addresses the shortcoming of the deprecated alterConfigs
API. In order not to break the compatibility, we will introduce a new setting to MirrorMaker to give the option to enable incrementalAlterConfigs
API for syncing topic configuraMirrorMaker syncs topic configurations whether they are filter or nottions. This new setting is expected to serve as a temporary measure until the next major release when the API is always used.
It also proposes to add an option to use the target cluster default values for the target topic configurations . By default it would use the source cluster default values as the default for which is the current behaviour.
Proposed Changes
1) Add a new method updateTopicConfigsUsingIncrementalAlterConfigs() to MirrorSourceConnector.java
which takes a list topic configurations as a parameter and calls incrementalAlterConfigs()
to the target cluster.
2) Add an extension
3) Add a new configuration setting to MirrorMaker:
...
When explicitly set to true, MirrorMaker will use incrementalAlterConfigs
API for syncing topic configurations. If it receives an error from an incompatible broker, the MirrorMaker will report this to the user and fail the connector.
34) Update syncTopicConfigs()
in MirrorSourceConnector.java
to call the updateTopicConfigsUsingIncrementalAlterConfigs()
instead of updateTopicConfigs()
if enable.incremental.alter.configs
is set to true.
45) From Kafka 4.0, syncTopicConfigs()
will only call updateTopicConfigsUsingIncrementalAlterConfigs()
by default. The existing updateTopicConfigs() method
will be removed.
...