...
Currently Mirrormaker uses deprecated alterConfigs
API when syncing topic configurations for broker compatibility to 0.11.0. However, the deprecated alterConfigs()
API API replaces any existing configuration with the new configuration. Due to this, additional configurations set on a remote topic get cleared up when MirrorMaker syncs topic configurations even if the configurations are filteredwhether they are filter or not. For example this prevents running Cruise Control on the target cluster as it may set follower.replication.throttled.replicas and leader.replication.throttled.replicas.
...
This KIP proposes to migrate to incrementalAlterConfigs
API for syncing topic configurations in MirrorMaker. The incrementalAlterConfigs
API API has been around for several years since it was introduced in Kafka 2.3.0 and addresses the shortcoming of the deprecated alterConfigs
API. In order not to not break the compatibility, we will introduce a new configuration setting to MirrorMaker connector to give the option to enable incrementalAlterConfigs
API for syncing topic configurations. This new configuration setting is expected to serve as a temporary measure until the next major release when the API is always used.
...
When set to false, MirrorMaker will keep using deprecated alterConfigs
API for syncing topic configurations.
When explicitly set to true, MirrorMaker will use incrementalAlterConfigs
API API for syncing topic configurations. If it receives an error from an incompatible broker, the MirrorMaker will report this to the user and fail the connector.
...
4) From Kafka 4.0, syncTopicConfigs()
will only call updateTopicConfigsUsingIncrementalAlterConfigs()
by default. The existing updateTopicConfigs()
will method
will be removed.
Compatibility, Deprecation, and Migration Plan
...
(Is it necessary to add system tests?)
Rejected Alternatives
- We could have changed the current updateConfigs() method to call
describeConfigs
API beforealterConfigs
so that it had an incremental mode. This would have avoided migrating to incrementalAlterConfigs API and adding a new configuration setting to MirrorMaker. Howeverthe migration, however, the topics configurations could change betweendescribeConfigs
andalterConfigs
calls so we could still end up applying syncing incorrect topic configurations. We would need to stop using the the deprecated API at some point anyway.