Status

Current state: "Under Discussion"  Accepted

Discussion thread: here 

JIRA: here 

...

Proposed Changes

This KIP proposes to migrate use the IncrementalAlterConfigs API for syncing topic configurations in MirrorMaker. The IncrementalAlterConfigs API has been around for several years since it was introduced in Kafka 2.3.0 and addresses the shortcoming of the AlterConfigs API. In order not to break the compatibility, we will introduce a new setting will be introduced to MirrorMaker to give that gives the option to enable and or disable IncrementalAlterConfigs API for syncing topic configurations. This new setting is expected to serve as a temporary measure until the next major release when the API the API is always used.

• A new configuration setting to MirrorMaker:
  • Name: use.incremental.alter.configs

• • Description: Deprecated. Whether to automatically use the IncrementalAlterConfigs API for syncing topic configurations. This configuration will be removed in Kafka 4.0 and the IncrementalAlterConfigs API will be used by default. Users should make sure that the target cluster is running at least Kafka 2.3.0 or later. 

• • Type: String

...

• • When set to "requested", MirrorMaker will use the IncrementalAlterConfigs for syncing topic configurations. If any request receives an error from an incompatible broker, it will fallback to the deprecated AlterConfigs API for the subsequent calls and log a WARN message e.g. "The target cluster <ALIAS> is not compatible with IncrementalAlterConfigs API. Therefore using deprecated AlterConfigs API for syncing topic configurations". Users would then need to restart or reconfigure to go back to using IncrementalAlterConfigs API. 
  • When explicitly set to "never", MirrorMaker will use the deprecated AlterConfigs API for syncing topic configurations.

• • When explicitly set to "required", MirrorMaker will use the IncrementalAlterConfigs API for syncing topic configurations. If it receives an error from an incompatible broker, the MirrorMaker will report this to the user and fail the connector.

From Kafka 4.0, MirrorMaker will use IncrementalAlterConfigs API for syncing topic configuration. 

If Currently if users delete a topic configuration in the source cluster, currently it would gets also be deleted from the target cluster due to the replace all behaviour of AlterConfigs API. However with IncrementalAlterConfigs, the topic configuration in the target cluster would potentially be left unchanged. Therefore the default topic configuration on the source cluster will explicitly be cleared from the target cluster via Delete operation when using IncrementalAlterConfigs API. 

...

• Extend ConfigPropertyFilter  class to have a new method called shouldReplicateSourceDefault which would will return false by default. If set to false while using IncrementalAlterConfigs API, the default configurations in the source cluster will explicitly be cleared from the target cluster using the Delete operation. User would will be able to configure this method to return true in order to include source cluster's default configurations in the replication. 

• Extend DefaultConfigPropertyFilter with a new property "use.defaults.from" that accepts "source" and or "target" values. This will allow users to control which cluster's defaults get applied on the target. The user can also use the existing "config.properties.exclude" property to exclude configuration from the replication regardless of which cluster's default is chosen. 

...