Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As of 3.0.0, the user can configure the client (i.e., producer, consumer, and admin) settings used by MirrorMaker 2 with {cluster}.[producer|consumer|admin].<configuration-property> syntax. The problem is that only the common client settings like bootstrap servers, SSL configurations, etc., are configurable. For this reason, the producer-specific options or consumer-only settings are not changeable - that is, the user configurations are silently ignored, dislike to the documentation.

us-east.producer.acks = all  // ignored
us-west.consumer.max.poll.interval.ms = 120000 // also ignored

(Note: for the proof, see here.)

Because of that, those values are always fixed to the defaults and it causes some serious problems of MirrorMaker 2's functionality, like:

• Case 1: Since the producer's acks configuration is fixed to its default, '1', the replicated message may be lost if the leader broker crashes before the record is replicated.
• Case 2: Since the consumer's max.poll.interval.ms configuration is fixed to its default, 5 minutes, we can't reduce the time for rebalancing.

At first glance, this problem looks like a simple bug. However, this problem is more complicated than expected. The documentation states:

MirrorMaker is based on the Kafka Connect framework. Any Kafka Connect, source connector, and sink connector settings as described in the documentation chapter on Kafka Connect can be used directly in the MirrorMaker configuration, without having to change or prefix the name of the configuration setting.

Most of the default Kafka Connect settings work well for MirrorMaker out-of-the-box, with the exception of tasks.max.

It raises the following issues:

1. As of present, Kafka connect allows the users to override the client settings with [producer|consumer].override.<configuration-property> property if connector.client.config.override.policy is All (default). However, MirrorMaker 2 does not support it. This is not only inconsistent but also confusing for the users.
2. Dislike the other connectors, MirrorMaker 2 supports standalone mode, which lacks connector.client.config.override.policy; then, how can we deal with [producer|consumer].override.<configuration-property> in standalone mode?
3. No preference is defined between [producer|consumer].override.<configuration-property> and MirrorMaker 2 specific, {cluster}.[producer|consumer|admin].{configuration-property} syntax.

present, MirrorMaker 2 (aka MM2) 's client configurtaion feature has some problems:

1. The replication-level client configuration works only to the common properties like bootstrap.servers, security.protocol, ssl, sasl, etc; that is, a configuration like 'A→B.producer.batch.size' is ignored.
   1. Also, which admin client is affected by the replication-level configuration like A→B.admin.retry.backoff.ms is unclear; MM2 uses two admin clients for both upstream and downstream clusters, respectively.
2. MM2 is based on Kafka Connect framework's connector; Since MM2 Connectors (MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector) are source connectors, they use producer instance created by Kafka Connector, which uses 'producer.override.{property-name}' in connector configuration; But, 'target.producer.{property-name}' are not automatically applied to 'producer.override.{property-name}' so not actually applied to producer instance.
3. MM2 requires to define the 'bootstrap.servers' of the clusters in cluster-level, like 'A.bootstrap.servers' or 'B.bootstrap.servers'; but it also allows to override them in cluster-level and replication-level configs, like 'A.producer.bootstrap.servers' or 'A→B.consumer.bootstrap.servers'; actually these configurations are not used but, it would be better to ignore it and give a warning.

For the proof of the problem, please refer hereFor the reasons above, this problem can't be fixed by simply filling up the omitted gaps; it requires finding a reasonable configuration scheme that is simple and consistent with both Kafka Connect and MirrorMaker 2 specific client configuration syntax.

Public Interfaces

This proposal does not introduce any new public interface or configurable settings; it only makes the existing ones (which are currently ignored by MirrorMaker 2) working by providing consistent, easy-to-understand configuration rules.

Proposed Changes

Here are the new rules for client config overriding policy:

1. MirrorMaker 2 specific syntax precedes Kafka Connect standard; that is, {cluster}.[producer|consumer].<configuration-property> always overrides [producer|consumer].override.<configuration-property>.
2. If MirrorMaker 2 is running in standalone mode, it fallbacks to the default value of connector.client.config.override.policy, i.e., All.
3. If connector.client.config.override.policy is not set to All, all [producer|consumer].override.<configuration-property> values are ignored.

...

1. The replication-level client configuration now works; A->B.producer.{property-name} and A->B.admin.{property-name} are applied to target cluster clients and A->B.consumer.{property-name} is applied to source cluster clients.
2. In standalone mode, target.producer.{property-name} is automatically copied to producer.override.{property-name}. In connector mode, the user should manually configure them, and the documentation will explicitly mention it.
3. Cluster-level and replication-level client bootstrap servers configurations will be ignored and show a warning message.

Compatibility, Deprecation, and Migration Plan

The users without any specific overrides will not experience a change. But those who are trying to override the clients' defaults will see the configurations are now working.

Rejected Alternatives

Don't support [producer|consumer].override.<configuration-property> syntax

It not only breaks the consistency with the other connectors but also contradicts what the current documentation states.

Support [producer|consumer].override.<configuration-property> but not allow it in standalone mode

In this case, the configuration that works in connector mode does not work in standalone mode - breaking consistency between two modes.

Make [producer|consumer].override.<configuration-property> preferable to MirrorMaker 2 specific configurations

This approach breaks the consistency between producer/consumer/admin, since there is no admin.override.<configuration-property> configuration. (Please note that [producer|consumer].override.<configuration-property> is allowed just for the consistency with Kafka Connect standard; using this syntax is not recommended.)

Since this proposal only makes currently not working settings work, there is no deprecation or migration plan.

Rejected Alternatives

NoneRejecting all the possible alternatives above, only the rules described in the Proposed Changes section remain.