You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-12379

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

To provide offset translation, MirrorMaker2 uses the mm2-offset-syncs topic. This topic is populated by the Source connector and then used by the Checkpoint connector to convert consumer group offsets. This topic is created on the source cluster. It means that when mirroring records from a source cluster to a target cluster, MirrorMaker2 requires write permission on the source cluster (at least for mm2-offset-syncs).

In secured environments, it would be preferrable to only grant reader access to the source cluster to MirrorMaker2.

As this topic is internal to MirrorMaker2 and never exposed to users we could make its location configurable by administrators. This would allow moving that topic to the target cluster if needed and only grant reader access to the source cluster.

Public Interfaces

The proposal is to introduce a new setting to MirrorMaker2 to control the location of the offset syncs topic.

Name: offset-syncs.topic.location
Doc: The location of the offset-syncs topic.
Type: String
Default: source
Valid values: source, target
Importance: Low

In dedicated mode, it can be used like:

clusters = us-east, us-west
us-east->us-west.enabled true
us-east->us-west.offset-syncs.topic.location = target


In Connect distributed mode, this setting needs to be set on both the Source and Checkpoint connectors.

Proposed Changes

Based on the new setting, the Source and Checkpoint connectors will use either the source cluster (default) or the target cluster for offset syncs.

Compatibility, Deprecation, and Migration Plan

The default value preserves the current behaviour so there is no impact on existing users.

Rejected Alternatives

None

  • No labels