...

Discussion thread: here

JIRA: here

Motivation

MirrorMaker 2 uses AdminClient directly to create topics and sync configurations like topic configs and ACLs by enabling sync.topic.configs.enabled  and sync.topic.acls.enabled , while this approach simplifies the resource management for MM2, it does create the following problems for users who have their systems or tooling to manage Kafka resources:

1. External Kafka resource management tool/system may override the same configurations that MM2 synced previously. 
2. Create/Modify resources using AdminClient directly lead to a lack of auditing and tracing resources.

In use-cases where users have their own centralized system to manage Kafka topics and ACLs, they should easily integrate MirrorMaker2 with their internal ecosystem. 

Currently, there are a couple of options to deal with these problems, 

1. Drop mirrored topics from a centralized system and accept the risk of not tracing these topics. This direction will avoid conflict between topics monitored by MM2 and the centralized system for resource
managment
2. management.
3. Creating topics up-front and disabling features like sync.topic.configs.enabled , sync.topic.acls.enabled.

The current approach also

...

assumes that the user

...

running MM2 has the Admin right to create topics, which is only

...

valid if the user who

...

runs MM2

...

also

...

manages both source and destination clusters. 

This KIP

...

proposes a way for users to run MM2 with custom implementation for the Kafka resource manager. 

Public Interfaces

The KIP is proposing proposes adding a new interface called KafkaResourceManager that define defines how MM2 will create, modify, and list any Kafka topics and ACLs. MirrorMaker2’s original behaviour will be kept in DefaultResourceManager.

The implemented class can be overridden using the following configurations.

• resource.manager.class default value will be set to DefaultResourceManager
• or can be configured based on cluster aliases using <cluster_alias>.resource.manager.class

...

KafkaResourceManager interface will define how to list, create, update topictopics, topic configs, and ACLs.

...

• When users upgrade an existing MM2 cluster they don’t need to change any of their current configuration as this proposal maintains the default behaviour for MM2.

Rejected Alternatives

• Manage creating and modifying
kafka
• Kafka topics and ACLs outside MM2 by building a separate tool that
monitor
• monitors the same set of topics as MirrorMaker2 and create/modify topics and ACLs once it detects configuration changes. The problem is this will be a duplicate effort as MM2 already has all this logic implemented; it only needs to use a different client than AdminClient.
Provide a way for
• Allow MM2 users to provide a custom implementation of the Admin interface instead of adding a new interface. The problem with this one is that the Admin interface has 86 methods, and
MM2 
• MM2 depends on 13 of them; this means any custom implementation of the Admin interface will be humongous
as
• . The user will need
to return
• to return dummy data for 73
method
• methods that MM2
doesn't
• does not use.