Status

Current state: Under Discussion

Discussion thread: here

JIRA: here

Motivation

MirrorMaker 2 uses AdminClient  directly to create topics, create new topic partitions and sync configurations like topic configs and ACLs by enabling sync.topic.configs.enabled  and sync.topic.acls.enabled. The current design runs with 2 main assumptions:

...

An example for such a watcher is Strimzi K8S Topic Operator. Which recosiliate Topic and Topic's config between ZK and Topic Resources on K8S however it doesn't have the same solution for UserOperator which make consumer group can't migrate without these ACLs created first. (more details below in The downsides of MM2 use AdminClient directly in the motivation)

The downsides of MM2 use AdminClient directly

As mentioned before the usage of AdminClient directly within MM2 simplify the resource management for MM2. However, it does create the following problems for any users who use IaC (Infra-as-Code), federated solutions, or have a capacity/budget planning system for Kafka destination clusters. Here is a list of potential undesired impact of MM2 bypass the organization ecosystem:

...

This KIP proposes a way for users to run MM2 with custom implementation for the Kafka resource manager in order to easily integrate MM2 with their ecosystem. 

Public Interfaces

The KIP proposes adding flexibility to how MM2 manage Kafka resources. By allowing MM2 to load custom implementation of Admin interface.

...

• forwarding.admin.class default value will be set to ForwardingAdmin with KafkaAdminClient as delegate.
• or can be configured based on cluster aliases using <cluster_alias>.forwarding.admin.class

Proposed Changes

• ForwardingAdmin class 

public class ForwardingAdmin implements Admin {
    private final Admin delegate;

    public ForwardingAdmin(Map<String, Object> config, Admin delegate) {
        this.delegate = delegate;
    }
    
    public ForwardingAdmin(final Admin delegate) {
        this.delegate = delegate;
    }
    
    @Override
    public CreateTopicsResult createTopics(Collection<NewTopic> newTopics, CreateTopicsOptions options) {
        return delegate.createTopics(newTopics, options);
    }
    
    // override rest of Admin interface to use delegate...
}

...

• When users upgrade an existing MM2 cluster they don’t need to change any of their current configuration as this proposal maintains the default behaviour for MM2.

Rejected Alternatives

• Adding a new interface called KafkaResourceManager ("was the original proposal") that defines how MM2 will create, modify, and list any Kafka topics and ACLs. MirrorMaker2’s original behaviour will be kept in DefaultResourceManager.

  And override the implementation using the following configurations.

  • resource.manager.class default value will be set to DefaultResourceManager
  • or can be configured based on cluster aliases using <cluster_alias>.resource.manager.class
  The downside with this is it may create confusing for users between Admin and ResourceManager 
• Manage creating and modifying Kafka topics and ACLs outside MM2 by building a separate tool that monitors the same set of topics as MirrorMaker2 and create/modify topics and ACLs once it detects configuration changes. The downsides with this are
  
  1. This will be a duplicate effort as MM2 already has all this logic implemented; it only needs to use a different client than AdminClient.
     
  2. MM2 still bypass any capacity safeguards the ecosystem would have in place. 
  3. Any downtime with this tool will cause conflict between MM2 mirrored resources and management resource system.

...