...
- The user running MM2 has the following ACLs
- `Create` ACLs for topics on the source clusters to create
heartbeat
topics. - `Create` and `Alter/AlterConfigs` ACLs to create topics, create partitions, update topics' config and topics' ACLs on the destination clusters.
- `Create` ACLs for topics on the source clusters to create
- MM2 can bypass any existing Kafka resource management solutions that organizations runs and create/update resources outside this system.
These assumption wouldn't work for any organization that runs any sort of resource management or federated solutions where these systems are usually the only application allowed to initializing a client with `Create` and`Alter/AlterConfigs` ACLs. And no other teams/groups/applications are allowed to have this same level of ACLs to create such a client outside these systems.
...
As mentioned before the usage of AdminClient directly within MM2 simplify the resource management for MM2. However, it does create the following problems for any users who use IaC (Infra-as-Code), federated solutions, or have a capacity/budget planning system for Kafka destination clusters. Here is a list of potential undesired impact of MM2 bypass the organization ecosystem:
...
This KIP proposes a way for users to run MM2 with custom implementation for the Kafka resource manager in order to easily integrate MM2 with their ecosystem.
Public Interfaces
The KIP proposes adding add flexibility to how MM2 manage Kafka resources.
There're 2 potential solutions under discussion now:
Adding a new interface called KafkaResourceManager that defines how MM2 will create, modify, and list any Kafka topics and ACLs. MirrorMaker2’s original behaviour will be kept in DefaultResourceManager.
The implemented class can be overridden using the following configurations.
resource.manager.class
default value will be set toDefaultResourceManager
- or can be configured based on cluster aliases using
<cluster_alias>.resource.manager.class
- Allow MM2 to load different implementation of Admin interface
The implemented class can be overridden using the following configurations.
admin.class
default value will be set toAdminClient
- or can be configured based on cluster aliases using
<cluster_alias>.admin.class
The downsides of this proposal are
- The Admin interface has 86 methods, and MM2 depends on 13 of them; this means any custom implementation of the Admin interface will be humongous. The user will need to return "NotImplementedException" for 73 methods that MM2 does not use.
- MM2 interactions with AdminClient is not listed in one place, so it's bit hard to list what interfaces you need to implement to MM2.
- Keeping up with the AdminClient interface which is enormous.
Proposed Changes
There're 2 potential solutions under discussion now:
1. Add new interface for MM2 to manage resources KafkaResourceManager
KafkaResourceManager
interface will define how to list, create, update topics, topic configs, and ACLs.
...
DefaultResourceManager
will use AdminClient
to manage the resources. And default value for resource.manager.class
will be DefaultResourceManager
for backward compatibility.
2. Add configuration to MM2 to load different implementation of Admin interface
- MM2 will use implementation form
admin.class
to initialize admin client. - MM2 interactions with Admin interface need to be in one place to make it easier to list what interfaces need to be implemented when users provide their own Admin implementation
Compatibility, Deprecation, and Migration Plan
- When users upgrade an existing MM2 cluster they don’t need to change any of their current configuration as this proposal maintains the default behaviour for MM2.
Rejected Alternatives
- Manage creating and modifying Kafka topics and ACLs outside MM2 by building a separate tool that monitors the same set of topics as MirrorMaker2 and create/modify topics and ACLs once it detects configuration changes. The downsides with this are
- This will be a duplicate effort as MM2 already has all this logic implemented; it only needs to use a different client than AdminClient.
- MM2 still bypass any capacity safeguards the ecosystem would have in place.
- Any downtime with this tool will cause conflict between MM2 mirrored resources and management resource system
- This will be a duplicate effort as MM2 already has all this logic implemented; it only needs to use a different client than AdminClient.
- The Admin interface has 86 methods, and MM2 depends on 13 of them; this means any custom implementation of the Admin interface will be humongous. The user will need to return "NotImplementedException" for 73 methods that MM2 does not use.
- MM2 interactions with AdminClient is not listed in one place, so it's bit hard to list what interfaces you need to implement to MM2. Keeping up with the AdminClient interface which is enormous
- .