Current state: "Under Discussion"
Discussion thread: here
JIRA: KAFKA-14903
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
MM2 supports a dynamic topic and group filtering mechanism in which the replicated topics/groups can change when the list of available topics/groups change, or the filter configuration is updated. Currently, there is no way to access the list of topics and groups replicated by MM2.
The closest options right now are:
To address this, MM2 should support a TopicListener (nested in MirrorSourceConnector) and a GroupListener (nested in MirrorCheckpointConnector) plugin, and notify this plugin when the list of replicated topics/groups changes. This will allow users to follow the current set of replicated topics and groups, even with the IdentityReplicationPolicy.
New interfaces in :connect:mirror - TopicListener and GroupListener
interface TopicListener extends Configurable, AutoCloseable {
void topicsChanged(Map<String, String> upstreamToDownstreamTopics);
}
interface GroupListener extends Configurable, AutoCloseable {
void groupsChanged(List<String> replicatedGroups);
}
New default implementations:
New configurations:
MirrorSourceConnector
MirrorCheckpointConnector (almost the same changes as in MirrorSourceConnector, but with groups)
No need for deprecation or migration, as it adds new capabilities, and the defaults of the new configurations do not change the current behavior. Existing users will not be impacted.
Unit tests should be sufficient to cover this feature.
As described in the motivation, the existing alternatives (using ReplicationPolicy or using MirrorMakerMetrics) does not fully fit the requirement.
Theoretically, the existing filter interfaces could be changed - instead of acting as predicates on a single topic/group, they could accept a set of topic/group names, and filter the set. This would allow the filter plugins to act as the "listeners" of the current topic/group list.
The drawback of this would be that filters would not be aware of the additional filtering rules of the MM2 Connectors (e.g. internal mm2 topics are filtered by default), and could potentially report false positives. Additionally, mixing different filter implementations with different "listener" implementations would become tedious, since instead of using composition with 2 different plugins, users would need to implement custom plugins to get the desired combination of filter and listener implementations.