THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This proposal defines a flexible way to specify topic-specific settings for new topics by introducing the concept of topic creation rules that are specified for a source connector entirely through connector-level configuration properties. Each topic creation rule has a name, a regular expression that is used to determine whether the rule applies to a particular topic, and the topic-specific settings that would be used when creating the new topic. When Kafka Connect determines that a new topic may need to be created, it will find and use only the first topic creation rule that applies to the new topic's name and then use the , pass that rule's topic-specific settings specified by that rule. Kafka Connect will then use a new Java API to let the source connector validate and override these topic-specific settingsfor validation / override, and then will use these final topic-specific settings when creating the new topic.
...
The following example defines two topic creation rules named "firstRule" and "defaultRule" that are applied in that order:
topic.creation=firstRule,defaultRuletopic.creation.firstRule.regex=MyPrefix.*topic.creation.firstRule.replication.factor=3topic.creation.firstRule.partitions=5topic.creation.firstRule.cleanup.policy=compacttopic.creation.firstRule.min.insync.replicas=2 topic.creation.firstRule.unclean.leader.election.enable=false topic.creation.defaultRule.regex=.*topic.creation.defaultRule.replication.factor=3topic.creation.defaultRule.partitions=1topic.creation.defaultRule.cleanup.policy=compacttopic.creation.defaultRule.min.insync.replicas=2topic.creation.defaultRule.unclean.leader.election.enable=false |
These properties can appear in the connector's configuration in any order, but the order of the names in topic.creation is important and defines the order in which the framework evaluates whether each rule applies to a topic with a given name. For example, if a new topic named "MyPrefixABC" is to be created, the framework would first use the regular expression of the "firstRule" to see if it matched the topic name "MyPrefixABC". Because it does, the topic-specific settings defined in the topic.creation.firstRule.* properties would be used and passed to the connector for validation / overrides and ultimately used to create the topic. However, a topic named "XYZ" would match the defaultRule, and thus the topic-specific settings defined in the topic.creation.defaultRule.* properties would be used and passed to the connector for validation / overrides and ultimately used to create the topic.
This style of configuration properties is very similar to those defined for Single Message Transforms.
Java API
To allow source connector implementations may the ability to validate or override some or all of these topic-specific settings, we will modify the following existing abstract class in the Kafka Connect public API:
...