...
The following example defines two topic creation rules named "firstRule
" and "defaultRule
":
Code Block | ||||
---|---|---|---|---|
| ||||
topic.creation=firstRule,defaultRule topic.creation.firstRule.regex=MyPrefix.* topic.creation.firstRule.replication.factor=3 topic.creation.firstRule.partitions=5 topic.creation.firstRule.cleanup.policy=compact topic.creation.firstRule.min.insync.replicas=2 topic.creation.firstRule.unclean.leader.election.enable=false topic.creation.defaultRule.regex=.* topic.creation.defaultRule.replication.factor=3 topic.creation.defaultRule.partitions=1 topic.creation.defaultRule.cleanup.policy=compact topic.creation.defaultRule.min.insync.replicas=2 topic.creation.defaultRule.unclean.leader.election.enable=false |
...
This style of configuration properties is very similar to those defined for . These properties can appear in the connector's configuration in any order, but the order of the names in "topic.creation
" is important and defines the order in which the framework evaluates whether each rule applies to a topic with a given name. For example, if a new topic named "MyPrefixABC
" is to be created, the framework would first use the regular expression of the "firstRule
" to see if it matched the topic name "MyPrefixABC
". Because it does, the topic-specific settings defined in the properties beginning with "topic.creation.firstRule.
" would be used and passed to the connector for validation / overrides and ultimately used to create the topic. However, a topic named "XYZ
" would not match the "firstRule
" but would match the "defaultRule
", and thus the topic-specific settings defined in the configuration properties beginning with "topic.creation.defaultRule.
" would be used and passed to the connector for validation / overrides and ultimately used to create the topic.
...
Here is another example that shows how to use one rule for topics whose names begin with "MyPrefix
", and a second default rule that will result in connector failure if the framework needs to create topics because they don't yet exist and are used in source records. This provides a way to enforce that a source connector does not attempt to create unwanted topics.
...
To allow source connector implementations may the ability to validate or override some or all of these topic-specific settings, we will modify the following existing abstract class in the Kafka Connect public API:
- org.apache.kafka.connect.source.SourceTask
by adding a non-abstract method with the following signature that will by default simply return the input TopicSettings:
...