Page History

...

The following example defines two topic creation rules named "firstRule" and "defaultRule":

topic.creation=firstRule,defaultRule

...

topic.creation.firstRule.regex=MyPrefix.*

...

topic.creation.firstRule.replication.factor=3

...

topic.creation.firstRule.partitions=5

...

topic.creation.firstRule.cleanup.policy=compact

...

topic.creation.firstRule.min.insync.replicas=2

...

topic.creation.firstRule.unclean.leader.election.enable=false

...

topic.creation.defaultRule.regex=.*

...

topic.creation.defaultRule.replication.factor=3

...

topic.creation.defaultRule.partitions=1

...

topic.creation.defaultRule.cleanup.policy=compact

...

topic.creation.defaultRule.min.insync.replicas=2

...

topic.creation.defaultRule.unclean.leader.election.enable=false

This style of configuration properties is very similar to those defined for . These properties can appear in the connector's configuration in any order, but the order of the names in "topic.creation" is important and defines the order in which the framework evaluates whether each rule applies to a topic with a given name. For example, if a new topic named "MyPrefixABC" is to be created, the framework would first use the regular expression of the "firstRule" to see if it matched the topic name "MyPrefixABC". Because it does, the topic-specific settings defined in the properties beginning with "topic.creation.firstRule." would be used and passed to the connector for validation / overrides and ultimately used to create the topic. However, a topic named "XYZ" would not match the "firstRule" but would match the "defaultRule", and thus the topic-specific settings defined in the configuration properties beginning with "topic.creation.defaultRule." would be used and passed to the connector for validation / overrides and ultimately used to create the topic.

...

Here is another example that shows how to use one rule for topics whose names begin with "MyPrefix", and a second default rule that will result in connector failure if the framework needs to create topics because they don't yet exist and are used in source records. This provides a way to enforce that a source connector does not attempt to create unwanted topics.

topic.creation=firstRule,failRule

...

topic.creation.firstRule.regex=MyPrefix.*

...

topic.creation.firstRule.replication.factor=3

...

topic.creation.firstRule.partitions=5

...

topic.creation.firstRule.cleanup.policy=compact

...

topic.creation.firstRule.min.insync.replicas=2

...

topic.creation.firstRule.unclean.leader.election.enable=false

...

topic.creation.failRule.regex=.*

...

topic.creation.failRule.policy=fail

Java API

To allow source connector implementations may the ability to validate or override some or all of these topic-specific settings, we will modify the following existing abstract class in the Kafka Connect public API:

• org.apache.kafka.connect.source.SourceTask

by adding a non-abstract method with the following signature that will by default simply return the input TopicSettings:

...