Status

Current state: Discarded, and superseded by KIP-215: Add topic regex support for Connect sinks

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

For now we provide topic names in a properties file as following:

topics=test1,test2

If there are a few topics to deal with, It's ok to list all the topics in a comma-seperated way. But when you have to deal with hundreds or thousands of topics, it doesn't help. In those situations, we would like to use a wildcard expression to specify topics.

Public Interfaces

• client configuration file

Proposed Changes

• add an aditional option "topics.regex"
• a user can only specify one of "topics" or "topics.regex"
• if a user specifies "topics" in their config file, it works the same way as the current version. 
• if a user specifies "topics.regex", it can handle all the topics matching the regular expressions.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
  • N/A
• If we are changing behavior how will we phase out the older behavior?
  • N/A
• If we need special migration tools, describe them here.
  • Not needed
• When will we remove the existing behavior?
  • N/A

Rejected Alternatives

It is possible to update the interpretation of the existing option "topics" in a way that our users can specify reglar expressions there. But since there are some special characters in the regular expression that can be used in topic names, eg. "." and "-",  we need some workaround. Also this aproach has some effects on existing users.