Status
Current state: Under Discussion
Discussion thread: here
JIRA:
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
In many use cases it is required to drop a few specific Kafka headers but a set of headers whose names can also dynamically change (e.g. when used with some end-to-end-encryption libraries, tracing etc.). To prevent those headers following a special pattern (which for example may or may not comply with downstream system format) to be further forwarded/processed downstream (e.g.header forwarding in HTTP Sinks), this KIP proposes to add a wildcard/regexp matching feature to the DropHeaders Transform.
Public Interfaces
It is proposed to add new configuration is to the SMT org.apache.kafka.connect.transforms.DropHeaders to allow regexp-based matching of heades to be dropped:
Name: headers.pattern
Description:List of regular expressions to match of the headers to be removed.Type: boolean
Default: "$^" (match empty)
Proposed Changes
It is proposed to use regexp-matchers inside the apply method in addition to a set-based contains (existing implementation).
Compatibility, Deprecation, and Migration Plan
If the functionality is added with regexp-matching in addition to the existing implementation, no breaking will be done, no migration is required as the additional feature comes with a standalone configuration option
Rejected Alternatives
None.