Table of Contents |
---|
This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: "Under Discussion"
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]
JIRA:
Jira | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
In many use cases you might not only want it is required to drop a few specific Kafka headers but a set of headers whose names can also dynamically change (e.g. when used with some end-to-end-encryption libraries, tracing etc.). To prevent those headers following a special pattern (which for example may or may not comply with downstream system format) to be further forwarded/processed downstream (e.g.header forwarding in Http HTTP Sinks), I suggest this KIP proposes to add a wildcard/regexp matching feature to the apply method instead of a set-based contains. Link to the relevant code:
https://github.com/apache/kafka/blob/7b5d640cc656443a078bda096d01910b3edfdb37/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/DropHeaders.java#L54
One common example is forwarding of encryption-headers that contain byte arrays to a HTTP server via HTTP Sinks which will not be accepted by the HTTP server.
Public Interfaces
DropHeaders Transform.
Public Interfaces
It is proposed to add new configuration is to the The SMT org.apache.kafka.connect.transforms.DropHeaders is to allow regexp-based matching of headers to be extended. Depending on the final decision on the implementation, an optional property might be added.
Proposed Changes
dropped:
Name: headers.patterns
Description:List of regular expressions to match of the headers to be removed.
Type: String (List)
Default: "$^" (match empty)
Proposed Changes
It is proposed to use regexp-matchers inside The suggestion is to add regexp matching to the apply method instead of *or in addition to ) a set-based contains .
An example implementation was done already for the option replacing the set-based method - link to PR: https://github.com/apache/kafka/pull/14536
Having backward compatibility in mind we should discuss whether to change the existing implementation an replace with a regexp-matcher or to add another configuration specifically for just using regular expressions(existing implementation).
Compatibility, Deprecation, and Migration Plan
If the functionality is added with regexp-matching in addition to the existing implementation, no breaking changes. In case the contains-implementation is replaced, there might be breaking changes in rare cases of headers that contain characters which are use in regular expressions.
Test Plan
Unit-Testing of the SMT with additional tests covering backwards-compatibility.will be done, no migration is required as the additional feature comes with a standalone configuration option
Rejected Alternatives
None.