This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: "Under Discussion"

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]

JIRA: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In many use cases

...

it is required to drop a few specific Kafka headers but a set of headers whose names can also dynamically change (e.g. when used with some end-to-end-encryption libraries, tracing etc.). To prevent those headers following a special pattern (which for example may or may not comply with downstream system format) to be further forwarded/processed downstream (e.g.header forwarding in

...

HTTP Sinks),

...

this KIP proposes to add a wildcard/regexp matching feature to the

...

DropHeaders Transform.

Public Interfaces

It is proposed to add new configuration is to the

One common example is forwarding of encryption-headers that contain byte arrays to a HTTP server via HTTP Sinks which will not be accepted by the HTTP server.

Public Interfaces

The SMT org.apache.kafka.connect.transforms.DropHeaders is to allow regexp-based matching of headers to be extended. Depending on the final decision on the implementation, an optional property might be added.

Proposed Changes

dropped:

Name: headers.patterns

Description:List of regular expressions to match of the headers to be removed.

Type: String (List)

Default: "$^" (match empty)

Proposed Changes

It is proposed to use regexp-matchers inside

...

the apply method

...

in addition to

...

a set-based contains

...

An example implementation was done already for the option replacing the set-based method - link to PR: https://github.com/apache/kafka/pull/14536

...

(existing implementation).

Compatibility, Deprecation, and Migration Plan

If the functionality is added with regexp-matching in addition to the existing implementation, no breaking changes. In case the contains-implementation is replaced, there might be breaking changes in rare cases of headers that contain characters which are use in regular expressions. 

Test Plan

Unit-Testing of the SMT with additional tests covering backwards-compatibility.will be done, no migration is required as the additional feature comes with a standalone configuration option

Rejected Alternatives

None.