Status

Current state: ["Under Discussion"]

Discussion thread: here

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Give users the ability to use different compressors without needing future changes in Kafka.

This proposal suggest adding a plugin class name option to the producer, broker and topic config.

The jars that implement the plugin can be added ad-hoc to the Kafka libs folder from the distribution package.

Public Interfaces

This feature introduces a new option, 'pluggable.factory.plugin' to the producer, topic and broker configuration. The type of this option is a string, with default value of null

compression.type=pluggable

pluggable.factory.plugin=<pluginClassname path>

The following public packages are affected: 

• org/apache/kafka/common

• org/apache/kafka/clients/producer

• org/apache/kafka/clients/consumer 

Proposed Changes

This option impacts the following processes:

• Producer compresses the batch of user-given messages
• Broker decompresses the batch when validating messages
• Broker recompresses the batch when compression of producer different from broker
• Producer decompresses the batch of user-given messages

When pluggable codec is used on the producer, consumer, the plugin jar will be added to the kafka/libs directory to make the plugin class available in the classpath. The plugin classname will also be added as a property in producer and server properties files.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.