...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Although compression is not a new problem, it has continued to be an important research topic.

The integration and testing of new compression algorithms into Kafka currently requires significant code changes and rebuilding of the distribution package for Kafka.

This  KIP give Give users the ability to use different compressors without needing future changes in Kafka.

This proposal suggest adding a plugin class name option to the producer, broker and topic config.

The jars that implement the plugin can be added ad-hoc to the Kafka libs folder from the distribution package.

modifying the message format by adding a new attribute for compression.

Image Added

This KIP does not supercede KIP-390 and KIP-780.  It can build on top of KIP-780 as the scope of this KIP goes beyond the scope of KIP-390/KIP-780

Public Interfaces

Current compression is NOT changed. The existing codecs are not affected because their code path is NOT affected. 

...

This feature introduces a new option, 'pluggable.factory.plugin' to the producer, topic and broker configuration. The type of this option is a string, with default value of null

...

Proposed Changes

This option impacts doesn't change the following processes because pluggable compression is implemented as a complimentary new codec:

• Producer compresses the batch of user-given messages
• Broker decompresses the batch when validating messages
• Broker recompresses the batch when compression of producer different from broker
• Producer decompresses the batch of user-given messages

When the pluggable codec interface is used on the producer, consumer, the plugin jar will be , metadata is added to the kafka/libs directory to make message format to indicate the plugin class available in the classpath. The plugin classname will also be added as a property in producer and server properties files.name path.

Image Added

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

...

1. The current proposal does not have any impact on existing users. 
2. We are not changing the behavior of current compression. We are simply adding a new codec. 
3. If message format changes in Kafka then this implementation will need to be updated
4. Regression testing show pluggable interface did not affect default performance using Snappy as a plugin

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.