THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- Core message size reduction
- remove overhead of 4 bytes for KeyLength when no headers using attributes bit
- reduce overhead of 4 bytes for KeyLength by using variable length encoded int
- reduce overhead of 4 bytes for ValueLength by using variable length encoded int
- Broker side interceptors
- with headers we could start introducing broker side message interceptors to append meta data or handle messages
- Single Record consumer API
- There is many uses cases where single record consumer/listener api is more user friendly - this is evident by the fact spring kafka have already created a wrapper, it would be good to support this natively.
Rejected Alternatives
https://cwiki.apache.org/confluence/display/KAFKA/Headers+Value+Message+Wrapper
Map<String, String> Headers added to the ProducerRecord
...
- Adds the ability for producers to set standard header key=value string value pairs
- No incompatible client api change (only new methods)
- Allows users to specify the serialisation of the key=value map (String(&=), JSON, AVRO).
- Provides a standardised interface to eco systems of tools can grow around the feature
- Disadvantages
- Change to the message object
- String key cause a large key, this can cause a message payload thats of 60bytes to be dwarfed by its headers
- String value again doesn't allow for compact values, and restricts that a value must be a String
- Not able to use headers on the broker side with custom serialisation
Status Quo - Keep Custom Value Message Wrapper - Message<H, P>
This concept is the current defacto way many users are having to temporally deal with the situation, but has some core key issues that it does not resolve.
- Benefits
- This will cause no broker side changes , and or message format changesto handle the message
- Disadvantages
- This would not work with compaction where headers are needed to be sent on delete record which then would not deliver on many of the requirements.
- Couple Serialisation and Deserialisation of the value for both the header and payload.
...