THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Table of Contents |
---|
Status
Current state: Under Discussion Accepted
Discussion thread: here
JIRA: KAFKA-2511
...
Code Block | ||
---|---|---|
| ||
MessageAndOffset => MessageSize Offset Message MessageSize => int32Message Offset => int64 <------ CHANGE: Base offset for wrapper message of compressed message, relative offset for inner compressed message. MessageSize => int32 Message => Crc MagicByte Attributes Timestamp KeyLength Key ValueLength Value Crc => int32 MagicByte => int8 Attributes => int8 KeyLength => int32 Key => bytes ValueLength => int32 Value => bytes |
...
- Set message.format.version=0 on brokers. (Broker will write MessageAndOffset V0 to disk)
- Create internal ApiVersion 0.9.0-1** which uses ProducerRequest V2 and FetchRequest V2.
- Configure the broker to use ApiVersion 0.9.0 (ProduceRequest V1 and FetchRequest V1).
- Do a rolling upgrade of the brokers to let the broker pick up the new code supporting ApiVersion 0.9.0-1.
- Bump up ApiVersion of broker to 0.9.0-1. to let the broker use FetchRequest V2 for replication.
- Upgraded brokers support both ProducerRequest V2 and FetchRequest V2 which uses magic byte 1 for MessageAndOffset.
- When broker sees a producer request V1 (MessageAndOffset = V0), it will decompress the message, assign offsets using absolute offsets and NOT re-compress the message.
- When broker sees a producer request V2 (MessageAndOffset = V1), it will decompress the message, assign offsets using absolute offsets and do re-compression. i.e. down-convert the message format to MessageAndOffset V0.(This is no worse than what the brokers are currently doing.)
- When broker sees a fetch request V1 (Supporting MessageAndOffset = V0), because the data format on disk is MessageAndOffset V0, it will use the zero-copy transfer to reply with fetch response V1 with MessageAndOffset V0.
- When broker sees a fetch request V2 (Supporting MessageAndOffset = V0, V1), because the data format on disk is MessageAndOffset V0, it will use zero-copy transfer to reply with fetch response V2 with MessageAndOffset V0.
- When broker sees a producer request V1 (MessageAndOffset = V0), it will decompress the message, assign offsets using absolute offsets and NOT re-compress the message.
- Upgrade consumer to send FetchRequest V2.
- Upgrade producer to send ProducerRequest V2.
...