THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- Set message.format.version=0 on brokers. (Broker will write MessageAndOffset V0 to disk)
- Create internal ApiVersion 0.9.0-1** which uses ProducerRequest V2 and FetchRequest V2.
- Configure the broker to use ApiVersion 0.9.0 (ProduceRequest V1 and FetchRequest V1).
- Do a rolling upgrade of the brokers to let the broker pick up the new code supporting ApiVersion 0.9.0-1.
- Bump up ApiVersion of broker to 0.9.0-1
- Do a rolling bounce of the brokers to let the broker use FetchRequest V2 for replication.
- Bump up ProducerRequest and FetchRequest version to V2, which supports both MessageAndOffset V0 and V1.
- Upgraded brokers support both ProducerRequest V2 and FetchRequest V2 which uses magic byte 1 for MessageAndOffset.
- When broker sees a producer request V1 (MessageAndOffset = V0), it will decompress the message, assign offsets using definitive absolute offsets and NOT re-compress the message.
- When broker sees a producer request V2 (MessageAndOffset = V1), it will decompress the message, assign offsets using definitive absolute offsets and do re-compression. i.e. downconvert the message format to MessageAndOffset V0.
- When broker sees a fetch request V1 (Supporting MessageAndOffset = V0), because the data format on disk is MessageAndOffset V0, it will use the zero-copy transfer to reply with fetch response V1 with MessageAndOffset V0.
- When broker sees a fetch request V2 (Supporting MessageAndOffset = V0, V1), because the data format on disk is MessageAndOffset V0, it will use zero-copy transfer to reply with fetch response V2 with MessageAndOffset V0.
- When broker sees a producer request V1 (MessageAndOffset = V0), it will decompress the message, assign offsets using definitive absolute offsets and NOT re-compress the message.
- Upgrade consumer to send FetchRequest V2.
- Upgrade producer to send ProducerRequest V2.
...
- After most of the consumers are upgraded, Bump up message.format.version=1 and rolling bounce the brokers.
- Upgraded brokers do the followings:
- When broker sees a producer request V1 (MessageAndOffset = V0), it will decompress the message, assign offsets using relative offsets and re-compress the message. i.e. upconvert the message format to MessageAndOffset V1.
- When broker sees a producer request V2 (MessageAndOffset = V1), it will decompress the message, assign offsets using relative offsets and NOT do re-compression.
- When broker sees a fetch request V1 (Supporting MessageAndOffset = V0), because the data format on disk is MessageAndOffset V1, it will NOT use the zero-copy transfer. Instead the broker will read the message from disk, down convert them to V0 and reply with using fetch response V1 with MessageAndOffset V0.
- When broker sees a fetch request V2 (Supporting MessageAndOffset = V0, V1), because the data format on disk is MessageAndOffset V1, it will use zero-copy transfer to reply with fetch response V2 with MessageAndOffset V1.
- When broker sees a producer request V1 (MessageAndOffset = V0), it will decompress the message, assign offsets using relative offsets and re-compress the message. i.e. upconvert the message format to MessageAndOffset V1.
...