Table of Contents |
---|
Status
Current state: "Under Discussion"Adopted
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: link
Vote thread: link
JIRA:
(3.0)Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-12944
(3.1)Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-13093
(4.0)Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-12872
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Four years later, it's time to sunset message formats v0 and v1 to establish a new baseline in terms of supported client/broker behavior and to improve maintainability & supportability of Kafka. This also aligns with KIP-679, which will enable the idempotent producer by default in Apache Kafka 3.0 (and requires message format v2). We propose the deprecation of message formats v0 and v1 in Apache Kafka 3.0 and to disallow writes with v0 or v1 in Apache Kafka 4.0.
...
Note that followers will continue to write the records they receive from leaders without conversion. Since leaders will use v2
for new requests, replicating old data is the only case where v0 or v1 records may be written to disk after the upgrade to Apache Kafka 3.0 and the inter.broker.protocol.version is 3.0.
Produce and fetch requests with v0 and v1 message formats would will be supported via up-conversion and down-conversion. Up-conversion and (especially) down-conversion have measurable performance impact due to increased CPU and memory usage, but the vast majority of Kafka clients have supported v2 for some time (even Spark, a notable late adopter, has supported v2 since Spark 2.4, which was released in October 2018).
...
Apache Kafka 4.0
We will remove all write support for message formats v0 and v1 in Apache Kafka 4.0. Consumers will continue to support message formats v0 and v1. Similarly, brokers will still return message formats v0 and v1 with fetch v4 or higher.
Compatibility, Deprecation, and Migration Plan
...
On the broker, we can achieve this by providing a mechanism to to force up-convert all the record batches on disk. For example, we introduce a new config log.record.version.force.upgrade
with two possible values: null
(default) and 2. If the value is set to 2
, the broker ensures all segments have records with format v2
after log recovery has completed during start-up. This can be extended to support newer message format versions if and when they are introduced. This config is only allowed to be non-null if the if the inter.broker.protocol.version
is 3.0 or higher. In a subsequent major release, we would drop support for reading v0 and v1 records from disk.
On the consumer, it's more complicated. Brokers assume that consumers can read all message format versions and brokers never up-convert records when handling fetch responses. Furthermore, we cannot change the behavior of older broker versions. The options here are a bit complex and require more thought.