Table of Contents |
---|
Status
Current state: Under DiscussionAdopted
Discussion thread: here
JIRAs:
- Apache Kafka 3.7:
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-15874 - Apache Kafka 4.0:
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-14560
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Remove support for the following protocol API versions (the range is inclusive) in Apache Kafka 4.0:
- Produce: V0-V6
- Fetch: V0-V9-V3 due to clients (kafka-python)
- ListOffset: V0 -V1 due to clients (librdkafka, KafkaJS, Sarama, kafka-python) support – V0-V3 per the baseline
- Metadata: V0-V3 due to clients (librdkafka, KafkaJS, Sarama, kafka-python) support – V0-V6 per the baseline
- OffsetCommit: V0-V2 V1 due to clients (KafkaJS, Sarama, kafka-python) support – V0-V5 per the baseline
- OffsetFetch: V0 -V2 due to clients (KafkaJS, kafka-python) support – V0-V4 per the baseline
- FindCoordinator: V0 none due to clients (Sarama, kafka-python) support – V0-V1 per the baseline
- JoinGroup: V0-V1 due to clients (Sarama, kafka-python) support – V0-V2 per the baseline
- Heartbeat: none due to clients (Sarama, kafka-python) support – V0-V1 per the baseline
- LeaveGroup: none due to clients (librdkafka, Sarama, kafka-python) support – V0-V1 per the baseline
- SyncGroup: none due to clients (Sarama) support – V0-V1 per the baseline
- DescribeGroups: none due to clients (librdkafka, Sarama) support – V0-V1 per the baseline
- ListGroups: none due to clients (librdkafka, Sarama) support – V0-V1 per the baseline
- SaslHandshake: none due to clients (kafka-python) - V0 per the baseline
- ApiVersions: none due to clients (kafka-python) - V0-V1 per the baseline
- CreateTopics: V0-V1 due to clients (Sarama) support – V0-V2 per the baseline
- DeleteTopics: V0 due to clients (librdkafka, KafkaJS, Sarama) support – V0-V2 per the baseline
- DeleteRecords: none due to clients (Sarama) support – V0 per the baseline
- InitProducerId: none due to clients (Sarama) support – V0 per the baseline
- OffsetForLeaderEpoch: V0-V1
- librdkafka: none
- KafkaJS: none
- Sarama: none
- kafka-python: none
- AddPartitionsToTxn: none due to clients (librdkafka, Sarama) support – V0 per the baseline
- AddOffsetsToTxn: none due to clients (librdkafka, Sarama) support – V0 per the baseline
- EndTxn: none due to clients (Sarama) support – V0 per the baseline
- WriteTxnMarkers: none due to baseline
- TxnOffsetCommit: none due to clients (KafkaJS, Sarama) support – V0-V1 per the baseline
- DescribeAcls: V0
- CreateAcls: V0
- DeleteAcls: V0
- DescribeConfigs: V0 due to clients (librdkafka) support – V0-V1 per the baseline
- AlterConfigs: none due to clients (librdkafka, Sarama) support – V0 per the baseline
- AlterReplicaLogDirs: V0
- librdkafka: none
- KafkaJS: none
- Sarama: none
- kafka-python: none
- DescribeLogDirs: V0
- librdkafka: none
- KafkaJS: none
- Sarama: V1
- kafka-python: none
- SaslAuthenticate: None: none due to baseline
- CreatePartitions: none due to clients (librdkafka, Sarama) support – V0 per the baseline
- CreateDelegationToken: V0
- librdkafka: none
- KafkaJS: none
- Sarama: none
- kafka-python: none
- RenewDelegationToken: V0
- librdkafka: none
- KafkaJS: none
- Sarama: none
- kafka-python: none
- ExpireDelegationToken: V0
- librdkafka: none
- KafkaJS: none
- Sarama: none
- kafka-python: none
- DescribeDelegationToken: V0
- librdkafka: none
- KafkaJS: none
- Sarama: none
- kafka-python: none
- DeleteGroups: V0
Support for the raw unversioned direct SASL protocol that preceded SaslHandshake and KIP-43 will also be removed in Apache Kafka 4.0.
Additional information regarding the client releases used for the analysis above:
...
The Java client is part of the Apache Kafka repository, it is released alongside the broker and it generally supports the latest protocol versions exposed by the broker. Given that, we know that the Java client has supported the protocol API versions baseline set by Apache Kafka 2.1 as part of that release in November 2018 and no further analysis is required.The inter.broker.protocol.version
would have to be 2.1 or higher.
Finally, a couple of observability changes in Apache Kafka 3.7:
- Introduce metric
kafka.network:type=RequestMetrics,name=DeprecatedRequestsPerSec,request=(api-name),version=(api-version),clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version)
- Add boolean field
requestApiVersionDeprecated
to the request header section of the request log (alongsiderequestApiKey
,requestApiVersion
,requestApiKeyName
, etc.).
Proposed Changes
As stated in the Public Interfaces
sections, a number of protocol API and inter broker protocol versions would no longer be supported. Kafka brokers would return the UNSUPPORTED_VERSION error when they receive a request with any of the removed API versions. Similarly, the Java clients (producer, consumer and admin) would throw UnsupportedVersionException when interacting with brokers that do not support the newly established minimum protocol API versions. A broker configured with an unsupported inter-broker protocol version would fail during start-up with an error.
...
Finally, users should ensure the brokers are running at least Apache Kafka 2.1 and inter.broker.protocol.version
is set to 2.1 or higher before upgrading brokers to Apache Kafka 4.0. The replacement of zookeeper with KRaft in Apache Kafka 4.0 will have a stricter requirement regarding the inter-broker protocol version, so this implication is unimportant.
* This is relevant for clients that do not auto negotiate protocol api versions and default to older versions - Sarama is one such client.
Test Plan
- Client compatibility system tests will updated so that 2.0.x clients fail with an UNSUPPORTED_VERSION error and older versions are removed.
- Client compatibility system tests will updated so that 4.0.x clients fail with an UNSUPPORTED_VERSION error when dealing with Apache Kafka brokers that are too old (2.0 and/or 1.x).
- Broker upgrade tests will be updated so that upgrades from 2.0.x fail with an appropriate error and upgrades from older broker versions are removed.
- Protocol API integration tests will be updated so that the highest unsupported version and lowest supported version are tested and older versions are not.
- Protocol API unit tests will continue to cover all versions, but the expectations will be updated for unsupported versions.
- Test simple flows end to end with the clients mentioned in this proposal against a cluster with this KIP implemented.
...