...

The Apache Kafka project has retained support for all protocol API versions since Apache Kafka 0.8.0 (released 9+ years ago on December 2013). As time goes on, the cost of maintaining support for all these versions goes up (both in code complexity and the testing matrix) and the value goes down (as users switch to clients that rely on newer protocol API versions). Apache Kafka 4.0 presents a good opportunity to re-evaluate this policy since it includes other modernization initiatives like dropping support for zk mode and Drop dropping support for message formats v0 and v1 (KIP-724). No date has been set for Apache Kafka 4.0, but it's safe to say it won't happen before the end of 2023.

...

• Produce: V0-V6
  • librdkafka: V7
  • KafkaJS: V7
  • Sarama: V7
  • kafka-python: V8
• Fetch: V0-V9
  • librdkafka: V11
  • KafkaJS: V11
  • Sarama: V11
  • kafka-python: V11
• ListOffset: V0-V1 due to clients support (V0-V3 per the baseline)
  • librdkafka: V2
  • KafkaJS: V3
  • Sarama: V2
  • kafka-python: V5
• Metadata: V0-V3 due to clients support (V0-V6 per the baseline)
  • librdkafka: V4
  • KafkaJS: V6
  • Sarama: V5
  • kafka-python: V5
• OffsetCommit: V0-V2 due to clients support (V0-V5 per the baseline)
  • librdkafka: V7
  • KafkaJS: V5
  • Sarama: V4
  • kafka-python: V3
• OffsetFetch: V0-V2 due to clients support (V0-V4 per the baseline)
  • librdkafka: V7
  • KafkaJS: V4
  • Sarama: V7
  • kafka-python: V3
• FindCoordinator: V0 due to clients support (V0-V1 per the baseline)
  • librdkafka: V2
  • KafkaJS: V2
  • Sarama:
  • kafka-python: V1
• JoinGroup: V0-V2
  • librdkafka: V5
  • KafkaJS: V5
  • Sarama:
  • kafka-python:
• Heartbeat: V0-V1
  • librdkafka: V3
  • KafkaJS: V3
  • Sarama:
  • kafka-python:
• LeaveGroup: V0 due to clients support (V0-V1 per the baseline)
  • librdkafka: V1
  • KafkaJS: V3
  • Sarama:
  • kafka-python:
• SyncGroup: V0-V1
  • librdkafka: V3
  • KafkaJS: V3
  • Sarama:
  • kafka-python:
• DescribeGroups: none due to clients support (V0-V1 per the baseline)
  • librdkafka: V0?
  • KafkaJS: V2
  • Sarama:
  • kafka-python:
• ListGroups: V0-V1
  • librdkafka: V0
  • KafkaJS: V2
  • Sarama:
  • kafka-python:
• SaslHandshake: V0
  • librdkafka: V1
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• ApiVersions: V0-V1
  • librdkafka: V3
  • KafkaJS: V2
  • Sarama:
  • kafka-python:
• CreateTopics: V0-V2
  • librdkafka: V4
  • KafkaJS: V3
  • Sarama:
  • kafka-python:
• DeleteTopics: V0-V2
  • librdkafka: V1
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• DeleteRecords: V0
  • librdkafka: V1
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• InitProducerId: V0
  • librdkafka: V4
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• OffsetForLeaderEpoch: V0-V1
  • librdkafka: none
  • KafkaJS: none
  • Sarama:
  • kafka-python:
• AddPartitionsToTxn: V0
  • librdkafka: V0
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• AddOffsetsToTxn: V0
  • librdkafka: V0
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• EndTxn: V0
  • librdkafka: V1
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• WriteTxnMarkers: Nonenone
• TxnOffsetCommit: V0-V1
  • librdkafka: V3
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• DescribeAcls: V0
  • librdkafka: none
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• CreateAcls: V0
  • librdkafka: none
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• DeleteAcls: V0
  • librdkafka: none
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• DescribeConfigs: V0-V1
  • librdkafka: V1
  • KafkaJS: V2
  • Sarama:
  • kafka-python:
• AlterConfigs: V0
  • librdkafka: V0
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• AlterReplicaLogDirs: V0
  • librdkafka: none
  • KafkaJS: none
  • Sarama:
  • kafka-python:
• DescribeLogDirs: V0
  • librdkafka: none
  • KafkaJS: none
  • Sarama:
  • kafka-python:
• SaslAuthenticate: None
• CreatePartitions: V0
  • librdkafka: V0
  • KafkaJS: V1
  • Sarama:
  • kafka-python:
• CreateDelegationToken: V0
  • librdkafka: none
  • KafkaJS: none
  • Sarama:
  • kafka-python:
• RenewDelegationToken: V0
  • librdkafka: none
  • KafkaJS: none
  • Sarama:
  • kafka-python:
• ExpireDelegationToken: V0
  • librdkafka: none
  • KafkaJS: none
  • Sarama:
  • kafka-python:
• DescribeDelegationToken: V0
  • librdkafka: none
  • KafkaJS: none
  • Sarama:
  • kafka-python:
• DeleteGroups: V0
  • librdkafka: V1
  • KafkaJS: V1
  • Sarama:
  • kafka-python:

Additional information regarding the client releases used for the analysis above:

...

TBD (talk about broker metrics users can rely on to know if they're impacted and what happens if an unsupported protocol version is used)

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

...

• Client compatibility system tests will updated so that 2.0.x clients fail with an UNSUPPORTED_VERSION error and older versions are removed.
• Protocol API integration tests will be updated so that the highest unsupported version and lowest supported version are tested and older versions are not.
• Protocol API unit tests will continue to cover all versions, but the expectations will be updated for unsupported versions.

Rejected Alternatives

1. Continue with the "support forever" compatibility policy: the benefit cost ratio is low given low usage of the older protocol API versions.