Page History

...

1. Set a reasonable default for all the features when formatting the cluster (most users)

2. Allow for flexibility in setting and upgrading any feature – we may want to set transaction protocol version separately from metadata version or kraft version (advanced users)

...

Further, we want to make the features interface easier to use with more features so we can introduce a transaction protocol version (TV) that will help with rolling out KIP-890: Transactions Server-Side Defense part 2 and KIP-939: Support Participation in 2PC.
We can also introduce a group coordinator version to help with rolling out KIP-848: The Next Generation of the Consumer Rebalance Protocol. 

...

This KIP proposes to add flags to the format (kafka-storage format) and feature upgrade tool (kafka-feature upgrade) to allow configuring other features.

It also proposes adding Transaction Protocol Version transaction version (TV) and Group Coordinator Version (GCV) as new features.

...

bin/kafka-storage.sh version-mapping // Returns latest stable version mapping, in this example, say 3.8-IV0 is latest stable
	metadata.version=19 (3.8-IV0)
	transaction.protocol  transaction.version=2
	group.coordinator.version=1
	kraft.version=0

bin/kafka-storage.sh version-mapping --release-version 3.6-IV1
	metadata.version=13 (3.6-IV1)
	transaction.protocol  transaction.version=0
	group.coordinator.version=0
	kraft.version=0
 
bin/kafka-storage.sh version-mapping --release-version 2.9-IV2 // throws error, not a valid version

bin/kafka-feature feature-dependencies --feature transaction.protocol.version=2
	transaction.protocol.verison=2 requires:
	metadata.version=4 (3.3-IV0) (listing any other version dependencies)

bin/kafka-feature feature-dependencies --feature metadata.version=17
	metadata.version=17 (3.7-IV2) has no dependencies

...

bin/kafka-storage.sh format --cluster-id=123456 // Gives the cluster all the latest stable feature versions
bin/kafka-storage.sh format --release-version 3.6-IVI --cluster-id=123456 // Gives the cluster all the versions that map to 3.6-IV1
bin/kafka-storage.sh format --feature metadata.version=16 --feature transaction.protocol.version=2 --feature group.coordinator.version=1 --cluster-id=123456 // Gives the cluster metadata.version 16, transaction protocol version 2, and group coordinator version 1
bin/kafka-storage.sh format --feature metadata.version=16 --feature transaction.protocol.version=2 --release-version 3.3-IV2 --cluster-id=123456 // throws error!
bin/kafka-storage.sh format --feature transaction.protocol.version=2 --cluster-id=123456 // Given TV 2 is not known by the storage tool, throws error!
bin/kafka-storage.sh format --feature group.coordinator.version=2 --release-version 3.3-IV2 --cluster-id=123456 // Given GCV2 requires MV 3.8-IV3, throws error!
 
bin/kafka-features.sh upgrade // Sets the cluster to all the latest stable feature versions
bin/kafka-features.sh upgrade --feature transaction.protocol.version=2 // Sets the transaction protocol version to 2
bin/kafka-features.sh upgrade --feature metadata.version=16 --feature transaction.protocol.version=2 --feature group.coordinator.version=1 // Upgrades the cluster to metadata.version 16, transaction protocol version 2, and group coordinator version 1
bin/kafka-features.sh upgrade --release-version 3.6-IVI // Upgrades all the features to versions that map to 3.6-IV1
bin/kafka-features.sh downgrade --feature metadata.version=16 --feature transaction.protocol.version=2 // Throws error if metadata version is < 16, and this would be an upgrade
bin/kafka-features.sh upgrade --feature transaction.protocol.version=2  --release-version 3.6-IVI // Throws error!

Add the new transaction protocol version and group coordinator feature versions. For every metadata version, we will map to the corresponding transaction and group coordinator version. For MVs that existed before these features, we map the new features to version 0.

...

Group coordinator version will be used to specify which version of the group coordinator is used. The first use case will be KIP-848. We will use version 1 of the flag to gate all the new records and the new consumer group APIs (Heartbeat, OffsetFetch, OffsetCommit, ConsumerGroupDescribe and ListGroup requests) present in AK 3.8. So version 0 will be the only the old protocol and version 1 will be the currently implemented new protocol. For these versions, there are no dependencies on the metadata version other than the need for the feature records like transaction protocol version. 

For both features, we will be finally using the flexible fields in KIP-915. This means that the records for the state topics of the transaction coordinator and the group coordinator will be determined by their respective feature versions and not metadata version. 

...

Feature: metadata.version             SupportedMinVersion: 3.0-IV1    SupportedMaxVersion: 3.8-IV0    FinalizedVersionLevel: 3.7-IV1       Epoch: 0 
Feature: transaction.protocol.version SupportedMinVersion: 0          SupportedMaxVersion: 2          FinalizedVersionLevel: 1             Epoch: 3 
Feature: group.coordinator.version    SupportedMinVersion: 0          SupportedMaxVersion: 1          FinalizedVersionLevel: 0             Epoch: 2 

...