Versions Compared

Old Version 27

changes.mady.by.user Unknown User (jolshan)

Saved on

compared with

New Version 28

changes.mady.by.user Unknown User (jolshan)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Further, we want to make the features interface easier to use with more features so we can introduce a transaction version (TV) that will help with rolling out KIP-890: Transactions Server-Side Defense part 2 and KIP-939: Support Participation in 2PC.
We can also introduce a group coordinator version to help with rolling out KIP-848: The Next Generation of the Consumer Rebalance Protocol. 

...

It also proposes adding Transaction Version (TV) and Group Coordinator Version (GCVGV) as new features.

Proposed Changes

...

Code Block
bin/kafka-storage.sh version-mapping // Returns latest stable version mapping, in this example, say 3.8-IV0 is latest stable
	metadata.version=19 (3.8-IV0)  transaction.version=2
	group.coordinator  group.version=1
	kraft.version=0

bin/kafka-storage.sh version-mapping --release-version 3.6-IV1
	metadata.version=13 (3.6-IV1)  transaction.version=0
	group.coordinator.  group.version=0
	kraft.version=0
 
bin/kafka-storage.sh version-mapping --release-version 2.9-IV2 // throws error, not a valid version

bin/kafka-feature feature-dependencies --feature transaction.version=2
	transaction.protocol.verisonversion=2 requires:
	metadata.version=4 (3.3-IV0) (listing any other version dependencies)

bin/kafka-feature feature-dependencies --feature metadata.version=17
	metadata.version=17 (3.7-IV2) has no dependencies

...

Code Block
bin/kafka-storage.sh format --cluster-id=123456 // Gives the cluster all the latest stable feature versions
bin/kafka-storage.sh format --release-version 3.6-IVI --cluster-id=123456 // Gives the cluster all the versions that map to 3.6-IV1
bin/kafka-storage.sh format --feature metadata.version=16 --feature transaction.version=2 --feature group.coordinator.version=1 --cluster-id=123456 // Gives the cluster metadata.version 16, transaction version 2, and group coordinator version 1
bin/kafka-storage.sh format --feature metadata.version=16 --feature transaction.version=2 --release-version 3.3-IV2 --cluster-id=123456 // throws error!
bin/kafka-storage.sh format --feature transaction.version=2 --cluster-id=123456 // Given TV 2 is not known by the storage tool, throws error!
bin/kafka-storage.sh format --feature group.coordinator.version=2 --release-version 3.3-IV2 --cluster-id=123456 // Given GCV2 requires MV 3.8-IV3, throws error!
 
bin/kafka-features.sh upgrade // Sets the cluster to all the latest stable feature versions
bin/kafka-features.sh upgrade --feature transaction.version=2 // Sets the transaction version to 2
bin/kafka-features.sh upgrade --feature metadata.version=16 --feature transaction.version=2 --feature group.coordinator.version=1 // Upgrades the cluster to metadata.version 16, transaction version 2, and group coordinator version 1
bin/kafka-features.sh upgrade --release-version 3.6-IVI // Upgrades all the features to versions that map to 3.6-IV1
bin/kafka-features.sh downgrade --feature metadata.version=16 --feature transaction.version=2 // Throws error if metadata version is < 16, and this would be an upgrade
bin/kafka-features.sh upgrade --feature transaction.version=2  --release-version 3.6-IVI // Throws error!

Add the new transaction version and group coordinator feature versionsversion features. For every metadata version, we will map to the corresponding transaction and group coordinator version. For MVs that existed before these features, we map the new features to version 0.

...

Transaction version 1 will include the flexible fields in the transaction state log as well as persisting the prevProducerId field from KIP-360, and transaction version 2 will include the changes to the transactional protocol as described by KIP-890 (epoch bumps and implicit add partitions.) Transaction version does not rely on any other components, so as long as metadata is 3.3-IV0, there are no failures. 

Group coordinator version will be used to specify which version of the group coordinator is used. The first use case will be KIP-848. We will use version 1 of the flag to gate all the new records and the new consumer group APIs (Heartbeat, OffsetFetch, OffsetCommit, ConsumerGroupDescribe and ListGroup requests) present in AK 3.8. So version 0 will be the only the old protocol and version 1 will be the currently implemented new protocol. For these versions, there are no dependencies on the metadata version other than the need for the feature records like transaction version. 

...

Code Block
Feature: metadata.version             SupportedMinVersion: 3.0-IV1    SupportedMaxVersion: 3.8-IV0    FinalizedVersionLevel: 3.7-IV1       Epoch: 0 
Feature: transaction.version SupportedMinVersion: 0          SupportedMaxVersion: 2          FinalizedVersionLevel: 1             Epoch: 3 
Feature: group.coordinator.version     SupportedMinVersion SupportedMinVersion: 0          SupportedMaxVersion: 1          FinalizedVersionLevel: 0             Epoch: 2

...