...

No argument behavior on the storage and upgrade tools is to set all the features to the latest version. 

For the format tool:

...

both tools:

• Add version-mapping command to to look up the corresponding features for a given metadata version. Using the command with no  --release-version  argument will return the mapping for the latest metadata version.
• Add dependencies command to look up dependencies for a given feature version supplied by --feature flag. If the feature is not known or the version not yet defined, throw an error. 
   

For the format tool:

• Update the functionality of --release-version flag to set reasonable defaults for the other feature versions known by the tool. Internally this will consist each feature having a mapping from metadata version to that version’s feature version. For example, if we input 3.8-IV2 that could map to TV1, 3.8-IV3 could also be TV1, flag to set reasonable defaults for the other feature versions known by the tool. Internally this will consist each feature having a mapping from metadata version to that version’s feature version. For example, if we input 3.8-IV2 that could map to TV1, 3.8-IV3 could also be TV1, and 3.8-IV4 could be TV2. There may be some scenarios where a MV doesn’t exist yet. In that case, we may create a MV to be used only by this tool in order to map to the new feature version.

...

bin/kafka-storage.sh format --cluster-id=123456 // Gives the cluster all the latest feature versionsversion-mapping
bin/kafka-storage.sh formatversion-mapping --release-version 3.6-IVI --cluster-id=123456IV1
bin/kafka-storage.sh version-mapping --release-version 2.9-IV2 // Givesthrows theerror, clusternot alla the versions that map to 3.6-IV1valid version

bin/kafka-features.sh version-mapping
bin/kafka-storagefeatures.sh formatversion-mapping --release-featureversion metadata.version=163.6-IV1

kafka-feature dependencies --feature transaction.protocol.version=2 --feature group.coordinator.version=1 --cluster-id=123456 // Gives the cluster metadata.version 16, transaction protocol version 2, and group coordinator version 1
bin/kafka-storage.sh format
> transaction.protocol.verison=2 requires metadata.version=4 (listing any other version dependencies)

kafka-feature dependencies --feature metadata.version=16 --feature transaction.protocol17
> metadata.version=17 has no dependencies

bin/kafka-storage.sh format2 --release-version 3.3-IV2 --cluster-id=123456 // throws error!Gives the cluster all the latest feature versions
bin/kafka-storage.sh format --featurerelease-version transaction.protocol.version=23.6-IVI --cluster-id=123456 // GivenGives TVthe 2cluster isall notthe knownversions bythat themap storage tool, throws error!to 3.6-IV1
bin/kafka-storage.sh format --feature group.coordinator metadata.version=16 --feature transaction.protocol.version=2 --release-version 3.3-IV2feature group.coordinator.version=1 --cluster-id=123456 // GivenGives GCP2the requirescluster MV 3.8-IV3, throws error!
 
bin/kafka-features.sh upgrade // Sets the cluster to all the latest feature versionsmetadata.version 16, transaction protocol version 2, and group coordinator version 1
bin/kafka-featuresstorage.sh upgrade format --feature metadata.version=16 --feature transaction.protocol.version=2 // Sets the transaction protocol version to 2--release-version 3.3-IV2 --cluster-id=123456 // throws error!
bin/kafka-featuresstorage.sh upgrade --feature metadata.version=16format --feature transaction.protocol.version=2 ---feature group.coordinator.version=1cluster-id=123456 // UpgradesGiven theTV cluster2 tois metadata.version 16, transaction protocol version 2, and group coordinator version 1not known by the storage tool, throws error!
bin/kafka-featuresstorage.sh upgrade format --feature group.coordinator.version=2 --release-version 3.6-IVI3-IV2 --cluster-id=123456 // UpgradesGiven allGCP2 therequires features to versions that map to 3.6-IV1MV 3.8-IV3, throws error!
 
bin/kafka-features.sh downgrade --feature metadata.version=16 upgrade // Sets the cluster to all the latest feature versions
bin/kafka-features.sh upgrade --feature transaction.protocol.version=2 // ThrowsSets errorthe iftransaction metadataprotocol version is < 16, and this would be an upgradeto 2
bin/kafka-features.sh upgrade --feature metadata.version=16 --feature transaction.protocol.version=2  --release-version 3.6-IVIfeature group.coordinator.version=1 // ThrowsUpgrades error!

Add the new transaction protocol version and group coordinator feature versions. For every metadata version, we will map to the corresponding transaction and group coordinator version. For MVs that existed before these features, we map the new features to version 0.

Add a CLI command to both tools to look up the corresponding features for a given metadata version. Using the command with no --release-version  argument will return the mapping for the latest metadata version.

Add a CLI command to look up dependencies for a given feature version. If the feature is not known or the version not yet defined, throw an error. 

bin/kafka-storage.sh version-mapping
bin/kafka-storage.sh version-mapping --release-version 3.6-IV1
bin/kafka-storage.sh version-mapping --release-version 2.9-IV2 // throws error, not a valid version

bin/kafka-features.sh version-mappingthe cluster to metadata.version 16, transaction protocol version 2, and group coordinator version 1
bin/kafka-features.sh upgrade --release-version 3.6-IVI // Upgrades all the features to versions that map to 3.6-IV1
bin/kafka-features.sh downgrade --feature metadata.version=16 --feature transaction.protocol.version=2 // Throws error if metadata version is < 16, and this would be an upgrade
bin/kafka-features.sh upgrade version-mapping--feature transaction.protocol.version=2  --release-version 3.6-IV1

kafka-feature dependencies --feature transaction.protocol.version=2
> transaction.protocol.verison=2 requires metadata.version=4 (listing any other version dependencies)

kafka-feature dependencies --feature metadata.version=17
> metadata.version=17 has no dependencies

The output of the features describe command will include the short version of new features.

Feature: metadata.version             SupportedMinVersion: 3.0-IV1    SupportedMaxVersion: 3.8-IV0    FinalizedVersionLevel: 3.7-IV1       Epoch: 0 
Feature: transaction.protocol.version SupportedMinVersion: 0          SupportedMaxVersion: 2          FinalizedVersionLevel: 1             Epoch: 3 
Feature: group.coordinator.version    SupportedMinVersion: 0          SupportedMaxVersion: 1          FinalizedVersionLevel: 0             Epoch: 2 

Compatibility, Deprecation, and Migration Plan

Old versions of the tool will continue to behave as they did. Features other than metadata can not be set or upgraded. The new version of upgrade tool can be used on clusters created with the old feature tool to update features. The apis already exist as of KIP-584.

A Note on Feature Interdependence

One aspect of compatibility is how we set features that depend on each other. KIP-584 was built to allow for interdependence. In other words, one feature may depend on another and a cluster may have valid “ranges” of versions for each feature depending on which features are enabled.

If feature Y version 12 depends on feature X version 14, we need to handle setting and upgrading these features properly. Given a cluster where feature is X version 13, Y version 12 is not compatible and an error should be thrown if trying to format with X version 13 and Y version 12 OR when trying to upgrade to Y version 12 when X is version 13. The only way to get to Y version 12 is by upgrading to X version 14 first or at the same time as setting Y version 12. Likewise, if we wanted to downgrade feature X to version 12, we would need to do it at the same time as downgrading feature Y or by doing Y first.

The tool should ensure that the features set are valid. This is done by marking features as dependent on others internally.

For this change specifically, all features rely on at least MV 3.3-IV0 (since this enables the writing of features to the metadata topic as per KIP-584). Any command to try to set a feature (either via storage or upgrade tool) when metadata version is below 3.3-IV0 will fail. (If an RPC is sent, it will also fail as the version is not supported)

Transaction version does not rely on any other components, so as long as metadata is 3.3-IV0, there are no failures. 

Group coordinator version will be used to specify which version of the group coordinator is used. For example, 0 is the classic (default), 1 is both classic and consumer groups, 2 could be just consumer groups. Depending on the configuration in the client, they may send different versions of Heartbeat, OffsetFetch, OffsetCommit, ConsumerGroupDescribe and ListGroup requests.

Test Plan

Tests will be added to the StorageToolTest and FeatureCommandTest.

Implicit in this KIP is also the necessity to test various combinations of feature versions. We should test all “release default” combinations and have a test for every version of a given feature with other default features, but do not commit to testing every combination of features.

Rejected Alternatives

Use release version to determine default feature versions

Originally there was concern about using MV to signify the default for all other feature versions. IVs of metadata won’t necessarily match with other features since other features don’t necessarily have the same number of versions per kafka release (3.6, 3.7, 4.0 etc). In addition, some features won’t have a new feature for a given release. However, there is some complexity in having multiple commits leading to different default versions for a given release version. MV handles this by having a single number that maps to the same features every time (once it is production ready).

Additionally, there was some concern with adding complexity or changes to the tool for the basic use case. With the current approach, users won’t see any changes to how they use the tool now. We extend extra functionality to folks who want finer grained control via the --features flag in the storage tool.

Running the storage tool missing a feature should cause an error

This was considered with the concern that folks would miss setting features. However, MV defaults to not needing a version and using the latest production. So the storage tool with some missing features should do that too.

Allow setting feature and release version flags in the upgrade tool

Allow setting both flags so that any features not covered by --feature can be covered by the --release-version flag. In other words, the --feature flag will take precedence over the other flag. The idea is this flag will be used by more advanced users who want finer grained control of specific features.

This was deemed to be too confusing and the usage should be consistent with the storage tool.

Use ApiVersions to query the supported versions on the server for a given MV 

Include an API to request the versions for a given metadata version. A target metadata version can be used to retrieve features supported by that metadata version. The ApiKeys will not change and FinalizedFeatures will be empty. The feature versions for the target metadata version can be found in SupportedFeatures.

{
  "apiKey": 18,
  "type": "request",
  "listeners": ["zkBroker", "broker", "controller"],
  "name": "ApiVersionsRequest",
  // Versions 0 through 2 of ApiVersionsRequest are the same.
  //
  // Version 3 is the first flexible version and adds ClientSoftwareName and ClientSoftwareVersion.
  //
  // Version 4 introduces optional target metadata version field to see the supported versions for a given metadata version
  "validVersions": "0-4",
  "flexibleVersions": "3+",
  "fields": [
    { "name": "ClientSoftwareName", "type": "string", "versions": "3+",
      "ignorable": true, "about": "The name of the client." },
    { "name": "ClientSoftwareVersion", "type": "string", "versions": "3+",
      "ignorable": true, "about": "The version of the client." }
	{ "name": "TargetMetadataVersion", "type": int16, "versions: "4+",
      "ignorable":true, "about": "The target metadata version used to see the corresponding feature versions"
    }
  ]
}

Include dependencies in the describe output

...

IVI // Throws error!

Add the new transaction protocol version and group coordinator feature versions. For every metadata version, we will map to the corresponding transaction and group coordinator version. For MVs that existed before these features, we map the new features to version 0.

Compatibility, Deprecation, and Migration Plan

Old versions of the tool will continue to behave as they did. Features other than metadata can not be set or upgraded. The new version of upgrade tool can be used on clusters created with the old feature tool to update features. The apis already exist as of KIP-584.

A Note on Feature Interdependence

One aspect of compatibility is how we set features that depend on each other. KIP-584 was built to allow for interdependence. In other words, one feature may depend on another and a cluster may have valid “ranges” of versions for each feature depending on which features are enabled.

If feature Y version 12 depends on feature X version 14, we need to handle setting and upgrading these features properly. Given a cluster where feature is X version 13, Y version 12 is not compatible and an error should be thrown if trying to format with X version 13 and Y version 12 OR when trying to upgrade to Y version 12 when X is version 13. The only way to get to Y version 12 is by upgrading to X version 14 first or at the same time as setting Y version 12. Likewise, if we wanted to downgrade feature X to version 12, we would need to do it at the same time as downgrading feature Y or by doing Y first.

The tool should ensure that the features set are valid. This is done by marking features as dependent on others internally.

For this change specifically, all features rely on at least MV 3.3-IV0 (since this enables the writing of features to the metadata topic as per KIP-584). Any command to try to set a feature (either via storage or upgrade tool) when metadata version is below 3.3-IV0 will fail. (If an RPC is sent, it will also fail as the version is not supported)

Transaction version does not rely on any other components, so as long as metadata is 3.3-IV0, there are no failures. 

Group coordinator version will be used to specify which version of the group coordinator is used.

The first use case will be KIP-848. We will use version 2 of the flag to gate all the new records and the new consumer group APIs (Heartbeat, OffsetFetch, OffsetCommit, ConsumerGroupDescribe and ListGroup requests) present in AK 3.8. So version 1 will be the only the old protocol and version 2 will be the currently implemented new protocol. For these versions, there are no dependencies on the metadata version. The changes are orthogonal.

Test Plan

Tests will be added to the StorageToolTest and FeatureCommandTest.

Implicit in this KIP is also the necessity to test various combinations of feature versions. We should test all “release default” combinations and have a test for every version of a given feature with other default features, but do not commit to testing every combination of features.

Rejected Alternatives

Use release version to determine default feature versions

Originally there was concern about using MV to signify the default for all other feature versions. IVs of metadata won’t necessarily match with other features since other features don’t necessarily have the same number of versions per kafka release (3.6, 3.7, 4.0 etc). In addition, some features won’t have a new feature for a given release. However, there is some complexity in having multiple commits leading to different default versions for a given release version. MV handles this by having a single number that maps to the same features every time (once it is production ready).

Additionally, there was some concern with adding complexity or changes to the tool for the basic use case. With the current approach, users won’t see any changes to how they use the tool now. We extend extra functionality to folks who want finer grained control via the --features flag in the storage tool.

Running the storage tool missing a feature should cause an error

This was considered with the concern that folks would miss setting features. However, MV defaults to not needing a version and using the latest production. So the storage tool with some missing features should do that too.

Allow setting feature and release version flags in the upgrade tool

Allow setting both flags so that any features not covered by --feature can be covered by the --release-version flag. In other words, the --feature flag will take precedence over the other flag. The idea is this flag will be used by more advanced users who want finer grained control of specific features.

This was deemed to be too confusing and the usage should be consistent with the storage tool.

Use ApiVersions to query the supported versions on the server for a given MV 

Include an API to request the versions for a given metadata version. A target metadata version can be used to retrieve features supported by that metadata version. The ApiKeys will not change and FinalizedFeatures will be empty. The feature versions for the target metadata version can be found in SupportedFeatures.

{
  "apiKey": 18,
  "type": "request",
  "listeners": ["zkBroker", "broker", "controller"],
  "name": "ApiVersionsRequest",
  // Versions 0 through 2 of ApiVersionsRequest are the same.
  //
  // Version 3 is the first flexible version and adds ClientSoftwareName and ClientSoftwareVersion.
  //
  // Version 4 introduces optional target metadata version field to see the supported versions for a given metadata version
  "validVersions": "0-4",
  "flexibleVersions": "3+",
  "fields": [
    { "name": "ClientSoftwareName", "type": "string", "versions": "3+",
      "ignorable": true, "about": "The name of the client." },
    { "name": "ClientSoftwareVersion", "type": "string", "versions": "3+",
      "ignorable": true, "about": "The version of the client." }
	{ "name": "TargetMetadataVersion", "type": int16, "versions: "4+",
      "ignorable":true, "about": "The target metadata version used to see the corresponding feature versions"
    }
  ]
}

Include dependencies in the describe output

Using the tool to describe dependencies is sufficient. 

Feature: metadata.version             SupportedMinVersion: 3.0-IV1    SupportedMaxVersion: 3.8-IV0    FinalizedVersionLevel: 3.7-IV1       Epoch: 0 
Feature: transaction.protocol.version SupportedMinVersion: 0          SupportedMaxVersion: 2          FinalizedVersionLevel: 1             Epoch: 3 
Feature: group.coordinator.version    SupportedMinVersion: 0          SupportedMaxVersion: 1          FinalizedVersionLevel: 0             Epoch: 2