This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

...

1. Set a reasonable default for all the features when formatting the cluster (most users)

2. Allow for flexibility in setting and upgrading any feature – we may want to set transaction protocol version separately from metadata version or kraft version (advanced users)

...

Further, we want to make the features interface easier to use with more features so we can introduce a transaction protocol version (TV) that will help with rolling out KIP-890: Transactions Server-Side Defense part 2 and KIP-939: Support Participation in 2PC.
We can also introduce a group coordinator version to help with rolling out KIP-848: The Next Generation of the Consumer Rebalance Protocol. 

...

It also proposes adding Transaction Protocol Version (TV) and Group Coordinator Version (GCV) as new features.

Proposed Changes

No argument behavior on the storage and upgrade tools is to set all the features to the latest version. 

For the format tool:

• Update the functionality of --release-version flag to set reasonable defaults for the other feature versions known by the tool. Internally this will consist each feature having a mapping from metadata version to that version’s feature version. For example, if we input 3.8-IV2 that could map to TV1, 3.8-IV3 could also be TV1, and 3.8-IV4 could be TV2. There may be some scenarios where a MV doesn’t exist yet. In that case, we may create a MV to be used only by this tool in order to map to the new feature version.

• Introduce --features flag feature flag where each feature can be configured by specifying the string name and short version of the feature. The command should fail for features (and versions) the tool is not familiar with. If not all features are covered with this flag, the command will just use the latest production version of the feature like it does for metadata version. We will will not allow setting both flags and any features not covered by --features can be covered by the --release-version flag. In other words, the --features flag will take precedence over the other flag. The idea is this flag will be used by more advanced users who want finer grained control of specific features.. 

For the features tool:

• Introduce --release-version. It will work just like the storage tool and upgrade all the features to a version determined by the metadata → feature version mapping from the storage tool. This is ideal for the average user who wants the latest stable features for a given version. Only allow one flag to be set here. Either Set a single features feature via --feature (or multiple usages of the flag for multiple features) or the --release-version flag
  
  

Examples:

bin/kafka-storage.sh format --cluster-id=123456 // Gives the cluster all the latest feature versions
bin/kafka-storage.sh format --release-version 3.6-IVI --cluster-id=123456 // Gives the cluster all the versions that map to 3.6-IV1
bin/kafka-storage.sh format --featuresfeature metadata.version=16,transaction --feature transaction.protocol.version=2, --feature group.coordinator.version=1 --cluster-id=123456 // Gives the cluster metadata.version 16, transaction protocol version 2, and group coordinator version 1
bin/kafka-storage.sh format --featuresfeature metadata.version=16,transaction --feature transaction.protocol.version=2 --release-version 3.3-IV2 --cluster-id=123456 // Gives the cluster metadata.version 16, transaction version 2, and any other features map to 3.3-IV2throws error!
bin/kafka-storage.sh format --featuresfeature transaction.version=2 --cluster-id=123456 // Given TV 2 is not known by the storage tool, throws error!
bin/kafka-storage.sh format --featuresfeature group.coordinator.version=2 --release-version 3.3-IV2 --cluster-id=123456 // Given GCP2 requires MV 3.8-IV3, throws error!
 
bin/kafka-features.sh upgrade // Sets the cluster to all the latest feature versions
bin/kafka-features.sh upgrade --feature transaction.version=2 // Sets the transaction protocol version to 2
bin/kafka-features.sh upgrade --feature metadata.version=16 --feature transaction.version=2 --feature group.coordinator.version=1 // Upgrades the cluster to metadata.version 16, transaction protocol version 2, and group coordinator version 1
bin/kafka-features.sh upgrade --release-version 3.6-IVI // Upgrades all the features to versions that map to 3.6-IV1
bin/kafka-features.sh downgrade --feature metadata.version=16 --transaction.version=2 // Throws error if metadata version is < 16, and this would be an upgrade
bin/kafka-features.sh upgrade --feature transaction.version=2  --release-version 3.6-IVI // Throws error!

Add the new transaction protocol version and group coordinator feature versions. For every metadata version, we will map to the corresponding transaction and group coordinator version. For MVs that existed before these features, we map the new features to version 0 (no feature enabled).

Include an API to request the versions for a given metadata version. A target metadata version can be used to retrieve features supported by that metadata version. The ApiKeys will not change and FinalizedFeatures will be empty. The feature versions for the target metadata version can be found in SupportedFeatures.

{
  "apiKey": 18,
  "type": "request",
  "listeners": ["zkBroker", "broker", "controller"],
  "name": "ApiVersionsRequest",
  // Versions 0 through 2 of ApiVersionsRequest are the same.
  //
  // Version 3 is the first flexible version and adds ClientSoftwareName and ClientSoftwareVersion.
  //
  // Version 4 introduces optional target metadata version field to see the supported versions for a given metadata version
  "validVersions": "0-4",
  "flexibleVersions": "3+",
  "fields": [
    { "name": "ClientSoftwareName", "type": "string", "versions": "3+",
      "ignorable": true, "about": "The name of the client." },
    { "name": "ClientSoftwareVersion", "type": "string", "versions": "3+",
      "ignorable": true, "about": "The version of the client." }
	{ "name": "TargetMetadataVersion", "type": , "versions: "4+",
      "ignorable":true, "about": "The target metadata version used to see the corresponding feature versions"
    }
  ]
}

Compatibility, Deprecation, and Migration Plan

...

This was considered with the concern that folks would miss setting features. However, MV defaults to not needing a version and using the latest production. So the storage tool with some missing features should do that too.

Allow setting feature and release version flags in the upgrade tool

Allow setting both flags so that any features not covered by --feature can be covered by the --release-version flag. In other words, the --feature flag will take precedence over the other flag. The idea is this flag will be used by more advanced users who want finer grained control of specific features.

This was deemed to be too confusing and the usage should be consistent with the storage tool.