...

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-584: Versioning scheme for features introduced a feature versioning scheme and a way to store and update various features in the cluster. It did not provide any way to set features other than Metadata Version (MV).

...

Further, we want to make the features interface easier to use with more features so we can introduce a transaction version (TV) that will help with rolling out KIP-890: Transactions Server-Side Defense part 2 and KIP-939: Support Participation in 2PC.
We can also introduce a group coordinator version to help with rolling out KIP-848: The Next Generation of the Consumer Rebalance Protocol. 

Public Interfaces

This KIP proposes to add flags to the format (kafka-storage format) and feature upgrade tool (kafka-feature upgrade) to allow configuring other features.

It also proposes adding Transaction Version (TV) and Group Coordinator Version (GCV) as new features.

Proposed Changes

For the format tool:

• Update the functionality of --release-version flag to set reasonable defaults for the other feature versions known by the tool. Internally this will consist each feature having a mapping from metadata version to that version’s feature version. For example, if we input 3.8-IV2 that could map to TV1, 3.8-IV3 could also be TV1, and 3.8-IV3 could be TV2. There may be some scenarios where a MV doesn’t exist yet. In that case, we may create a MV to be used only by this tool in order to map to the new feature version.

...

Add the new transaction version and group coordinator feature versions. For every metadata version, we will map to the corresponding transaction and group coordinator version. For MVs that existed before these features, we map the new features to version 0 (no feature enabled).

Compatibility, Deprecation, and Migration Plan

Old versions of the tool will continue to behave as they did. Features other than metadata can not be set or upgraded. The new version of upgrade tool can be used on clusters created with the old feature tool to update features. The apis already exist as of KIP-584.

A Note on Feature Interdependence

One aspect of compatibility is how we set features that depend on each other. KIP-584 was built to allow for interdependence. In other words, one feature may depend on another and a cluster may have valid “ranges” of versions for each feature depending on which features are enabled.

...

The tool should ensure that the features set are valid. This is done by marking features as dependent on others internally.

Test Plan

Tests will be added to the StorageToolTest and FeatureCommandTest.

Implicit in this KIP is also the necessity to test various combinations of feature versions. We should test all “release default” combinations and have a test for every version of a given feature with other default features, but do not commit to testing every combination of features.

Rejected Alternatives

Use release version to determine default feature versions

Originally there was concern about using MV to signify the default for all other feature versions. IVs of metadata won’t necessarily match with other features since other features don’t necessarily have the same number of versions per kafka release (3.6, 3.7, 4.0 etc). In addition, some features won’t have a new feature for a given release. However, there is some complexity in having multiple commits leading to different default versions for a given release version. MV handles this by having a single number that maps to the same features every time (once it is production ready).

Additionally, there was some concern with adding complexity or changes to the tool for the basic use case. With the current approach, users won’t see any changes to how they use the tool now. We extend extra functionality to folks who want finer grained control via the --features flag in the storage tool.

Running the storage tool missing a feature should cause an error

This was considered with the concern that folks would miss setting features. However, MV defaults to not needing a version and using the latest production. So the storage tool with some missing features should do that too.

...