Page History

...

Sometimes there can be a need to deprecate a specific feature version (see Non-goals section). This requirement translates to increasing the cluster-wide finalized minimum version level of one or more features in the ZK '/features' node. It is important to note that we only allow the finalized feature maximum version level to be increased/decreased dynamically via the new controller API. Contrastingly, the minimum version level can not be
mutated via the Controller API. This is because, the minimum version level is usually increased only to indicate the intent to stop support for a certain feature version. We would usually deprecate features during broker releases,
after prior announcements. Therefore, this is not a dynamic operation, and such a mutation is not supported through the ApiKeys.UPDATE_FEATURES controller API.

Instead it is sufficient if such changes are done directly by the controller i.e. during a certain Kafka release we would change the controller code to mutate the '/features' ZK node increasing the minimum version level
of one or more finalized features (this will be a planned change, as determined by Kafka developers). Then, as this Broker release gets rolled out to a cluster, the feature versions will become permanently deprecated.

Client discovery of finalized feature versions

...

We shall introduce a FeatureCommand CLI tool backed by a new admin command library. This CLI tool will be maintained in the Kafka repository, alongside the code for the Kafka Broker.
The section on Guidelines on feature versions and workflows, is a recommended read ahead of using the CLI tool.

Note the following: The CLI tool internally has knowledge about a map of features to their respective max versions supported by the Broker. The tool's knowledge of features and their version values, is limited to the version of the CLI tool itself i.e. the information is packaged into the CLI tool when it is released. Whenever a Kafka release introduces a new feature version, or modifies an existing feature version, the CLI tool shall also be updated with this information. Newer versions of the CLI tool will be released as part of the Kafka releases.

...

1. Read cluster-wide finalized feature versions from a broker or a controller via it’s ApiKeys.API_VERSIONS API.

2. Upgrade the max version levels of all features, to their latest values, as known to the CLI tool internally. This becomes useful after completing the deployment of a new Kafka Broker release onto an existing cluster. This removes the burden to individually finalize feature upgrades. 
3. Downgrade the max version levels of all features, to the values known to the CLI tool internally. This becomes useful during a an emergency cluster downgrade, after finalizing feature levels from a previous upgrade.
   
   

=== DESCRIBE FEATURES ===

# Get cluster-wide finalized features, and features supported by a specific broker.
#  - Use `--bootstrap-server` to provide a broker host:port to which queries should be issued.
#  - Optionally, provide `--controller` flag directing the tool to issue the query to the
#    controller (while discovering the controller via the bootstrap server).
#    This can be useful for debugging purposes.

$> kafka-features.sh describe \
     --bootstrap-server kafka-broker0.prn1:9071 \
     [--controller]

{
	"status": "OK",
	"supported_features": {
		"group_coordinator": {
            "min_version": 1,
            "max_version": 2
        },
        "transaction_coordinator": {
        	"min_version": 1,
        	"max_version": 5
        },
        "consumer_offsets_topic_schema": { 
            "min_version": 1,
        	"max_version": 1
        }
	},
	"finalized_features": {
        "epoch": 0,
        "group_coordinator": {
			"min_version_level": 1,
            "max_version_level": 1
        },
        "transaction_coordinator": {
			"min_version_level": 1,
        	"max_version_level": 4
        }
   },
   "host": "kafka-broker0.prn1",
   "port": 9071
}

=== UPGRADE TO ALL LATEST FEATURES ===

# Upgrade to the max version levels of all features, as internally known to the CLI tool.
#
# This command removes the burden to individually finalize feature upgrades.
# This becomes handy to a cluster operator intending to finalize a cluster with all the latest
# available feature version levels. This usually happens after completing the deployment
# of a newer Kafka Broker release onto an existing cluster.

$> kafka-features.sh finalize-all-latest \
     --bootstrap-server kafka-broker0.prn1:9071

{
	"status": "OK",
	"supported_features": {
		"group_coordinator": {
            "min_version": 1,
            "max_version": 3
        },
        "transaction_coordinator": {
        	"min_version": 1,
        	"max_version": 6
        },
        "consumer_offsets_topic_schema": { 
            "min_version": 1,
        	"max_version": 3
        }
	},
	"finalized_features": {
		"epoch": 3,
		"group_coordinator": {
			"min_version_level": 1,
            "max_version_level": 3
        },
        "transaction_coordinator": {
			"min_version_level": 1,
            "max_version_level": 6
        },
        "consumer_offsets_topic_schema": { 
			"min_version_level": 1,
            "max_version_level": 3
        }
   },
   "host": "kafka-broker0.prn1",
   "port": 9071
}

=== EMERGENCY DOWNGRADE ALL FEATURES ===

# Downgrade to the max version levels of all features known to the CLI tool.
#
# This command removes the burden to individually finalize feature version
# downgrades. This becomes handy to a cluster operator intending to downgrade all
# feature version levels, just prior to rolling back a Kafka Broker deployment
# on a cluster, to a previous Broker release.

$> kafka-features.sh downgrade-all \
     --bootstrap-server kafka-broker0.prn1:9071

{
	"status": "OK",
	"supported_features": {
		"group_coordinator": {
            "min_version": 1,
            "max_version": 3
        },
        "transaction_coordinator": {
        	"min_version": 1,
        	"max_version": 6
        },
        "consumer_offsets_topic_schema": { 
            "min_version": 1,
        	"max_version": 3
        }
	},
	"finalized_features": {
		"epoch": 3,
		"group_coordinator": {
			"min_version_level": 1,
            "max_version_level": 3
        },
        "transaction_coordinator": {
			"min_version_level": 1,
            "max_version_level": 6
        },
        "consumer_offsets_topic_schema": { 
			"min_version_level": 1,
            "max_version_level": 3
        }
   },
   "host": "kafka-broker0.prn1",
   "port": 9071
}

...

When to use feature versions?

• As part of Kafka operations, there are occasions when it becomes paramount to safely activate certain breaking changes introduced by a newer Broker version. Such changes become good candidates to be released behind a feature flag . The max feature version level can be safely finalized (using the provided A "breaking change" typically happens whenever the underlying Broker logic has introduced a different concept (such as a message type, or a new field in a protocol) which only certain newer versions of the Broker code can understand/process. The older Broker versions could consider the change to be alien i.e. the older Broker versions would exhibit undefined or terminal behavior if they notice effects of the breaking change. Similarly, Kafka clients could exhibit undefined behavior if they start usage of a feature version on the cluster, assuming all Brokers support it, while some don't.
  As part of Kafka operations, there are occasions when it becomes paramount to safely activate certain breaking changes introduced by a newer Broker version. Such changes become good candidates to be released behind a feature flag . The cluster-wide max feature version level can be safely finalized (using the provided mechanism in this KIP), after the cluster-wide deployment of the required Broker version . Whenever is over. With this procedure, whenever the effects of the underlying breaking change is activated by the cluster operator, the versioning system provides a guarantee that it will not bring ill effects to the Broker cluster or the clients.
  Note that a "breaking change" typically happens whenever the underlying logic has introduced a different concept (such as a message type, or a new field in a protocol) which only certain newer versions of the Broker code can understand/process. The older Broker versions could consider the change to be alien i.e. the older Broker versions would exhibit undefined or terminal behavior if they notice effects of the breaking change. Similarly, Kafka clients could exhibit undefined behavior if they activate a feature version assuming all Brokers support it, while some don't.
• As a guideline, min/max supported feature version values should be modified if and only if the change is a breaking change. Non-breaking changes could still continue to be made to the Broker code without modifying the feature version values.

...

• Most common workflow for the feature versioning system would involve finalizing upgrades a cluster upgrade to max feature version levels for all features, after completing the release of a new Kafka Broker binary on a cluster. This is explained  under Regular CLI tool usage section.
• Downgrades to max Emergency downgrades of max version levels of features, is rare especially after they are finalized is rare  (because these are breaking changes, and finalizing these changes can have consequences). For such rare situations, the user could still downgrade specific features using the CLI tool. Also, all features can be downgraded using the CLI tool belonging to a specific Broker release. For examples of these, please refer to the Tooling support section.

...