Page History

...

1. Downgrade of feature version level:
   A feature "downgrade" refers to dropping support across the entire cluster for a feature version level. This means reducing the finalized maximum feature version level X to a version level Y, where Y < X. In other words, dropping cluster-wide support for an existing feature that was already finalized at a newer version level. Firstly, we leave it to the cluster operator (i.e. human) to decide whether the above actions are backwards compatible. It is not within the scope of this KIP to provide help to the cluster operator to achieve this step. After the cluster operator is past this step, we do provide the following support:

   1. Just like with upgrades, a downgrade request to reduce feature version level is rejected by the system, unless, all brokers support the downgraded versions of the feature. In the example above, the system expects all brokers to support the downgraded feature version Y.

   2. We assume that, downgrades of finalized max feature version levels, are rare. For safety reasons, to restrict user intent, we request for the human user to specify an explicit "allow downgrade" flag (in the API request) to safeguard against easy accidental downgrades attempts to downgrade version levels. Note that despite the override setting this flag, certain downgrades may be rejected by the system if it is impossible.

2. Deprecation of feature version level:

   1. A need can arise to deprecate the usage of a certain version of one or more broker feature. A feature "deprecation" refers to increasing the finalized minimum feature version level X to a version level Y, where Y > X. We note that feature versions are typically deprecated during Kafka Broker releases. This is very unlike max feature version level upgrades, which can happen dynamically, after broker bits are deployed to a cluster.

   2. Firstly, the cluster operator (i.e. human) should use external means to establish that it is safe to stop supporting a particular version of broker feature. For example, verify (if needed) that no clients are actively using the version, before deciding to stop supporting it. It is not within the scope of this KIP to provide help to the cluster operator to achieve this step. After the cluster operator is past this step, we do provide the support that during a specific release of Kafka the system would mutate the persisted cluster-wide finalized feature versions to the desired value signaling feature deprecation.

...

• The API requires AclOperation.ALTER on ResourceType.CLUSTER.

• The API request contains a list of FeatureUpdate that need to be applied, as explained below:

  • Each item specifies the finalized feature to be added or updated or deleted, along with the new max feature version level value..

  • To add a new finalized feature version level, or update an existing one, the user must specify the version level starting from 1 (and increasing).
  • Downgrade or deletion of Max feature version level downgrades are , is not a regular operation. Each /intent. As an override, each item optionally can specify an allowDowngrade flag, which can be used to allow version level downgrades (or deletions)to convey user intent to attempt max version level downgrade/deletion. Note that despite this allowDowngrade flag being set, certain downgrades may be rejected by the system if it is impossible.
  • To add a new finalized feature version level, or update an existing one, the user must specify the version level starting from 1 (and increasing).

  • If a finalized feature needs to be permanently deleted, the user must specify a max version level value < 1, and should also set the allowDowngrade flag.

• The API response contains an error code and an error message.
• Changes to cluster-wide finalized minimum feature version level, can not be carried out using this API. This can be only done as explained later under Feature version deprecation section.

...

• A "breaking change" typically happens whenever the underlying Broker logic has introduced a different concept (such as a message type, or a new field in a protocol) which only certain newer versions of the Broker code can understand/process. The older Broker versions could consider such a change to be alien i.e. the older Broker versions could exhibit undefined or terminal behavior if they notice effects of the breaking change. Similarly, Kafka clients could exhibit undesirable behavior if they commence usage of a feature version on the cluster, assuming all Brokers support it, while in reality some Brokers don't.
  As part of Kafka operations, there are occasions when it becomes necessary to safely activate certain breaking changes introduced by a newer Broker version. Such changes become good candidates to be released behind a feature flag . The cluster-wide max feature version level can be safely increased/finalized (using the provided mechanism in this KIP), after the cluster-wide deployment of the required Broker version is over. With the versioning system in place, whenever the effects of the underlying breaking change is activated by the cluster operator, the system protects the cluster and Kafka clients from bad effects due to breaking changes.
• As a guideline, max supported feature version values should be increased if and only if the change is a breaking change. Also, min feature version value should be increased if and only if a feature version needs to be deprecated. Non-breaking changes or non-deprecating changes could continue to be made to the Broker code without modifying the feature version values.

Workflows:

• Upgrades: Most common workflow for the feature versioning system would involve finalizing a cluster upgrade to max feature version levels for all features, after completing the release of a new Kafka Broker binary on a cluster. This is explained  under Regular CLI tool usage section.
• Downgrades: Emergency downgrades of max version levels of features are rare, especially after they are finalized  (because these are breaking changes, and finalizing these changes can have consequences). For such rare situations, the user could still downgrade specific features using the CLI tool. Also, all features can be downgraded .  If the user is setting the `allowDowngrade` field in an `UpdateFeaturesRequest`, then they must have had a good reason to attempt downgrading. That's because, the CLI tool only allows a downgrade in conjunction with specific flags and sub-commands:
  • In the CLI tool, if the user passes '--allow-downgrade' flag when updating a specific feature, then the tool will translate this ask to setting 'allowDowngrade' field in the request to the server.
  • Emergency downgrade of all features can be attempted using the CLI tool belonging to a specific Broker release. This is by using the 'downgrade-all' sub-command. For examples of these, please refer to the Tooling support section.
  • Note: Please note that some of the downgrades may still be blocked by Controller if it is deemed impossible to downgrade.

Use case: group_coordinator feature flag (for KIP-447)

...