Page History

...

1. For correctness reasons, the controller shall allow processing of only one inflight ApiKeys.UPDATE_FEATURES API call at any given time.

2. Prior to mutating '/features' in ZK, the implementation verifies that all broker advertisements (as seen by it thus far) contained no incompatibilities with the provided List<FeatureUpdate> (see Validations section below).

   1. Note that feature version deletions (using a version level < 1) can be used in circumstances where a finalized feature needs to be garbage collected. This type mutates '/features' in ZK without any guarding checks on feature versions.

3. Once validations in #2 are successful, the List<FeatureUpdate> are applied to contents of '/features' ZK node. Upon successful write, ZK should automatically increment the version of the node by 1.

4. Finally, an UpdateFeatureResponse is returned to the user.

Validations

For any some <feature_name>, the above API implementation guards against a change for the related entry in '/features' from {"max_version_level": X} to {"max_version_level": X’}, unless, it notices that each live broker in the deployment advertises {"maxVersion": Y >= X’} and {"minVersion": Z <= X’} in it’s BrokerIdZnode (for the same <feature_name>imagine that:

• X is the existing finalized feature max version level specified in '/features' ZK node.
• Y is the new finalized feature max version level, specified in the ApiKeys.UPDATE_FEATURES API request.

Then, the API implementation allows a change from X to Y in the ZK node, only if either of the following is true for the same <feature_name>:

• Y falls within the  [min_version, max_version] range advertised by each live broker in the cluster.
• Y is < 1, indicating a request for feature deletion (see #2 below).

Note the following rules that are applied:

1. The usual regular usage of the API is a feature upgrade. This is by specifying a higher new max feature version level value (i.e.  Y > X' > X).

2. By default, the API disallows max feature version level downgrades (X' < Y < X) and finalized feature deletions (X' < Y < 1). Such changes are allowed to be attempted only if the allowDowngrade flag is specified. Note that despite this allowDowngrade flag being set, certain downgrades may be rejected by the controller if it is deemed impossible.

3. If any broker does not contain a required feature, this is considered an incompatibility → such a case will fail the API request.

4. If any broker contains an additional feature that’s not required → this is not considered an incompatibility.

5. Some/all of the above logic will also be used by the broker (not just the controller) for it’s protections (see this section of this KIP).

6. Activating the effects of a feature version cluster-wide is left to the discretion of the logic implementing the feature (ex: can be done via dynamic broker config).

7. Deprecating/eliminating the presence/effects of a specific feature version cluster-wide is left to the discretion of the logic backing the feature (ex: can be done via dynamic broker config). However, it may need further external verification to ensure no entity (ex: a consumer, or a broker) is actively using the feature (see Non-goals section).

...

• A "breaking change" typically happens whenever the underlying Broker logic has introduced a different concept (such as a message type, or a new field in a protocol) which only certain newer versions of the Broker code can understand/process. The older Broker versions could consider such a change to be alien i.e. the older Broker versions could exhibit undefined or terminal behavior if they notice effects of the breaking change. Similarly, Kafka clients could exhibit undesirable behavior if they commence usage of a feature version on the cluster, assuming all Brokers support it, while in reality some Brokers don't.
  As part of Kafka operations, there are occasions when it becomes necessary to safely activate certain breaking changes introduced by a newer Broker version. Such changes become good candidates to be released behind a feature flag . The cluster-wide max feature version level can be safely increased/finalized (using the provided mechanism in this KIP), after the cluster-wide deployment of the required Broker version is over. With the versioning system in place, whenever the effects of the underlying breaking change is activated by the cluster operator, the system protects the cluster and Kafka clients from bad effects due to breaking changes.
• As a guideline, max supported feature version values should be increased if and only if the change is a breaking change. Also, min feature version value should be increased if and only if a feature version needs to be deprecated. Non-breaking changes or non-deprecating changes could continue to be made to the Broker code without modifying the feature version values.
• As a guideline, whenever each feature's supported version range is modified, or if feature versions are permanently deprecated, these should be documented in the upgrade section of Kafka release notes.

What are the common workflows?

...