THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
The schema for the advertised information is similar to the one in /features ZK node (see this section). Here is an example of a BrokerIdZnode with the proposed additional metadata towards the bottom.
...
The node data shall be readable via existing ZK tooling. An eventually consistent copy of the node data shall also be made readable via existing ApiKeys.API_VERSIONS API served by any broker in the cluster. (see this section)
During regular operations, the data in the ZK node can be mutated only via a specific admin API served only by the controller.
...
To help explain things better, here are the request and response definitions for the new API (also see section #6 showing related pseudocode for the Admin API):
...
For correctness reasons, the controller shall allow processing of only one inflight ApiKeys.UPDATE_FEATURESAPI call at any given time.
Prior to mutating /features in ZK, the implementation verifies that all broker advertisements (as seen by it thus far) contained no incompatibilities with the provided List<FeatureUpdate> (see Validations section below).
Once validations in #2 are successful, the List<FeatureUpdate> are applied to contents of /features ZK node, and it’s __data_version__ is incremented by 1.
If #3 is successful, ApiKeys.UPDATE_METADATA requests are issued to all brokers in the cluster. This can incur a slight latency cost, so we expect this API call to succeed/fail in worst case within few seconds (depending on the size of the cluster).
Finally, an UpdateFeaturesResponse is returned to the user. The metadata version number is the latest __data_version__ in the ZK node: /features after the mutations were applied to it.
...
For the above guarding check, in each live broker,
The absence of a required feature is considered an incompatibility → such a case will fail the API request.
Whereas, if any broker contains an additional feature that’s not required → this is not considered an incompatibility.
Some/all of the above logic will also be used by the broker (not just the controller) for it’s protections (see this section of this KIP).
Enabling the actual semantics of a feature version cluster-wide is left to the discretion of the logic implementing the feature (ex: can be done via dynamic broker config).
Deprecating/eliminating the semantics of a specific feature version cluster-wide is left to the discretion of the logic backing the feature (ex: can be done via dynamic broker config). However, it may need further external verification to ensure no entity (ex: a consumer, or a broker) is actively using the feature (see Non-goals section).
Guarantees
When a call to the ApiKeys.UPDATE_FEATURES API returns a success return code to the client, the following is guaranteed:
Each FeatureUpdate provided in the request was valid, and all updates were persisted to the /features ZK node by the controller (along with a bump to the node’s __data_version__ ).
An ApiKeys.UPDATE_METADATA call was successfully made by the controller to all live brokers (as seen by the controller) with the latest dictionary of cluster-wide finalized feature versions.
Future ApiKeys.UPDATE_METADATA calls from controller to brokers will contain the latest set of cluster-wide feature versions, enabling brokers to shutdown when incompatibilities are detected (see Broker protections section).
The returned object in the ApiKeys.UPDATE_FEATURES API call contains the latest set of cluster-wide feature versions (after all updates were applied), which can be useful to the client for debugging/logging purposes.
...
In the UpdateMetadataRequest request handler path, we will populate the MetadataCache with the latest list of finalized features as sent to it by the controller (also see Broker protections section in this doc).
We will modify the implementation of the ApiKeys.API_VERSIONS API to populate the information about finalized features in the response. This will contain the cluster-wide finalized versions from the broker’s internal MetadataCache (as explained above), as well as the broker-level supported versions.
The finalized feature versions populated in the response will be eventually consistent, since it is served from information propagated via ApiKeys.UPDATE_METADATA API by the controller.
...
NOTE: The logic for the validations will be exactly the same as the one described under Validations sub- section under Controller API.
Tooling support
...
The stream clients shall not downgrade to group_coordinator:v0 once they upgrade to group_coordinator:v1, even if the leader instructs them to do so. This is because, the feature version in the DescribeFeatures for group_coordinator:v1 is higher than earlier response. This will be therefore used by the client to ignore the earlier response from the broker. (see this section to learn about the monotonically increasing metadata version)
It is important to note that the user is only upgrading the cluster, from enabling group_coordinator:v0 to enabling group_coordinator:v0-v1. Therefore, once a client sees that v1 is enabled on any broker, it should be a clear signal that all broker binaries within the cluster would support group_coordinator:v1 semantic, it’s just a matter of time before feature version update metadata propagates to every broker from controller.
...
group_coordinator: As described in the above section.
transaction_coordinator: This feature flag could cover changes to the message format for the transaction state internal topic schema, and protocol changes related to transaction coordinator. The advantages are similar to #2.
consumer_offsets_topic_schema: Currently the message format used in the consumer internal topics is tied to IBP. It’s a candidate for a feature, because we will be able to dynamically finalize a new version of the topic schema, without having to roll the brokers twice.
inter_broker_protocol: For transitional purposes, the inter.broker.protocol itself can be made a coarse-grained feature. This could be a way to operationally migrate away avoiding the double roll during IBP bumps.
...