Page History

...

Supported feature flags depend on the version of the code and represent the capabilities of the current binary. A broker defines a minimum and maximum supported version for each feature flag. Finalized feature flags are dynamic and set by an operator using the “kafka-features.sh” script. The operator defines a minimum and maximum finalized version for the given feature flag which is used to convey the “in-use” versions version of the feature within the cluster.  To To date, no features in Kafka are utilizing these feature flags.

...

Define new KIP-584 feature flag "metadata.version" with an initial version of zero. 

Introduce an IBP version to indicate the lowest software version that supports metadata.version. Below this IBP, the metadata.version is undefined and will not be examined. At or above this IBP, the metadata.version must be 0 for ZooKeeper clusters and will be initialized as 1 for KRaft clusters.

Replace "AllowDowngrade" with "DowngradeType" in UpdateFeaturesRequest. Also add new "DryRun" field to correspond with the existing --dry-run  flag.

...

• Operator performs rolling restart of cluster with a new software version
• Operator increases metadata.version feature flag using kafka-features.sh tool
  • UpdateFeaturesRequest is sent to the active controller
  • The controller validates that the cluster can be upgraded to this version
  • FeatureLevelRecord is written to the metadata log
  • Metadata snapshot is generated and sent to the other nodes
  • Components reload Components reload their state with new version

...

• Operator decreases metadata.version feature flag using kafka-features.sh tool
  • UpdateFeaturesRequest is sent to the active controller
  • The controller validates that the cluster can be safely downgraded to this version (override with --force)
  • FeatureLevelRecord is written to the metadata log
  • Metadata Controller generates new snapshot is generated and sent to the other inactive controllers and to brokers and components reload their state with it (this snapshot may be lossy!)
  • Broker replicates FeatureLevelRecord for downgrade
  • Broker generates new snapshot and components Components reload their state to recognize the new (old) versionwith it (this snapshot may be lossy!)
• Operator performs rolling restart of cluster with downgraded software version

...

One major difference with the static IBP-based upgrade is that the metadata.version may be changed arbitrarily at runtime. This means broker and controller components which depend on this version will need to dynamically adjust their state and behavior as the version changes. The quorum controller can assist with this process by generating a metadata snapshot after a metadata.version increase has been committed to the metadata log. This snapshot will be a convenient way to let broker and controller components rebuild their entire in-memory state following an upgrade. 

ApiVersions

Now that the RPCs in-use by a broker or controller can change at runtime (due to changing metadata.version), we will need a way to inform a node's remote clients that new RPCs are available. Brokers will be able to observe changes to metadata.version by observing the metadata log, and could then submit a new ApiVersionsRequest to the other Kafka nodes. In order for clients to learn about new broker RPCs, closing the connection would trigger a new ApiVersionsRequest to be sent by the client. We may want to investigate alternative approaches here in a future KIP.

...

One of the goals of this design is to provide a clear path for metadata.version downgrades and software downgrades. Since metadata.version can introduce backwards incompatible formats of persisted data, we can classify downgrades into lossless and lossy. If the target downgrade version is fully compatible with the origin starting version, the downgrade can be executed without any loss of metadata. However, if a new metadata record has been introduced, or an incompatible change was made to a record, a downgrade is only possible if some metadata is removed from the log.

...

When performing a lossless downgrade, no modifications are made to the metadata records. A snapshot is generated which includes the FeatureLevelRecord that downgrades the metadata.version. Metadata consumers, including brokers and controllers, will read records at the new new (lower) metadata.version level. Since a lossless downgrade requires fully backwards compatible changes, the only differences in the record formats can be tagged fields.

If a new record type, or new required fields are added, a lossy downgrade is possiblerequired. In this case, the snapshot generated by the controller will controllers and brokers will exclude new record types and will write metadata records at the downgraded version (thereby excluding new fields). By default, the controller should not perform this kind of downgrade since crucial metadata may be lost. The addition of --forceunsafe  in the kafka-features.sh tool (and the corresponding ForceDowngrade DowngradeType  field value of "2" in UpdateFeaturesRequest) is meant to override this behavior. Using this the example above, a lossy downgrade from version 5 to version 2 would mean that "Bar" records would be omitted from the snapshot.

Once the downgrade snapshot has been loaded by brokersall nodes, a software downgrade is now possible. In both lossy and lossless downgrade scenarios, there may be tagged fields present in the metadata records from previous newer version, but these are transparently skipped over during record deserialization.  

...