Page History

...

Consumers should be downgraded first for mainly two reasons. First, consumers using the new protocol may be migrated to a downgraded group coordinator when a broker goes down. The coordinator won’t be able to understand the Heartbeat API which will cause consume downtime. This complicates the downgrade scenario because we need either a) backward compatible consumers or b) forward compatible group coordinators. Note that a) may be provided with KIP-848 but even with backward compatible consumers, consumers may flip-flop between using old vs. new protocol while partitions/groups are migrated during broker downgrades which is not favorable. However, if a broker is down during the v.

Second and more importantly, downgrading consumers first is useful because the group coordinator appends tombstones for new protocol records when a group transitions to the old protocol (when the last new protocol consumer leaves the group). Similarly it will append old protocol records to convert the group. As the group fully reverts to the old protocol the log cleaner will clean new protocol records in the background so we can expect all new record types to be eventually removed from the log assuming there are no bugs in this process. This makes it easier to reason about how a downgraded coordinator should handle these new record types.

...

To use this, we would need to have topicId modified as a tagged field as part of KIP-848 so that the downgraded coordinator can skip it. Also, we need to change the logic in GroupMetadataManager.readOffsetMessageValue() so that if the version is unknown, we deserialize it with the highest known version instead of throwing an IllegalStateException. The alternative approach is to ignore unknown versions which would work but we would lose the committed offsets after the coordinator is downgraded. Once the OffsetCommitValue record becomes flexible, we can introduce new tagged fields without a version bump as it is guaranteed by KIP-482.

OffsetCommitValue.json

This KIP introduces the tag, taggedVersions properties to the topicId field.

{
  "type": "data",  
  "name": "OffsetCommitValue",  
  "validVersions": "0-4",  
  "flexibleVersions": "4+",  
  "fields": [
    { "name": "offset", "type": "int64", "versions": "0+" },    
    { "name": "leaderEpoch", "type": "int32", "versions": "3+", "default": -1, "ignorable": true},    
    { "name": "metadata", "type": "string", "versions": "0+" },    { "name": "commitTimestamp", "type": "int64", "versions": "0+" },    
    { "name": "expireTimestamp", "type": "int64", "versions": "1", "default": -1, "ignorable": true}
    // KIP-848 Adds TopicId field. KIP-915 adds tag and taggedVersions
    { "name": "topicId", "type": "uuid", "tag": 0, "taggedVersions": "4+", "versions": "4", "ignorable": true }
  ]
}

Benefits

...

Restrictions and Guidelines

Using tagged fields has comes with some restrictions: new fields added to the OffsetCommitValue record will always need to be a tagged field. This doesn't apply to other record types because they are assumed to be deleted. New record types will always be deleted during the group conversion as mentioned above. Using tagged fields also comes with some rules. We can remove tagged fields, but will need a version bump.

From KIP-482:

1. a tag number can never be reused
2. the format of a taggedField can never be modified.

...

1. .

Once OffsetCommitValue is bumped to a flexible version, we can safely assume that the downgraded coordinator can downgrade offset commit records that are unknown to the latest known version and ignore tagged fields. 

...