Page History

...

This KIP proposes to make changes in the existing protocol so that the coordinator ignores record types that are unknown to the coordinator and tie that to the old protocol’s IBP/feature version. The existing implementation throws an IllegalStateException when the record key is unknown. As we expect the log cleaner to eventually clean up new record types in the log, we don’t have to manually add tombstones. Note that this is only true if consumers are downgraded first. We can add WARN level logs to indicate that the coordinator read an unknown record type. We can automate the cleanup by writing tombstones when the coordinator reads unrecognizable records. This may add duplicate work if tombstones were already added but not yet pruned by the log cleaner. Although we can force the log cleaner to run before downgrading this is not strictly necessary and will slow down the downgrade. This also acts as a safeguard for downgrades where consumers were not fully downgraded first.

Note that this approach is only applicable to the __consumer_offsets topic because the coordinator appends tombstones when members leave and groups become empty. This approach proposes a way to handle new record types that will be introduced to the __consumer_offsets topic in the future and not just in KIP-848. However, server logs may flood with WARN level logs if we don't proactively remove them so we propose to have the new group coordinator append tombstones for all newly introduced record types when a group is converted from consumer to generic as we do so in KIP-848.

...

Once OffsetCommitValue is bumped to a flexible version, we can safely assume that the downgraded coordinator can downgrade any record type that are unknown to the latest known version and ignore unknown tagged fields. However, once we introduce tagged fields to OffsetCommitValue we can never downgrade the cluster to a version older than the first flexible version (4). This is okay if we use feature flag versioning and introduce a flexible OffsetCommitValue record for the first group coordinator version. On the other hand, if we stick to IBP/MV then there isn't Unfortunately there is no a clear solution at the moment. 

OffsetCommitValue.json

This KIP introduces the tag, taggedVersions properties to the topicId field.

...