Page History

...

Cluster Mirroring synchronizes group offsets from the source cluster to the destination cluster, enabling consumers to resume consumption from their last position after failover. Both traditional consumer groups and share consumer groups (Kafka Queue functionality) are supported, with offset synchronization running in two separate phases to prevent cross-type conflicts. For consumer groups, committed offsets are periodically fetched from the source and applied to the destination. For share groups, which use a different offset management model based on Share-Partition Start Offset (SPSO) and Share-Partition End Offset (SPEO), the current SPSO is retrieved from the source and applied to the destination, which also initializes the group state in both the group coordinator and share coordinator. This means a share group can be initialized in the destination cluster even if it doesn't exist yet, eliminating the need for pre-creation or complex state management.                                                                                                

Each phase lists only groups of its own type on both source and destination, preventing a consumer group on the source from overwriting a share group with the same name on the destination (or vice versa). Kafka enforces that consumer group and share group names must be unique within a single cluster. When a name conflict does occur across types, the offset commit operation will fail for the affected group, which is logged and skipped without affecting other groups. Users can resolve these conflicts by deleting the conflicting group in the destination cluster or excluding it from offset synchronization. These conflicts affect only offset synchronization and do not impact data mirroring itself.

During offset synchronization, the committed offset in the destination cluster may temporarily exceed the current log end offset (LEO) of the mirror topic. For example, if a consumer commits offset 100 in the source cluster but the destination cluster has only mirrored up to offset 80, the offset 100 is still committed to the destination. This is acceptable because the mirror leader continues fetching data and the LEO will eventually catch up. However, if a failover occurs before the mirrored data catches up, consumers attempting to resume from offset 100 will receive an OffsetOutOfRangeException. To handle this gracefully, consumers should configure auto.offset.reset=latest when consuming from mirror topics, ensuring that if a committed offset is beyond the current LEO after failover, the consumer automatically resets to the latest available offset.

...