Page History

...

• Operational Burden: MM2 runs as standalone Connect workers external to Kafka brokers, requiring separate deployment, monitoring, and lifecycle management. Operators must provision additional hosts, manage Connect-specific configurations, and coordinate MM2 upgrades independently from Kafka broker upgrades.
• Compression Inefficiency: If source cluster records are compressed, MM2 will decompress and compress them again when producing to the destination cluster. These redundant operations decrease the mirroring throughput and increase latency.
• Lossy Offset Translation: The offset translation process in MM2 is inherently lossy. When translating an offset from the source cluster to the target cluster, MM2 cannot guarantee returning the exact same record. It is not possible to maintain a complete in-memory mapping of source to target offsets for all mirrored records. When an exact translation is unavailable, MM2 guarantees that the record at the translated offset is always earlier than the actual record, ensuring consuming applications never skip data at the cost of potential reprocessing. This conservative approach can lead to significant duplicate processing during failover scenarios, particularly for high-throughput topics where offset translation granularity is coarse.
• External Offset Complexity: Advanced streaming platforms like Apache Flink and Apache Spark, Kafka Connect source connectors, and transactional applications often store consumed offsets externally rather than in Kafka's __consumer_offsets topic. When a failover happens, these applications face additional complexity because source and destination offsets don't match. Applications must query MM2's offset-sync internal topic to translate offsets, adding operational complexity and potential failure points. This offset translation dependency complicates DR procedures and increases the risk of incorrect offset mapping leading to data loss or duplicate processing.
• Unclean Leader Election: MM2 does not handle unclean leader elections on elections on the source cluster gracefully. When an unclean leader election occurs, data loss on the source may not be reflected on the destination, leading to divergent state between clusters.

...

• Integrated Architecture: Replication logic runs within broker processes, eliminating external dependencies and reducing the operational footprint.
• Simplified Configuration: Creating a cluster mirror requires a single command-line invocation or Admin API call with bootstrap servers and security credentials.
• Metadata Synchronization: Topic configurations, consumer group offsets, and ACLs are periodically synchronized from source to destination cluster without additional configuration.
• Unified Monitoring: Cluster Mirroring metrics are exposed through standard Kafka broker JMX metrics alongside existing replication metrics. Operators use Operators use familiar tools and dashboards to monitor cross-cluster replication.
• Faster Failover: The failover operation is simplified because metadata synchronization is continuous and automatic. Consumer applications can resume processing immediately after switching clusters without any offset translation.
• Delta Failback: Destination leader acts as a follower with regards to source leader, so it will always fetch from the local log end offset to catch up with the leader, making it possible to mirror only the delta when failing back (reverse mirroring).
• Version Compatibility: For migration or DR use cases where failback is not required, this proposal supports source brokers up to version 2.1.0 included, leveraging the client/broker forward compatibility introduced in v4.0.
• Unclean Leader ElectionULE Support: Cluster mirroring Mirroring can detect and reconcile such scenarios unclean leader elections with additional configuration, ensuring consistency across clusters.

...