Versions Compared

Old Version 74

changes.mady.by.user Federico Valeri

Saved on

compared with

New Version 75

changes.mady.by.user Unknown User (omnia)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Add advanced streaming (like Flink) DR gaps with MM2 to motivations

...

  • Operational Burden: MM2 runs as standalone Connect workers external to Kafka brokers, requiring separate deployment, monitoring, and lifecycle management. Operators must provision additional hosts, manage Connect-specific configurations, and coordinate MM2 upgrades independently from Kafka broker upgrades.
  • Compression Cost: If source cluster records are compressed, MM2 will decompress and compress them again when producing to the destination cluster. These redundant operations decrease the mirroring throughput and increase latency.
  • Lossy Offset: The offset translation process in MM2 is inherently lossy. When translating an offset from the source cluster to the target cluster, MM2 cannot guarantee returning the exact same record. It is not possible to maintain a complete in-memory mapping of source to target offsets for all mirrored records. When an exact translation is unavailable, MM2 guarantees that the record at the translated offset is always earlier than the actual record, ensuring consuming applications never skip data at the cost of potential reprocessing. This conservative approach can lead to significant duplicate processing during failover scenarios, particularly for high-throughput topics where offset translation granularity is coarse.
  • External Offset Management: Advanced streaming platforms (like Apache Flink and Apache Spark), Kafka Connect source connectors and exactly-once semantics (EOS) consumer applications often store consumed offsets externally rather than in Kafka's __consumer_offsets topic. During failover to a mirrored topic on the destination cluster, these applications face additional complexity because source and destination offsets don't match. Applications must query MM2's offset-sync internal topic to translate offsets, adding operational complexity and potential failure points during critical failover scenarios. This offset translation dependency complicates disaster recovery procedures and increases the risk of incorrect offset mapping leading to data loss or duplicate processing.

Goals

Cluster Mirroring addresses these operational challenges by integrating cross-cluster replication directly into Kafka brokers, providing a simpler and more robust solution for cross-cluster replication.

...