...

• Separate Thread Pools: Cross-cluster fetcher threads run in a dedicated thread pool, which is independent from the intra-cluster fetcher thread pool. This separation ensures that cross-cluster replication latency does not impact local replica synchronization.
• Network I/O Overhead: Read-only leaders perform additional network I/O to fetch from source clusters. This overhead is proportional to the number of mirror partitions and the replication throughput. Brokers with many mirror partitions may experience increased CPU usage for network processing and data serialization.
• Memory Footprint: Each mirror fetcher thread maintains its own fetch session state, partition state map, and response buffers. With default configuration, memory overhead is comparable to standard replica fetchers. The metadata manager maintains connection pools and metadata caches, adding minimal memory overhead.
• Bandwidth Consumption: Cross-cluster traffic between source and destination clusters consumes WAN bandwidth. For large-scale deployments, operators should provision adequate inter-datacenter connectivity or configure throttling.
• State Management: Mirror partition state management is evenly distributed to available brokers to avoid any hot spot, especially during rolling update or restart events.

Future Work

Synchronous mirroring: Currently, mirroring is asynchronous. The source cluster acknowledges the producer without waiting for the destination to replicate the data. Sync mirroring would guarantee that records are replicated to the destination cluster before the source acknowledges the produce request, providing stronger durability guarantees at the cost of higher latency. This would be useful for workloads where zero data loss across clusters is a strict requirement.

Future extensions to synchronous mirroring could enable preservation of transactional semantics across clusters. Streaming platforms using exactly-once mode (Apache Kafka Streams, Apache Flink, Apache Spark) rely on the source cluster's transactional protocol and coordination. During failover or migration scenarios, transactional metadata for pending transactions does not transfer to the destination cluster, potentially breaking exactly-once guarantees. Supporting transactional cross-cluster replication would require coordinating transactional metadata and ensuring transaction state consistency across clusters—something MM2's Connect-based architecture cannot support.

Tiered storage: Mirror topics in the destination cluster currently only replicate data from local storage on the source broker. Integrating with tiered storage would allow mirroring to handle data that has been offloaded to remote storage (e.g., S3, HDFS), enabling full replication of topics with long retention periods without requiring all data to reside in local broker storage.

Diskless topics: Diskless topics store data exclusively in tiered storage, with no local log segments on brokers. Supporting mirroring for diskless topics requires adapting the fetch and replication mechanisms to work without local storage, which introduces changes to how mirror offsets are tracked and how truncation is handled during failover.

Source cluster mirroring replication quota: Add source-side throttling that allows source cluster leaders to limit bandwidth served to all mirror fetchers, similar to how leader.replication.throttled.rate controls intra-cluster replication. This provides independent control over mirror catch-up traffic without impacting local replication or consumer workloads. Combined with destination-side throttling, operators gain complete bidirectional bandwidth control for mirror traffic.

Test Plan

Unit Tests

Unit tests will cover individual component behavior:

• MirrorCoordinator: State loading, partition assignment, metadata persistence.
• MirrorMetadataManager: Topic creation, config sync, offset commit, ACL sync.
• MirrorFetcherThread: Epoch tracking, fetch processing, leader change handling.
• MirrorCommand: Command-line parsing, Admin API invocation, error handling.
• Protocol Serialization: Extended and new APIs serialization and deserialization.

Integration Tests

Integration tests will validate end-to-end functionality across multiple brokers:

• CLI Workflow: Create mirror with kafka-mirrors.sh, add topics, verify replication
• Basic Replication: Create mirror via API, replicate topic, verify data consistency
• Metadata Sync: Modify topic config in source, verify automatic sync to destination
• Partition Expansion: Add partitions to source topic, verify destination expands
• Consumer Groups: Commit offsets in source, verify replication to destination
• ACL Replication: Create ACL in source, verify creation in destination
• Leader Changes: Trigger leader election in source, verify fetcher reconnects
• Broker Failures: Stop destination broker, verify replication continues after recovery

System Tests

System tests will validate behavior under realistic production conditions:

...

Test Plan

Unit Tests

Unit tests will cover individual component behavior:

• MirrorCoordinator: State loading, partition assignment, metadata persistence.
• MirrorMetadataManager: Topic creation, config sync, offset commit, ACL sync.
• MirrorFetcherThread: Epoch tracking, fetch processing, leader change handling.
• MirrorCommand: Command-line parsing, Admin API invocation, error handling.
• Protocol Serialization: Extended and new APIs serialization and deserialization.

Integration Tests

Integration tests will validate end-to-end functionality across multiple brokers:

• CLI Workflow: Create mirror with kafka-mirrors.sh, add topics, verify replication
• Basic Replication: Create mirror via API, replicate topic, verify data consistency
• Metadata Sync: Modify topic config in source, verify automatic sync to destination
• Partition Expansion: Add partitions to source topic, verify destination expands
• Consumer Groups: Commit offsets in source, verify replication to destination
• ACL Replication: Create ACL in source, verify creation in destination
• Leader Changes: Trigger leader election in source, verify fetcher reconnects
• Broker Failures: Stop destination broker, verify replication continues after recovery

System Tests

System tests will validate behavior under realistic production conditions:

• Performance Benchmark: Measure replication throughput and latency across WAN.
• Scalability Test: Replicate 1000 topics with 100,000 partitions across clusters.
• Failover Test: Simulate source cluster failure, measure consumer recovery time.
• Long-Running Stability: Run continuous replication for 7 days, verify no memory leaks or performance degradation.
• Security Validation: Test all authentication mechanisms (SASL PLAIN, SCRAM, Kerberos, mTLS) via kafka-mirrors.sh config files.

Future Work

Synchronous mirroring: Currently, mirroring is asynchronous. The source cluster acknowledges the producer without waiting for the destination to replicate the data. Sync mirroring would guarantee that records are replicated to the destination cluster before the source acknowledges the produce request, providing stronger durability guarantees at the cost of higher latency. This would be useful for workloads where zero data loss across clusters is a strict requirement.

Future extensions to synchronous mirroring could enable preservation of transactional semantics across clusters. Streaming platforms using exactly-once mode (Apache Kafka Streams, Apache Flink, Apache Spark) rely on the source cluster's transactional protocol and coordination. During failover or migration scenarios, transactional metadata for pending transactions does not transfer to the destination cluster, potentially breaking exactly-once guarantees. Supporting transactional cross-cluster replication would require coordinating transactional metadata and ensuring transaction state consistency across clusters—something MM2's Connect-based architecture cannot support.

Tiered storage: Mirror topics in the destination cluster currently only replicate data from local storage on the source broker. Integrating with tiered storage would allow mirroring to handle data that has been offloaded to remote storage (e.g., S3, HDFS), enabling full replication of topics with long retention periods without requiring all data to reside in local broker storage.

Diskless topics: Diskless topics store data exclusively in tiered storage, with no local log segments on brokers. Supporting mirroring for diskless topics requires adapting the fetch and replication mechanisms to work without local storage, which introduces changes to how mirror offsets are tracked and how truncation is handled during failover.

Source cluster mirroring replication quota: Add source-side throttling that allows source cluster leaders to limit bandwidth served to all mirror fetchers, similar to how leader.replication.throttled.rate controls intra-cluster replication. This provides independent control over mirror catch-up traffic without impacting local replication or consumer workloads. Combined with destination-side throttling, operators gain complete bidirectional bandwidth control for mirror traffic

...

.

Rejected Alternatives

Keep Using MirrorMaker 2

...