DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Cluster Mirroring addresses these operational challenges by integrating cross-cluster replication directly into Kafka brokers, providing a simpler and more robust solution for cross-cluster replication.
Figure 1: Cluster Mirroring Setup.
- Integrated Architecture: Replication logic runs within broker processes, eliminating external dependencies and reducing the operational footprint.
- Simplified Configuration: Creating a cluster mirror requires a single command-line invocation or Admin API call with bootstrap servers and security credentials.
- Metadata Synchronization: Topic configurations, consumer group offsets, and ACLs are periodically synchronized from source to destination cluster without additional configuration.
- Unified Monitoring: Mirroring metrics are exposed through standard Kafka broker JMX metrics alongside existing replication metrics. Administrators use familiar tools and dashboards to monitor cross-cluster replication.
- Faster Failover: The failover operation is simplified because metadata synchronization is continuous and automatic. Consumer applications can resume processing immediately after switching clusters without any offset translation.
- Delta Failback: Destination leader acts as a follower with regards to source leader, so it will always fetch from the local log end offset to catch up with the leader, making it possible to mirror only the delta when failing back (reverse mirroring).
- Version Compatibility: For migration or DR use cases where only failover is needed without failback, this proposal supports any Kafka version from v2.1 onward as the source cluster, leveraging the client/broker forward compatibility introduced in v4.0.
...
Cluster Mirroring introduces a coordinator-based architecture integrated into Kafka brokers for managing cross-cluster replication. The design consists of three primary components that work together to provide automatic metadata synchronization and data replication. The following diagram illustrates how these components are wired together.
Figure 2: High Level Architecture.
The mirror name is stored as a topic-level configuration (mirror.name) that propagates through Kafka's metadata log as configuration change records. When topics are added to a mirror via the addTopicsToMirror API, the controller generates configuration records that are replicated to all brokers through the standard metadata update mechanism.
...
- State Management: Mirror configuration and partition states are stored in the internal topic. The coordinator loads the state on startup and partition leadership changes.
- Partition Assignment: Cluster mirrors are assigned to coordinator partitions using consistent hashing based on the mirror name. This distributes coordinator load across all brokers and allows for horizontal scaling. The number of coordinator partitions is configurable via mirror.topic.num.partitions.
- Leader Election: When a broker becomes the leader for a __mirror_state partition, it loads the mirror metadata for all mirrors assigned to that partition and begins coordinating those mirrors. On resignation, it clears its in-memory state to avoid stale metadata.
- Metadata Refresh Scheduling: The coordinator schedules periodic metadata refresh operations by invoking a metadata manager every 30 seconds by default. This ensures that topology changes, configuration updates, and offset commits in the source cluster are continuously propagated to the destination cluster.
- State Transitions: The coordinator manages asynchronous state transitions for mirror partitions. Each partition is an independent replication unit with its own state. When the coordinator is the leader for a mirror partition, it writes the state updates directly to the internal topic. Remote brokers read and write partition state via new RPCs, enabling distributed coordination across the cluster. Both local and remote state updates trigger callbacks to execute appropriate actions for each state.
Figure 3: Mirror Partition Lifecycle.
States descriptions:
- UNKNOWN: The partition has no cached state (broker just became leader, state not loaded yet). Not an explicit API-driven state, just the absence of state.
- PREPARING: The coordinator for this partition detects via onMetadataUpdate that it leads a mirror partition. It fetches last mirrored offsets from the source cluster and truncate logs to align the local log with the source. Valid from: null, UNKNOWN, STOPPED, FAILED.
- MIRRORING: All ISR members have completed truncation. A MirrorFetcherThread is started to continuously replicate records from the source cluster. Valid from: PREPARING only.
- STOPPING: Triggered by RemoveTopicsFromMirror API (user wants to fail over) or topic deletion on the source. The system records the last mirrored offset to the internal topic. Valid from: PREPARING, MIRRORING.
- STOPPED: Last mirrored offsets have been persisted. The topic becomes writable on the destination cluster (the mirror fetcher is removed and the read-only flag is cleared). Valid from: STOPPING only.
- FAILED: An error occurred. Can be entered from any state. Can transition back to PREPARING to retry. Valid from any state.
...
A mirror leader partition begins with an unknown source leader epoch. When it sends Fetch requests to the source cluster, the source leader may respond with a FencedLeaderEpochException. When such an error occurs, the mirror fetcher extracts the current source leader epoch from the error response and updates its internal fetch state to track the source cluster's actual leader epoch. The last fetched epoch is always set to empty to disable log divergence checks due to unclean leader election (see non-goals section).
Figure 4: Mirror Leader Fetch State.
On subsequent Fetch requests:
...