Page History

...

CreatePartitions, OffsetCommit, IncrementalAlterConfigs, CreateAcls, and DeleteAcls are issued internally by MMM through the inter-broker channel or direct coordinator calls, bypassing normal ACL checks. No explicit ACL grants are needed for these operations.

Idempotent Producer

The idempotent producers rely on producer IDs to detect duplicate writes and ensure idempotent production. To avoid conflicts with the destination cluster's producer ID space, we rewrite source producer IDs to occupy the unused negative space by applying the formula: 

destinationProducerId = -(sourceProducerId + 2)

The rationale of this formula is to keep the existing semantic of NO_PRODUCER_ID (-1) but still have a way to avoid the conflict. The CRC checksum is automatically recalculated after the producer ID changes to maintain batch integrity. Producer epochs from the source cluster are preserved exactly as they appear in the source batches. This ensures the last stable offset is correctly reflected because the producer state is updated after each append.

...

approach is proactively expires stale producer state on failover. The key insight is that during mirroring, the destination partition is read-only: no local producers exist, so all ProducerStateManager (PSM) entries originate from mirrored data. When mirroring stops, all PSM entries are stale and can be safely expired. Records from the source are stored as-is on the destination, with no PID modification, which otherwise would require a checksum recalculation. A MIRROR_PID_RESET control batch (type 7) is written to each destination partition's log during the STOPPING state transition, after the fetcher has been removed and truncation to last stable offset is complete, but before the partition becomes writable.

Image AddedFigure 5: PID Cache Reset After Failover.

The mirror partition state transitions are:

• STOPPING: Remove fetchers, truncate to LSO, persist last mirrored offsets, write MIRROR_PID_RESET barrier.
• STOPPED:  Partition is writable (terminal state, no actions).

The key follows the standard control record format (version=0, type=7). The value uses the MirrorPidResetRecord schema:

{
  "type": "data",
  "name": "MirrorPidResetRecord",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "Version", "type": "int16", "versions": "0",
      "about": "The version of the mirror PID reset record."},
    { "name": "SourceClusterId", "type": "string", "versions": "0",
      "about": "The source cluster UUID for verification."}
  ]
}

The SourceClusterId field records which source cluster the mirrored data came from, enabling future validation (e.g. detecting unexpected source cluster changes) and data provenance tracing from the log itself.

When the barrier batch is encountered during append or during log recovery, all producer entries are removed from the PSM. This ensures leaders, followers, and recovery all handle the barrier consistently. Because the partition is read-only during mirroring, all PSM entries originate from mirrored data. Expiring all entries is safe: no local producer state exists to preserve. Control batches are filtered out by the consumer fetcher via isControlBatch checks. The barrier is invisible to application consumers, just like transaction markers (commit/abort). The log dump tool is enhanced to deserialize and display MIRROR_PID_RESET records.

The barrier approach works correctly with all practical mirroring topologies:                                                                                                                                                                                                            

• Active-passive (A to B): B mirrors from A, stores records as-is. On failover, barrier expires all PSM entries. Local producers get fresh PIDs from the coordinator with no collision risk.
• Failback (A to B, then B to A): After failover, B becomes writable. Later, A starts mirroring from B and truncates its log to the LSO. A then stores B's records as-is. B's barrier record is included in the fetched data and appended to A's log, triggering PSM expiration on A. This is consistent with the general rule: when the barrier batch is encountered during append or during log recovery, all producer entries are removed from the PSM. A will write its own barrier when it eventually stops mirroring from B, producing a clean slate before A becomes writable again.
• Fan-out (A to B, A to C): B and C mirror independently from A, each with its own PSM per partition. On failover, each writes its own barrier independently.
• Fan-in (A to C, B to C, different topics): Each topic's partitions have independent PSMs. The barrier is written per partition during the STOPPING transition of each mirror.
• Chain (A to B to C): B mirrors from A, stores records as-is. C mirrors from B, stores records as-is. On failover at any point in the chain, the barrier expires all PSM entries on the stopping node. Longer chains work inductively by the same principle.

Exactly-Once Semantics

Cluster Mirroring ensures transactional consistency when stopping by truncating to the LSO. Note that this doesn’t mean it supports exactly-once semantics (EOS) across clusters, which would require synchronous communication.

...