A producer ID (PID) is a 64-bit identifier assigned by the broker to each idempotent or transactional producer. It has three key properties:
NO_PRODUCER_ID) marks non-idempotent batches.Without PID mapping, two independent clusters can assign the same producer ID to different producers. When records from both source clusters are mirrored into the same destination partition, the ProducerStateManager (PSM) sees two unrelated producers sharing one PID.
The following PID transformation is applied before appending mirrored data to the destination cluster:
-(PID + 2) |
This is a simple negation that maps all non-negative PIDs into the negative space. The +2 offset avoids mapping PID 0 to 0 and keeps PID -1 (non-idempotent) untouched.
There are a couple of problems with this approach that are evident when looking at the chained mirroring use case.
When B mirrors to C, PIDs already negative from A get re-transformed: -((-7) + 2) = 5 , which restores the original PID and collides with local producers on C.
A B C D
-1 -------> -1 -------> -1 -------> -1
5 -------> -7 --------> 5 -------> -7
5 -------> -7 # collision |
Even if we make the mapping idempotent by skipping negative PIDs, when A has local PID 5 and B also has local PID 5, both map to -7 on any downstream cluster. These are different producers, but they become indistinguishable. The PSM cache stores the transformed PID with no awareness of its origin, so a collision silently overwrites the previous entry breaking txn consistency within the log.
A B C 5 -------> -7 -------> -7 5 -------> -7 # collision |
We identified the following scenarios caused by interleaving records from a PID collision:
OutOfOrderSequenceException. Batches are silently accepted under the same PID as they are coming from the leader (append origin == REPLICATION). The PSM cache entry is updated with whatever arrives last. Silent data corruption with zero signals, not even a warning.We apply a stateless PID transformation for mirrored records using a bit-field layout that encodes the source cluster identity into the negative PID space, making the mapping idempotent and safe for chained mirroring (A -> B -> C). Negative PIDs pass through unchanged (idempotency). We divide the 64 bits into three fields: bit 63 (sign bit), bits 62-31 (region selector), bits 30-0 (producer identity). The region selector is computed with the XOR-fold of the source cluster UUID's most and least significant 64-bit halves, masked to 32 bits. This means that the negative PID space is partitioned into 4.29 billion (2^32) fixed-capacity regions, one for each possible source cluster, and we can have 2.15 billions (2^31) PIDs for each region. For example, this is how a PID 5 from two different source clusters would be encoded:
sign (bit 63) cid-hash 42 (32 bits) pid 5 (31 bits)
1|00000000000000000000000000101010|0000000000000000000000000000101
1|00000000000000000000000001100011|0000000000000000000000000000101
cid-hash 99 (32 bits) |
The following scenarios are failing with current approach, but working with the bit-field approach.
In this scenario we have different producers with the same PID running on different source clusters.
A B C D
5 -------> F(42,5) ------> F(42,5) ----> F(42,5)
5 ------------> F(99,5) ----> F(99,5)
5 ----------> F(17,5) # all unique |
In this scenario we have C first mirroring from A, then B, then A again.
A ----> C # phase 1: A's PID 5 = F(42,5) B ----> C # phase 2: B's PID 5 = F(99,5) A ----> C # phase 3: A's PID 5 = F(42,5) # same producer, same pid |
With this new approach, a PID collision requires two things to happen simultaneously:
On collision, the cluster hash is perturbed with a random salt until a free slot is found among free regions. Collision resolutions must survive leader elections. Without persistence, a new leader would recompute the mapped PID because the ephemeral origins map is lost. The computed salt is therefore stored as a record in the __mirror_state internal topic, which is replicated across the cluster. This is a synchronous call, but collisions are rare enough that the overhead is negligible.
Key schema:
# key
{
"apiKey": 3,
"type": "coordinator-key",
"name": "MirrorPidCollisionKey",
"validVersions": "0",
"flexibleVersions": "none",
"fields": [
{ "name": "TopicId", "type": "uuid", "versions": "0",
"about": "The destination topic ID where the collision occurred."},
{ "name": "Partition", "type": "int32", "versions": "0",
"about": "The partition index."},
{ "name": "SourceClusterId", "type": "string", "versions": "0",
"about": "The source cluster UUID whose producer ID was remapped."},
{ "name": "OriginalPid", "type": "int64", "versions": "0",
"about": "The original producer ID on the source cluster before mapping."}
]
}
# value
{
"apiKey": 3,
"type": "coordinator-value",
"name": "MirrorPidCollisionValue",
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{ "name": "Salt", "type": "int64", "versions": "0",
"about": "The salt added to the cluster hash to resolve the collision."},
{ "name": "CollidingClusterId", "type": "string", "versions": "0",
"about": "The cluster UUID that already held the mapped PID slot."},
{ "name": "TimestampMs", "type": "int64", "versions": "0",
"about": "Wall clock time when the collision was resolved."}
]
} |
On leader election, the coordinator loads collision records from __mirror_state and populates the collision cache. When mapping a PID, checks this cache first. If a known salt exists, it is applied directly without re-probing the origins map. A null value (tombstone) deletes the record when the corresponding producer state expires.