A producer ID (PID) is a 64-bits long that has the following rules:

1. Identity: a PID identifies exactly one logical idempotent producer within a partition's log.
2. Stability: a logical idempotent producer always maps to the same PID, regardless of topology changes or broker restarts.
3. Positivity: local PIDs are non-negative, allocated sequentially from 0. PID -1 means non-idempotent producer.

Current approach: simple stateless transformation

The following PID transformation is applied before appending mirrored data to the destination cluster:

-(PID + 2)

This is a simple negation that maps all non-negative PIDs into the negative space. The +2 offset avoids mapping PID 0 to 0 and keeps PID -1 (non-transactional) untouched. 

Problems

There are a couple of problems with this approach that are evident when looking at the chained mirroring use case.

Non-idempotent transformation

When B mirrors to C, PIDs already negative from A get re-transformed: -((-7) + 2) = 5, which restores the original PID and collides with local producers on C.

 A --------> B --------> C
-1 -------> -1 -------> -1
 5 -------> -7 --------> 5 # positive

PID collisions when having local producers

Even if we make the mapping idempotent by skipping negative PIDs, when A has local PID 5 and B also has local PID 5, both map to -7 on any downstream cluster. These are different producers, but they become indistinguishable. The PID cache stores the transformed PID with no awareness of its origin, so a collision silently overwrites the previous entry breaking txn consistency within the log.

A --------> B --------> C
5 -------> -7 -------> -7
            5 -------> -7 # collision

The consequences of a PID collision cannot be ignored:

• Scenario 1 (same epoch, wrong sequence): No OutOfOrderSequenceException because Cluster Mirroring fetchers use REPLICATION as append origin and most checks are skipped because records are coming from the leader (source of truth). Producers' batches are silently accepted under the same PID. The ProducerStateManager entry is updated with whatever arrives last. Silent data corruption with zero signals, not even a warning.
• Scenario 2 (different epochs): No fencing exception. Lower epoch batch is accepted with a warning log. Both producers coexist under the same PID. Silent corruption, only a WARN log line as a hint.
• Scenario 3 (transactional interleaving): Commit/abort markers from one producer close the other's transaction. No exception. Silent transaction corruption.

New approach: bit-field mapping with collision detection

Apply a stateless producer ID transformation for mirrored records using a bit-field layout that encodes the source cluster identity into the negative PID space, making the mapping idempotent and safe for chained mirroring (A -> B -> C). Negative PIDs pass through unchanged (idempotency). We divide the 64 bits into three fields: bit 63 (sign bit), bits 62-31 (region selector), bits 30-0 (producer identity). This means that the negative PID space is partitioned into 4.29 billion (2^32) fixed-capacity regions, one for each possible source cluster, and we can have 2.15 billions (2^31) PIDs for each region. For example, this is how a PID 5 from two different source clusters would be encoded:

sign (bit 63)       cid-hash 42 (32 bits)             pid 5 (31 bits)
            1|00000000000000000000000000101010|0000000000000000000000000000101
            1|00000000000000000000000001100011|0000000000000000000000000000101
                    cid-hash 99 (32 bits)

Scenarios

The following scenarios are failing with current approach, but working with the bit-field approach.

Chained with local producer

In this scenario we have different producers with the same PID running on different source clusters.

A ----------> B ----------> C ----------> D
5 -------> F(42,5) ------> F(42,5) ----> F(42,5)
              5 ---------> F(99,5) ----> F(99,5)
                            5 ---------> F(17,5) # all unique

Source cluster reassignment

In this scenario we have C first mirroring from A, then B, then A again.

  A ----> C   # phase 1: A's PID 5 = F(42,5)
  B ----> C   # phase 2: B's PID 5 = F(99,5)
  A ----> C   # phase 3: A's PID 5 = F(42,5) # same producer, same pid

Collision detection tradeoff

With this new approach, a PID collision requires two things to happen simultaneously: 

1. Two source clusters hash to the same 32-bit region. Collision risk is ~1 in 4 billion per cluster pair.
2. Both clusters have an active producer with the same local PID. This is likely since Kafka allocates them sequentially from 0.

With a small number of source clusters (say 2-5), the probability of collision 1 is negligible. With many sources, the birthday paradox applies: you would need ~65,000 source clusters before reaching a 50% chance that any two share a hash. That is well beyond any realistic deployment.

Given the consequences, it is prudent to also add a resolution strategy. We introduce the sourceClusterId optional field in ProducerSnapshot.json schema, bumping the version to 2. The sourceClusterId will be set only for mirrored PIDs, null for local producers.

{
  "name": "SourceClusterId",
  "type": "string",
  "versions": "2+",
  "nullableVersions": "2+",
  "default": "null",
  "about": "The source cluster ID for mirrored producer IDs, null for local producers"
}

Collision detection checks the PID cache for existing entries with a different source cluster ID, rehashing with an incrementing salt on conflict and forcing a PID cache snapshot.