A producer ID (PID) is a 64-bits long that has the following rules:

Identity: a PID identifies exactly one logical idempotent producer within a partition's log.
Stability: a logical idempotent producer always maps to the same PID, regardless of topology changes or broker restarts.
Positivity: local PIDs are non-negative, allocated sequentially from 0. PID -1 means non-idempotent producer.

Current approach: simple stateless transformation

The following PID transformation is applied before appending mirrored data to the destination cluster:

-(PID + 2)

This is a simple negation that maps all non-negative PIDs into the negative space. The +2 offset avoids mapping PID 0 to 0 and keeps PID -1 (non-transactional) untouched.

Problems

There are a couple of problems with this approach that are evident when looking at the chained mirroring use case.

Non-idempotent transformation

When B mirrors to C, PIDs already negative from A get re-transformed: -((-7) + 2) = 5, which restores the original PID and collides with local producers on C.

 A --------> B --------> C
-1 -------> -1 -------> -1
 5 -------> -7 --------> 5 # positive

PID collisions when having local producers

Even if we make the mapping idempotent by skipping negative PIDs, when A has local PID 5 and B also has local PID 5, both map to -7 on any downstream cluster. These are different producers, but they become indistinguishable. The PID cache stores the transformed PID with no awareness of its origin, so a collision silently overwrites the previous entry breaking txn consistency within the log.

A --------> B --------> C
5 -------> -7 -------> -7
            5 -------> -7 # collision

The consequences of a PID collision cannot be ignored:

Scenario 1 (same epoch, wrong sequence): No OutOfOrderSequenceException. Batches are silently accepted under the same PID as they are coming from the leader (append origin == REPLICATION). The PID cache entry is updated with whatever arrives last. Silent data corruption with zero signals, not even a warning.
Scenario 2 (different epochs): No fencing exception. Lower epoch batch is accepted with a warning log. Both producers coexist under the same PID. Silent corruption, only a WARN log line as a hint.
Scenario 3 (transactional interleaving): Commit/abort markers from one producer close the other's transaction. No exception. Silent transaction corruption.

New approach: bit-field mapping with collision detection

Apply a stateless producer ID transformation for mirrored records using a bit-field layout that encodes the source cluster identity into the negative PID space, making the mapping idempotent and safe for chained mirroring (A -> B -> C). Negative PIDs pass through unchanged (idempotency). We divide the 64 bits into three fields: bit 63 (sign bit), bits 62-31 (region selector), bits 30-0 (producer identity). This means that the negative PID space is partitioned into 4.29 billion (2^32) fixed-capacity regions, one for each possible source cluster, and we can have 2.15 billions (2^31) PIDs for each region. For example, this is how a PID 5 from two different source clusters would be encoded:

sign (bit 63)       cid-hash 42 (32 bits)             pid 5 (31 bits)
            1|00000000000000000000000000101010|0000000000000000000000000000101
            1|00000000000000000000000001100011|0000000000000000000000000000101
                    cid-hash 99 (32 bits)

Scenarios

The following scenarios are failing with current approach, but working with the bit-field approach.

Chained with local producer

In this scenario we have different producers with the same PID running on different source clusters.

A ----------> B ----------> C ----------> D
5 -------> F(42,5) ------> F(42,5) ----> F(42,5)
              5 ---------> F(99,5) ----> F(99,5)
                            5 ---------> F(17,5) # all unique

Source cluster reassignment

In this scenario we have C first mirroring from A, then B, then A again.

  A ----> C   # phase 1: A's PID 5 = F(42,5)
  B ----> C   # phase 2: B's PID 5 = F(99,5)
  A ----> C   # phase 3: A's PID 5 = F(42,5) # same producer, same pid

Collision detection

With this new approach, a PID collision requires two things to happen simultaneously:

Two source clusters hash to the same 32-bit region. Collision risk is ~1 in 4 billion per cluster pair.
Both clusters have an active producer with the same local PID. This is likely since Kafka allocates them sequentially from 0.

With a small number of source clusters (say 2-5), the probability of collision 1 is negligible. With many sources, the birthday paradox applies: you would need ~65,000 source clusters before reaching a 50% chance that any two share a hash. That is well beyond any realistic deployment.

Given the consequences, it is prudent to also have a resolution strategy. We introduce the sourceClusterId optional field in ProducerSnapshot.json schema, bumping the version to 2. The sourceClusterId will be set only for mirrored PIDs, null for local producers.

{
  "name": "SourceClusterId",
  "type": "string",
  "versions": "2+",
  "nullableVersions": "2+",
  "default": "null",
  "about": "The source cluster ID for mirrored producer IDs, null for local producers"
}

Collision detection checks the PID cache for existing entries with a different source cluster ID, rehashing with an incrementing salt on conflict and forcing a PID cache snapshot.