DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
A producer ID (PID) is a 64-bits long that has the following rules:
- Identity: a A PID identifies exactly one logical idempotent producer within a partition's log.
- Stability: a A logical idempotent producer always maps to the same PID, regardless of mirror topology changes or broker restarts.
- Positivity: local Local PIDs are non-negative, allocated sequentially from 0. PID -1 means non-idempotent producer.
Without PID mapping, two independent clusters can assign the same producer ID to different producers. When records from both clusters are mirrored into the same destination partition, the ProducerStateManager (PSM) sees two unrelated producers sharing one PID.
Current approach: simple stateless transformation
...
When B mirrors to C, PIDs already negative from A get re-transformed: -((-7) + 2) = 5, which restores the original PID and collides with local producers on C.
| Code Block |
|---|
A - A B C D -1 -------> B-1 --------> C -1 -------> -1 5 -------> -1 57 --------> -7 -5 -------> -7 5 -------> 5-7 # positivecollision |
PID collisions when having local producers
Even if we make the mapping idempotent by skipping negative PIDs, when A has local PID 5 and B also has local PID 5, both map to -7 on any downstream cluster. These are different producers, but they become indistinguishable. The PID PSM cache stores the transformed PID with no awareness of its origin, so a collision silently overwrites the previous entry breaking txn consistency within the log.
| Code Block |
|---|
A --------> B --------> B C 5 -------> -7 -------> -7 5 -------> -7 # collision |
...
- Scenario 1 (same epoch, wrong sequence): No OutOfOrderSequenceException. Batches are silently accepted under the same PID as they are coming from the leader (append origin == REPLICATION). The PID PSM cache entry is updated with whatever arrives last. Silent data corruption with zero signals, not even a warning.
- Scenario 2 (different epochs): No fencing exception. Lower epoch batch is accepted with a warning log. Both producers coexist under the same PID. Silent corruption, only a WARN log line as a hint.
- Scenario 3 (transactional interleaving): Commit/abort markers from one producer close the other's transaction. No exception. Silent transaction corruption.
...
In this scenario we have different producers with the same PID running on different source clusters.
| Code Block |
|---|
A ----------> B ----------> C ----------> B C D 5 -------> F(42,5) ------> F(42,5) ----> F(42,5) 5 ------------> F(99,5) ----> F(99,5) 5 ----------> F(17,5) # all unique |
...
Collision detection checks the PID PSM cache for existing entries with a different source cluster ID, rehashing with an incrementing salt on conflict and forcing a PID PSM cache snapshot.