...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, when multiple producers can write to the same partitionWhen using the current Producer APIs, the resulting state of the log is hard to predict or control: it's possible for a producer to end up writing duplicate messages to Kafka, or have multiple competing tasks write to the same commit log and create an inconsistent state.

...

Any two publishers running the same code will agree on the offset for a particular message, so pipelining for the publisher is safe. If the publish succeeds for a message at offset n, this implies that the previous messages in the series have all completed successfully as welltoo.

Commit Log

In other cases, Suppose we may want to use a Kafka partition as a 'commit log' for a key-value store some task's state — where every operation that changes the state is committed to the log, so it can be recovered replayed on failure. This is a bit different than the first case, since Here, the partition is not just an output stream, but also a point of coordination. Duplicates are also a problem here, but we also need to make sure we don't interleave the writes from two concurrent producers. (Suppose we two servers A and B that are both accepting writes, and two concurrent increment operations for key0 and key1. If A tries to increment key0 before key1, and B tries to increment key1 before key0, we don't want to end up with a commit log that holds the increment of key0 from A and then the increment of key0 from B.) In particular, this means pipelining does not seem to be safe if operations are not idempotent.In particular: if we (temporarily) have duplicate instances of the task writing to the same partition, it's important to avoid interleaving messages from the two producers. (If not, we might end up with an invalid series of operations in the log and be unable to recover our task's state.)

Imagined usage:

1. Bootstrap the state until the very end of the partition. Let's say the upcoming offset is N.
2. While we're accepting writes:
   • Accumulate a set of messages and send a publish request with expected offsets set.
   • If the publish fails, continue from step 2. Otherwise, go back to step 1.

...