Page History

the messages to the broker

Kafka currently provides at-least-once messaging guarantees. Duplicates can arise due to either producer retries or consumer restarts after failure. One way to provide exactly-once messaging semantics is to implement an idempotent producer. This has been covered at length in the proposal for an Idempotent Producer. An alternative and more general approach is to support transactional messaging. This can enable use-cases such as replicated logging for transactional data services in addition to the classic idempotent producer use-cases.

...

Furthermore, we have the following implementation requirementsrequirements pertaining to performance, usability and implementation complexity:

1. The implementation should be scalable. E.g., a dedicated log per transaction is unacceptable.
2. Performance:
   1. The throughput of a transactional producer should be comparable to that of a non-transactional producer.
   2. Acceptable latency. E.g., avoid copying the transactional data as much as possible.
   3. Any implementation should not make the partition unavailable (say, due to locking) for an unreasonable period of time.
3. Client simplicity: Favor a scheme that lend to a simpler client-side implementation (even if it adds more complexity to the broker). For example, it is acceptable (but not ideal) for a consumer implementation to (internally) buffer and subsequently discard messages from uncommitted transactions. i.e., if the chosen implementation allows the broker to materialize messages from uncommitted transactions in the data logs.

...