...

Kafka transaction uses a two-phase model, where the transaction state is stored by transaction coordinator. As of current, the way to make it work is through AddPartitionToTxn call each time we see a new output partition. In reality though, the set of output partition for a producer doesn't change very often, or even deterministic over time, so it is a waste of network round trips to update that set unnecessarily.

Similarly for partition offsets, it is very unlikely to have the transactional producer register more than one consumer group, which means the consumer offset topic partition shall be repeatedly updated. It is favorable to save this round trip as well.

The other issue is the infinite retry mechanism for AddPartitionToTxn when being served as the starting request for a new transaction. Right now the EndTxn call returns immediately when the txn commit message is materialized in transaction log. However broker needs to propagate this information to all the affected parties to complete the transaction, which means there is a black-out period that no more transaction could be started.

...

We propose to make AddPartitionToTxn a blocking call when there is other pending transaction going on. We would also like to retain a transaction session so that we don't need to call AddPartitionsToTxn all the time when a new partition is emerged. On the other hand, we decide to support a pre-registration API for applications that already know which partitions to write to.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

...

Binary log format

...

The network protocol and api behavior

...

Any class in the public packages under clientsConfiguration, especially client configuration

• org/apache/kafka/common/serialization

• org/apache/kafka/common

• org/apache/kafka/common/errors

• org/apache/kafka/clients/producer

• org/apache/kafka/clients/consumer (eventually, once stable)

...

Monitoring

...

Command line tools and arguments

We shall bump the EndTxn to include the actual written partitions so that the session gets updated. 

We would also bump beginTransaction API on the KafkaProducer to give user the access for including the pre-registered partitions. The only exception is that during the transaction session the offset topic is still able to be added.

    /**
     * Should be called before the start of each new transaction. Note that prior to the first invocation
     * of this method, you must invoke {@link #initTransactions()} exactly one time.
     *
     * @throws IllegalStateException if no transactional.id has been configured or if {@link #initTransactions()}
     *         has not yet been invoked
     * @throws ProducerFencedException if another producer with the same transactional.id is active
     * @throws org.apache.kafka.common.errors.UnsupportedVersionException fatal error indicating the broker
     *         does not support transactions (i.e. if its version is lower than 0.11.0.0)
     * @throws org.apache.kafka.common.errors.AuthorizationException fatal error indicating that the configured
     *         transactional.id is not authorized. See the exception for more details
     * @throws KafkaException if the producer has encountered a previous fatal error or for any other unexpected error
     */
    public void beginTransaction(Set<TopicPartitions> outputPartitions) throws ProducerFencedException;

...

Compatibility, Deprecation, and Migration Plan

...