Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

None of these fields are producer-specific. The naming is historical -- producerId and producerEpoch could equally be called sessionId and sessionEpoch. The wire protocol RPCs (InitProducerId, AddPartitionsToTxn, EndTxn, TxnHeartbeat) carry only these identity fields. They do not carry a "client type" discriminator.

What KIP-939 Solved

KIP-939 enabled Kafka to act as a formal participant in Two-Phase Commit (2PC) transactions by:

  • Adding Recovery APIs: Introduced prepareTransaction() and completeTransaction(), allowing external systems to coordinate Kafka's commit/abort status.

  • Resilient Resumption: Allowed new producers to resume a "prepared" transaction after a crash without automatically aborting it.

  • Removing Timeouts: Provided an enable2Pc mode that sets transaction timeouts to infinity, preventing premature aborts during external coordination.

The Unsolved Problem

While KIP-939 added the right tools, it put them in the wrong place:

  • Wrong Abstraction: Transaction methods (like preparing and completing) are forced into the KafkaProducer class, even though they don't involve producing records.

  • Heavyweight Bloat: To simply commit a transaction, you are forced to instantiate a full KafkaProducer with its entire infrastructure (buffer pools, sender threads, serializers).

  • Inaccessible Logic: The core "brain" of transactions—the state machine and coordinator logic—is buried in internal producer packages, making it impossible for other clients (like consumers or custom coordinators) to use transactions independently.


1.2 Entities Already Need Transaction Participation

...

EntityRoleCurrent WorkaroundCore Pain Point
KafkaProducerOwnerNative APITransaction logic is monolithic and coupled to the produce path.
Kafka StreamsEOS ProcessorWraps ProducerLifecycle is forced to match record batching.
Connect SourceEOS IngestWraps ProducerBoundary management is coupled to producer lifecycle.
Connect SinkKIP-1302N/ANo way to share transaction identity between consumer and producer.
Flink / 2PCExternal CommitterReflectionFragile; manually forces state into TransactionManager internals.
Share ConsumerKIP-1289N/AMust "borrow" Producer IDs without a proper client abstraction.


...

2. Public Interfaces

2.1 New Class: TransactionSession

Package: org.apache.kafka.clients.transaction

TransactionSession is a lightweight, thread-safe object that holds transaction identity and provides lifecycle operations. It does NOT produce records, consume records, or manage batching/serialization. It owns only the coordinator interaction.client focused strictly on transaction coordinator interaction.

It decouples lifecycle management from record production.

```java

public

...

class

...

TransactionSession

...

implements

...

Closeable

...

{

    public TransactionSession(Map<String, Object> configs);

    public static TransactionSession resume(
        String transactionalId,
        long producerId,
        short producerEpoch,
        Map<String, Object> configs
    );

    // --- Lifecycle ---
    public void initialize();
    public void beginTransaction();
    public PreparedTxnState prepareTransaction();
    public void completeTransaction(PreparedTxnState preparedTxnState);
    public void commitTransaction();
    public void abortTransaction();

    // --- Identity ---
    public String transactionalId();

    // --- Participation ---
    public void addPartitionsToTransaction(Collection<TopicPartition> partitions);
    
    public void sendOffsetsToTransaction(
        Map<TopicPartition, OffsetAndMetadata> offsets,
        ConsumerGroupMetadata groupMetadata
    );

    public void addShareAcksToTransaction(
        String groupId,
        Collection<ShareAcknowledgment> acknowledgments
    );

    // --- Liveness ---
    public void heartbeat();

    @Override
    public void close();
}


```

2.2 Configuration

TransactionSession accepts a subset of existing producer configs plus one new config:

...