DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- New transient state TX_PENDING
- On WriteTxnMarkers commit: TX_PENDING(ACCEPT) → ACKNOWLEDGED; same for RELEASE and REJECT.
- On WriteTxnMarkers abort: TX_PENDING(*) → back to ACQUIRED (lock continues; consumer can retry the work).
Pseudo code
| Code Block |
|---|
// Kafka-as-destination path (true EOS via TxnShareAcknowledge): batch boundary (or per-record, depending on tx granularity): producer.beginTransaction() for each (record, output) in batch:One-time setup shareConsumer.subscribe(List.of("source-topic")); producer.initTransactions(); // exactly once per producer instance // CTP loop while (running) { ConsumerRecords<K, V> records = shareConsumer.poll(Duration.ofSeconds(5)); if (records.isEmpty()) continue; try { producer.beginTransaction(); Map<TopicIdPartition, Acknowledgements> acks = new HashMap<>(); for (ConsumerRecord<K, V> record : records) { V output = process(record); producer.send(new ProducerRecord<>("destination-topic", output)); acks.computeIfAbsent( new TopicIdPartition(record.topicId(), new TopicPartition(record.topic(), record.partition())), k -> Acknowledgements.empty() ).add(record.offset(), AcknowledgeType.ACCEPT); // accumulate } // Compress acks to AcknowledgementBatch wire form and stage in the txn. producer.sendShareAcknowledgementsToTransaction( toBatches(acks), // Map<TopicIdPartition, List<AcknowledgementBatch>> shareConsumer.shareGroupMetadata() ); producer.send(destinationTopic, output) producer.sendShareAcknowledgementsToTransaction(commitTransaction(); // BLOCKS — true synchronisation point ackMap,// Records are now both produced AND acknowledged ATOMICALLY. // Do NOT call shareConsumer.acknowledge(...) on these records. // if called mistakenly then non-transactional acknowledge is // record→ACCEPT for processed records shareConsumer.shareGroupMetadata() )rejected (TX_PENDING to ACKNOWLEDGED - InvalidRecordStateException); the transactional path works. } catch (ProducerFencedException | UnsupportedVersionException fatal) { producer.commitTransaction() //throw Nofatal; separate consumer.acknowledge() — the ACK is in the transaction. ------------------- // Pure-external-sink path (at-least-once + idempotent destination):[this KIP is not changin anything of// this flow] worker thread per record:unrecoverable — kill the process output} =catch process(recordKafkaException abortable) { externalSink producer.write(output, idempotencyKey)abortTransaction(); // BLOCKS — txn // user task does this reverted; staged records revert to ACQUIRED on broker consumer.acknowledgeAsync(record, ACCEPT) .thenAccept(_ -> markComplete(record)) // Records will be re-delivered on next poll(); retry naturally. } } |
Corner cases
Here are your corner cases formatted using the same clean, un-phased, human-style layout. The wording, titles, technical specifications, and internal error codes remain exactly as provided.
...
1. Drain in-flight TX_PENDING:
- Halt producers calling sendShareAcknowledgementsToTransaction
- Wait for transaction.timeout.ms (default 60s) for any abandoned txns to clear
2. kafka-features.sh downgrade --feature share.version --version 1
Broker rejects the downgrade RPC if any in-memory TX_PENDING exists, preventing data inconsistency.
Software downgrade (binary): must follow feature downgrade; standard rolling restart.
Test Plan
Rough implementation [The actual implementation will be phasewise with multiple smaller PRs]:
https://github.com/apache/kafka/pull/22357
The verification strategy focuses on state machine integrity and fault tolerance under high-concurrency and failure scenarios.
Unit Tests: Validates state transitions (e.g.,
ACQUIREDtoACKNOWLEDGEDon commit vs.AVAILABLEon abort), idempotency of operations, and transaction timeout/auto-abort logic.Integration Tests: Focuses on end-to-end commit/abort flows, coordinator recovery, and multi-consumer behavior within a single group during network partitions.
System & Performance Tests: Benchmarks transactional vs. non-transactional modes and verifies exactly-once delivery.
Chaos Tests: Simulates broker and coordinator crashes specifically during critical phases like
PREPARE_COMMITto ensure protocol durability.
Follow-up KIP (deferred)
...
- Persistent TX_PENDING in __share_group_state for crash-resilient EOS on broker failover during staging (addresses Corner Cases).
...
- Per-share-partition metric for TX_PENDING residency time— would add a histogram of "time spent in TX_PENDING" useful for diagnosing slow producers; can be added in the persistence KIP without compatibility concerns.
- External processing engines or database as 2PC participants for write/sink records.
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.