DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- New transient state TX_PENDING
- On WriteTxnMarkers commit: TX_PENDING(ACCEPT) → ACKNOWLEDGED; same for RELEASE and REJECT.
- On WriteTxnMarkers abort: TX_PENDING(*) → back to ACQUIRED (lock continues; consumer can retry the work).
Pseudo code
| Code Block |
|---|
// Kafka-as-destination path (true EOS via TxnShareAcknowledge): batch boundary (or per-record, depending on tx granularity): producer.beginTransaction() for each (record, output) in batch: One-time setup shareConsumer.subscribe(List.of("source-topic")); producer.initTransactions(); // exactly once per producer instance // CTP loop while (running) { ConsumerRecords<K, V> records = shareConsumer.poll(Duration.ofSeconds(5)); if (records.isEmpty()) continue; try { producer.beginTransaction(); Map<TopicIdPartition, Acknowledgements> acks = new HashMap<>(); for (ConsumerRecord<K, V> record : records) { V output = process(record); producer.send(new ProducerRecord<>("destination-topic", output)); acks.computeIfAbsent( new TopicIdPartition(record.topicId(), new TopicPartition(record.topic(), record.partition())), k -> Acknowledgements.empty() ).add(record.offset(), AcknowledgeType.ACCEPT); // accumulate } // Compress acks to AcknowledgementBatch wire form and stage in the txn. producer.sendShareAcknowledgementsToTransaction( toBatches(acks), // Map<TopicIdPartition, List<AcknowledgementBatch>> shareConsumer.shareGroupMetadata() producer.send(destinationTopic, output); producer.sendShareAcknowledgementsToTransactioncommitTransaction( ); // BLOCKS — true ackMap,synchronisation point // Records are now both produced AND acknowledged ATOMICALLY. // Do NOT // record→ACCEPT for processed records shareConsumer.shareGroupMetadata() call shareConsumer.acknowledge(...) on these records. } catch (ProducerFencedException | UnsupportedVersionException fatal) { ) throw fatal; producer.commitTransaction() // No separate consumer.acknowledge() — the ACK is in the transaction. ------------------- // Pure-external-sink path (at-least-once + idempotent destination):[this KIP is not changin anything of this flow] worker thread per record: output = process(record) externalSink.write(output, idempotencyKey) // unrecoverable — kill the process } catch (KafkaException abortable) { producer.abortTransaction(); // BLOCKS — txn reverted; staged records revert to ACQUIRED on broker // userRecords taskwill does this consumer.acknowledgeAsync(record, ACCEPT) .thenAccept(_ -> markComplete(record))be re-delivered on next poll(); retry naturally. } } |
Corner cases
Here are your corner cases formatted using the same clean, un-phased, human-style layout. The wording, titles, technical specifications, and internal error codes remain exactly as provided.
...
Unit Tests: Validates state transitions (e.g.,
ACQUIREDtoACKNOWLEDGEDon commit vs.AVAILABLEon abort), idempotency of operations, and transaction timeout/auto-abort logic.Integration Tests: Focuses on end-to-end commit/abort flows, coordinator recovery, and multi-consumer behavior within a single group during network partitions.
System & Performance Tests: Benchmarks transactional vs. non-transactional modes and verifies exactly-once delivery.
Chaos Tests: Simulates broker and coordinator crashes specifically during critical phases like
PREPARE_COMMITto ensure protocol durability.
Follow-up KIP (deferred)
...
- Persistent TX_PENDING in __share_group_state for crash-resilient EOS on broker failover during staging (addresses Corner Cases).
...
- Per-share-partition metric for TX_PENDING residency time— would add a histogram of "time spent in TX_PENDING" useful for diagnosing slow producers; can be added in the persistence KIP without compatibility concerns.
- External processing engines or database as 2PC participants for write/sink records.
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.