DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Each error code always represent the same class and rarely rely on client state to determine how to handle. Additionally, while it is good to have a mapping, It is also useful to have a general strategy – ie a typical unknown error (not specified by the client to have a type) should probably be application recoverable.
NOTENote: For Retriable errors, the producer handles retries internally, keeping the failure details hidden from the application. Conversely, other types of exceptions will be surfaced to the application code for handling.
...
| Code Block |
|---|
public class TransactionalClientDemo {
private static final String CONSUMER_GROUP_ID = "my-group-id";
private static final String OUTPUT_TOPIC = "output";
private static final String INPUT_TOPIC = "input";
private static KafkaConsumer<String, String> consumer;
private static KafkaProducer<String, String> producer;
public static void main(String[] args) {
initializeApplication();
boolean isRunning = true;
// Continuously poll for records
while (isRunning) {
try {
try {
// Poll records from Kafka for a timeout of 60 seconds
ConsumerRecords<String, String> records = consumer.poll(ofSeconds(60));
// Process records to generate word count map
Map<String, Integer> wordCountMap = new HashMap<>();
for (ConsumerRecord<String, String> record : records) {
String[] words = record.value().split(" ");
for (String word : words) {UnknownProducerIdException
wordCountMap.merge(word, 1, Integer::sum);
}
}
// Begin transaction
producer.beginTransaction();
// Produce word count results to output topic
wordCountMap.forEach((key, value) ->
producer.send(new ProducerRecord<>(OUTPUT_TOPIC, key, value.toString())));
// Determine offsets to commit
Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>();
for (TopicPartition partition : records.partitions()) {
List<ConsumerRecord<String, String>> partitionedRecords = records.records(partition);
long offset = partitionedRecords.get(partitionedRecords.size() - 1).offset();
offsetsToCommit.put(partition, new OffsetAndMetadata(offset + 1));
}
// Send offsets to transaction for atomic commit
producer.sendOffsetsToTransaction(offsetsToCommit, CONSUMER_GROUP_ID);
// Commit transaction
producer.commitTransaction();
} catch (TransactionAbortableException e) {
// Abortable Exception: Handle Kafka exception by aborting transaction. producer.abortTransaction() should not throw abortable exception.
producer.abortTransaction();
resetToLastCommittedPositions(consumer);
}
} catch (InvalidConfigurationException e) {
// Fatal Error: The error is bubbled up to the application layer. The application can decide what to do
closeAll();
throw e;
} catch (KafkaException | ApplicationRecoverableException e) {
// Application Recoverable: The application must restart
closeAll();
initializeApplication();
}
}
} |
Full example can be accessed at: https://github.com/apache/kafka/pull/15913/files
Note: The producer.abortTransaction() method should not throw an abortable exception to avoid creating a loop.
Compatibility, Deprecation, and Migration Plan
...
Applications using newer code, as mentioned in Clientsidecodeexample, have the flexibility to adjust their exception handling logic based on the specific type of exception encountered, enabling them to take appropriate actions.
Handling for OutOfOrderSequenceException OutOfOrderSequenceException requires additional considerations which is out of scope for this KIP. OutOfOrderSequenceException and UnknownProducerIdException OutOfOrderSequenceException andUnknownProducerIdException (which extends OutOfOrderSequenceException ) will not be categorised under any of the defined exception groups in this KIP. This will not impact applications using newer code.
...