DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
Status
Current state: Under Voting Complete
Discussion thread: here
JIRA: here
...
Each error code always represent the same class and rarely rely on client state to determine how to handle. Additionally, while it is good to have a mapping, It is also useful to have a general strategy – ie a typical unknown error (not specified by the client to have a type) should probably be application recoverable.
NOTENote: For Retriable errors, the producer handles retries internally, keeping the failure details hidden from the application. Conversely, other types of exceptions will be surfaced to the application code for handling.
...
Cross → Exception handling doesn't match as mentioned in KIP-691
Exception/Error Names | Current handling | New Handling | |||
|---|---|---|---|---|---|
Producer API | Transaction API | Producer API | Transaction API | Comments | |
CorruptRecordException NotEnoughReplicasAfterAppendException NotEnoughReplicasException TimeoutException | Retriable if the error is retriable. Otherwise abortable | Retriable if the error is retriable otherwise abortable | Producer Retriable | Producer Retriable | |
ConcurrentTransactionsException | Retriable | Retriable | Producer Retriable (KIP-890 may see this) | Producer Retriable | |
CoordinatorLoadInProgressException | Retriable | Retriable | Producer Retriable | Producer Retriable | |
Exception/Error Names | Current handling | New Handling | |||
|---|---|---|---|---|---|
Producer API | Transaction API | Producer API | Transaction API | Comments | |
UnknownTopicOrPartitionException NotLeaderOrFollowerException | Retriable if the error is retriable. Otherwise abortable | Retriable if the error is retriable otherwise abortable | Refresh + Retriable | Refresh + Retriable | Both UnknownTopicOrPartitionException and NotLeaderOrFollowerException extends InvalidMetadataException Current class hierarchy in code is: InvalidMetadataException < RetriableException < ApiException
InvalidMetadataException < < RetriableException < ApiException |
NotCoordinatorException CoordinatorNotAvailableException | Retriable | Retriable | Refresh + Retriable | Refresh + Retriable | |
TransactionAbortableException
Exception/Error Names | Current handling | New Handling | |||
|---|---|---|---|---|---|
Producer API | Transaction API | Producer API | Transaction API | Comments | |
TransactionAbortableException | Producer Abortable | Producer Abortable | Producer Abortable | Producer Abortable | |
ApplicationRecoverableException
Exception/Error Names | Current handling | New Handling | |||
|---|---|---|---|---|---|
Producer API | Transaction API | Producer API | Transaction API | Comments | |
IllegalStateException | Abortable |
Fatal
Sometimes Fatal depending on whether application or Sender caused issue. See: kafka: KAFKA-14831: Illegal state errors should be fatal in transactional producerCLOSED | Application Recoverable |
ProducerFencedException
Fatal
(probably not expected) | Application Recoverable |
Application Recoverable
InvalidPidMappingException
Sometimes retriable (no longer returned) or abortable
Abortable
Application Recoverable
Application Recoverable
FencedInstanceIdException
CommitFailedException
UnknownMemberIdException
IllegalGenerationExceiption
N/A
Abortable (TxnOffsetCommit Only)
N/A
Application Recoverable
CorrelationIdMismatchException
Application Recoverable
Application Recoverable
We will handle all default exceptions as generic unknown errors, which will be application recoverable. Below are few such exceptions:
Exception/Error Names
Current handling
New Handling
Producer API
Transaction API
Producer API
Transaction API
IllegalStateException
Abortable
Sometimes Fatal depending on whether application or Sender caused issue. See: kafka: KAFKA-14831: Illegal state errors should be fatal in transactional producerCLOSED
We will handle all default exceptions as generic unknown errors, which will be application recoverable | |||||
KafkaException | Abortable (default seems to be abortable) | Fatal in most cases, but abortable when there are partition errors | Application Recoverable (not expected) | Application Recoverable (not expected) | |
RuntimeException | Abortable (default seems to be abortable) | Fatal – only thrown as this generic type when correlation ID is wrong. This should be updated as KIP-691 suggests | Application Recoverable (not expected) | Application Recoverable (not expected) | |
InvalidProducerEpochException | Abortable | Fatal | Application Recoverable | Application Recoverable | |
ProducerFencedException | Fatal | Fatal | Application Recoverable |
Application Recoverable |
InvalidPidMappingException |
Abortable (default seems to be abortable)
Fatal in most cases, but abortable when there are partition errors
Sometimes retriable (no longer returned) or abortable | Abortable | Application Recoverable | Application Recoverable |
Exception/Error Names
Current handling
Expected Handling
Producer API
Transaction API
Producer API
Transaction API
| ||
FencedInstanceIdException CommitFailedException UnknownMemberIdException IllegalGenerationExceiption | N/A | Abortable ( |
Fatal – only thrown as this generic type when correlation ID is wrong. This should be updated as KIP-691 suggests
Application Recoverable (not expected)
Application Recoverable (not expected)
InvalidTxnStateException requires special handling where producer API and Transactional API has different handling:
TxnOffsetCommit Only) | N/A | Application Recoverable | CommitFailedException extends KafkaException, which is default exception handled as Application Recoverable. This class will not be extended by ApplicationRecoverableException | ||
CorrelationIdMismatchException | Application Recoverable | Application Recoverable | CorrelationIdMismatchException extends IllegalStateException, which is default exception handled as Application Recoverable. This class will not be extended by ApplicationRecoverableException |
InvalidTxnStateException | Abortable | Fatal | Producer Abortable (KIP-890 relies on this – note this is the only one that differs with Produce API) | Application Recoverable | InvalidTxnStateException requires special handling where producer API and Transactional API has different handling |
InvalidConfigurationException
Exception/Error Names | Current handling | Expected Handling | |||
|---|---|---|---|---|---|
Producer API | Transaction API | Producer API | Transaction API | Comments | |
AuthenticationException | Abortable | Fatal | Invalid Configuration | Invalid Configuration (not expected) | |
ClusterAuthorizationException TransactionalIdAuthorizationException UnsupportedVersionException UnsupportedForMessageFormatException | Fatal, except UnsupportedForMessageFormatException which is Abortable | Cluster/Transaction Auth → abortable on InitProducerId, other errors abortable TransactionAuth → fatal, others abortable for AddPartitions, Find Coordinator, EndTxn, AddOffsets Transaction Auth, UnssupportedForMessageFormat → fatal for OffsetCommit | Invalid Configuration | Invalid Configuration | |
InvalidRecordException InvalidRequiredAcksException RecordBatchTooLargeException InvalidTopicException | Retriable if the error is retriable. Otherwise abortable | Retriable if the error is retriable otherwise abortable | Invalid Configuration | Invalid Configuration | |
TopicAuthorizationException | |||||
GroupAuthorizationException
Abortable
Abortable
Invalid Configuration
Invalid Configuration
GroupAuthorizationException | Abortable | Abortable | Invalid Configuration | Invalid Configuration | Both TopicAuthorizationException and GroupAuthorizationException extends AuthorizationException Current class hierarchy in code is: TopicAuthorizationException < AuthorizationException < ApiException |
Public Interfaces
We have below exception classes available in current Kafka code
| Code Block |
|---|
// Producer Abortable Transaction public class TransactionAbortableException extends ApiException { ... } // Producer AbortableRetriable Transaction public abstract class TransactionAbortableExceptionRetriableException extends ApiException { ... } // Invalid-Configuration public class InvalidConfigurationException extends ApiException { ... } |
We will add new exception types, as listed in the table below, that extend the existing exceptions.
| Code Block |
|---|
//Producer Refresh and Retriable public abstract class RetriableExceptionRefreshRetriableException extends ApiExceptionRetriableException { ... } // Invalid-ConfigurationApplication-Recoverable new public abstract class InvalidConfigurationExceptionApplicationRecoverableException extends ApiException { ... } |
We will add new exception types, as listed in the table below, that extend the existing exceptions.extend below existing exceptions types to maintain class hierarchy
| Code Block |
|---|
//Producer Refresh and Retriable public abstract UnknownTopicOrPartitionException and NotLeaderOrFollowerException extends InvalidMetadataException public class RefreshRetriableExceptionInvalidMetadataException extends RetriableExceptionRefreshRetriableException { ... } //Application-Recoverable new TopicAuthorizationException and GroupAuthorizationException extends AuthorizationException public abstract class ApplicationRecoverableExceptionAuthorizationException extends ApiExceptionInvalidConfigurationException { ... } |
Client side code example
| Code Block |
|---|
public class TransactionalClientDemo {
private static final String CONSUMER_GROUP_ID = "my-group-id";
private static final String OUTPUT_TOPIC = "output";
private static final String INPUT_TOPIC = "input";
private static KafkaConsumer<String, String> consumer;
private static KafkaProducer<String, String> producer;
public static void main(String[] args) {
initializeApplication();
boolean isRunning = true;
// Continuously poll for records
while (isRunning) {
try {
try {
// Poll records from Kafka for a timeout of 60 seconds
ConsumerRecords<String, String> records = consumer.poll(ofSeconds(60));
// Process records to generate word count map
Map<String, Integer> wordCountMap = new HashMap<>();
for (ConsumerRecord<String, String> record : records) {
String[] words = record.value().split(" ");
for (String word : words) {UnknownProducerIdException words) {
wordCountMap.merge(word, 1, Integer::sum);
}
}
// Begin transaction
producer.beginTransaction();
// Produce word count results to output topic
wordCountMap.forEach((key, value) ->
producer.send(new ProducerRecord<>(OUTPUT_TOPIC, key, value.toString())));
// Determine offsets to commit
Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>();
for (TopicPartition partition : records.partitions()) {
List<ConsumerRecord<String, String>> partitionedRecords = records.records(partition);
long offset = partitionedRecords.get(partitionedRecords.size() - 1).offset();
offsetsToCommit.put(partition, new OffsetAndMetadata(offset + 1));
}
// Send offsets to transaction for atomic commit
producer.sendOffsetsToTransaction(offsetsToCommit, CONSUMER_GROUP_ID);
// Commit transaction
producer.commitTransaction();
} catch (TransactionAbortableException e) {
// Abortable Exception: Handle Kafka exception by aborting transaction. producer.abortTransaction() should not throw abortable exception.
producer.abortTransaction();
resetToLastCommittedPositions(consumer);
}
} catch (InvalidConfigurationException e) {
// Fatal Error: The error is bubbled up to the application layer. The application can decide what to do
closeAll();
throw e;
} catch (KafkaException | ApplicationRecoverableException e) {
// Application Recoverable: The application must restart
closeAll();
initializeApplication();
}
}
} |
Full example can be accessed at: https://github.com/apache/kafka/pull/15913/files
Note: The producer.abortTransaction() method will not throw an abortable exception to avoid creating a loop.
Compatibility, Deprecation, and Migration Plan
...
Currently, the transactional producer.send returns retriable exception types, such as TimeoutException , which poses a risk of duplicates in Kafka. In this KIP, we will update the transactional producer to return TransactionAbortableException instead of TimeoutException .send path such that all retriable exceptions will be translated to TransactionAbortableException. Older clients that are using the transactional producer and handling TimeoutException by retrying the produce operation can update to handle TransactionAbortableException .
...
Applications using newer code, as mentioned in Clientsidecodeexample, have the flexibility to adjust their exception handling logic based on the specific type of exception encountered, enabling them to take appropriate actions.
Handling for OutOfOrderSequenceException OutOfOrderSequenceException requires additional considerations which is out of scope for this KIP. OutOfOrderSequenceException and UnknownProducerIdException OutOfOrderSequenceException andUnknownProducerIdException (which extends OutOfOrderSequenceException ) will not be categorised under any of the defined exception groups in this KIP. This will not impact applications using newer code.
...