Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Status

Current state: Under Voting  Complete 

Discussion thread: here

JIRA: here 

...

Each error code always represent the same class and rarely rely on client state to determine how to handle. Additionally, while it is good to have a mapping, It is also useful to have a general strategy – ie a typical unknown error (not specified by the client to have a type) should probably be application recoverable.


NOTENote: For Retriable errors, the producer handles retries internally, keeping the failure details hidden from the application. Conversely, other types of exceptions will be surfaced to the application code for handling.

...

Cross Exception handling doesn't match as mentioned in KIP-691

RetriableException

Exception/Error Names

Current handling

New Handling


Producer API

Transaction API

Producer API

Transaction API

Comments

CorruptRecordException

NotEnoughReplicasAfterAppendException

NotEnoughReplicasException

TimeoutException

Retriable if the error is retriable. Otherwise abortable

Retriable if the error is retriable otherwise abortable

Producer Retriable Image Modified

Producer Retriable Image Modified


ConcurrentTransactionsException

Retriable

Retriable

Producer Retriable (KIP-890 may see this)

Producer Retriable


CoordinatorLoadInProgressException

Retriable

Retriable

Producer Retriable

Producer Retriable


RefreshRetriableException

Exception/Error Names

Current handling

New Handling


Producer API

Transaction API

Producer API

Transaction API

Comments

UnknownTopicOrPartitionException

NotLeaderOrFollowerException

Retriable if the error is retriable. Otherwise abortable

Retriable if the error is retriable otherwise abortable

Refresh + Retriable Image Modified

Refresh + Retriable Image Modified

Both UnknownTopicOrPartitionException and NotLeaderOrFollowerException  extends InvalidMetadataException 


Current class hierarchy in code is:

InvalidMetadataException < RetriableException < ApiException



New class hierarchy in code will be:

InvalidMetadataException < RefreshRetriableException < RetriableException < ApiException

NotCoordinatorException

CoordinatorNotAvailableException

Retriable

Retriable

Refresh + Retriable

Refresh + Retriable


TransactionAbortableException

Exception/Error Names

Current handling

New Handling



Producer API

Transaction API

Producer API

Transaction API

Comments

TransactionAbortableException
** note – Added via KIP-890

Producer Abortable

Producer Abortable

Producer Abortable

Producer Abortable


ApplicationRecoverableException

Exception/Error Names

Current handling

New Handling



Producer API

Transaction API

Producer API

Transaction API

Comments

InvalidProducerEpochException

IllegalStateException

Abortable

Fatal

Application Recoverable Image Removed

Sometimes Fatal depending on whether application or Sender caused issue. See: kafka: KAFKA-14831: Illegal state errors should be fatal in transactional producerCLOSED

Application Recoverable

Image Removed

ProducerFencedException

Fatal

Fatal

(probably not expected) Image Added

Application RecoverableImage Modified

Application Recoverable Image Removed

InvalidPidMappingException

Sometimes retriable (no longer returned) or abortable

Abortable

Application Recoverable Image Removed

Application Recoverable Image Removed

FencedInstanceIdException

CommitFailedException

UnknownMemberIdException

IllegalGenerationExceiption

N/A

Abortable (TxnOffsetCommit Only)

N/A

Application Recoverable Image Removed

CorrelationIdMismatchException

Application Recoverable

Application Recoverable

We will handle all default exceptions as generic unknown errors, which will be application recoverable. Below are few such exceptions:

Exception/Error Names

Current handling

New Handling

Producer API

Transaction API

Producer API

Transaction API

IllegalStateException

Abortable

Sometimes Fatal depending on whether application or Sender caused issue. See: kafka: KAFKA-14831: Illegal state errors should be fatal in transactional producerCLOSED

Application Recoverable (probably not expected)


We will handle all default exceptions as generic unknown errors, which will be application recoverable


KafkaException

Abortable (default seems to be abortable)

Fatal in most cases, but abortable when there are partition errors

Application Recoverable (not expected) Image Added

Application Recoverable (not expected) Image Added

RuntimeException

Abortable (default seems to be abortable)

Fatal – only thrown as this generic type when correlation ID is wrong. This should be updated as KIP-691 suggests

Application Recoverable (not expected) Image Added

Application Recoverable (not expected) Image Added

InvalidProducerEpochException

Abortable

Fatal

Application Recoverable Image Added

Application Recoverable Image Added


ProducerFencedException

Fatal

Fatal

Application Recoverable

Image Modified

Application Recoverable Image Modified

KafkaException


InvalidPidMappingException

Abortable (default seems to be abortable)

Fatal in most cases, but abortable when there are partition errors

Application Recoverable (not expected)

Sometimes retriable (no longer returned) or abortable

Abortable

Application Recoverable Image Modified

Application Recoverable

(not expected) Image RemovedRuntimeException

Exception/Error Names

Current handling

Expected Handling

Producer API

Transaction API

Producer API

Transaction API

Image Added


FencedInstanceIdException

CommitFailedException

UnknownMemberIdException

IllegalGenerationExceiption

N/A

Abortable (

default seems to be abortable)

Fatal – only thrown as this generic type when correlation ID is wrong. This should be updated as KIP-691 suggests

Application Recoverable (not expected) Image Removed

Application Recoverable (not expected) Image Removed

InvalidTxnStateException requires special handling where producer API and Transactional API has different handling:

TxnOffsetCommit Only)

N/A

Application Recoverable Image Added

CommitFailedException extends KafkaException, which is default exception handled as Application Recoverable. This class will not be extended by ApplicationRecoverableException

CorrelationIdMismatchException



Application Recoverable

Application Recoverable

CorrelationIdMismatchException extends IllegalStateException, which is default exception handled as Application Recoverable. This class will not be extended by ApplicationRecoverableException

InvalidTxnStateException

Abortable

Fatal

Producer Abortable (KIP-890 relies on this – note this is the only one that differs with Produce API) Image Modified

Application Recoverable Image Modified

InvalidTxnStateException requires special handling where producer API and Transactional API has different handling


InvalidConfigurationException

Exception/Error Names

Current handling

Expected Handling


Producer API

Transaction API

Producer API

Transaction API

Comments

AuthenticationException

Abortable

Fatal

Invalid Configuration Image Modified

Invalid Configuration (not expected) Image Modified


ClusterAuthorizationException

TransactionalIdAuthorizationException

UnsupportedVersionException

UnsupportedForMessageFormatException

Fatal, except UnsupportedForMessageFormatException which is Abortable

Cluster/Transaction Auth → abortable on InitProducerId, other errors abortable

TransactionAuth → fatal, others abortable for AddPartitions, Find Coordinator, EndTxn, AddOffsets

Transaction Auth, UnssupportedForMessageFormat → fatal for OffsetCommit

Invalid Configuration Image Modified

Invalid Configuration Image Modified


InvalidRecordException

InvalidRequiredAcksException

RecordBatchTooLargeException

InvalidTopicException

Retriable if the error is retriable. Otherwise abortable

Retriable if the error is retriable otherwise abortable

Invalid Configuration Image Modified

Invalid Configuration Image Modified


TopicAuthorizationException

GroupAuthorizationException

Abortable

Abortable

Invalid Configuration Image Removed

Invalid Configuration Image Removed

GroupAuthorizationException

Abortable

Abortable

Invalid Configuration Image Added

Invalid Configuration Image Added

Both TopicAuthorizationException and GroupAuthorizationException extends AuthorizationException 


Current class hierarchy in code is:

TopicAuthorizationException < AuthorizationException < ApiException

New class hierarchy in code will be TopicAuthorizationException < AuthorizationException < InvalidConfigurationException < ApiException`


Public Interfaces

We have below exception classes available in current Kafka code

Code Block
// Producer Abortable Transaction
public class TransactionAbortableException extends ApiException {
    ...
}

// Producer AbortableRetriable Transaction 
public abstract class TransactionAbortableExceptionRetriableException extends ApiException {
   ...
}


// Invalid-Configuration
public class InvalidConfigurationException extends ApiException {
 	...
}


We will add new exception types, as listed in the table below, that extend the existing exceptions.

Code Block
//Producer Refresh and Retriable  
public abstract class RetriableExceptionRefreshRetriableException extends ApiExceptionRetriableException {
   	...
}


// Invalid-ConfigurationApplication-Recoverable new
public abstract class InvalidConfigurationExceptionApplicationRecoverableException extends ApiException {
  	  ...
}


We will add new exception types, as listed in the table below, that extend the existing exceptions.extend below existing exceptions types to maintain class hierarchy 

Code Block
//Producer Refresh and Retriable
public abstract UnknownTopicOrPartitionException and NotLeaderOrFollowerException  extends InvalidMetadataException 
public class RefreshRetriableExceptionInvalidMetadataException extends RetriableExceptionRefreshRetriableException {
	...
}

//Application-Recoverable new TopicAuthorizationException and GroupAuthorizationException extends AuthorizationException 
public abstract class ApplicationRecoverableExceptionAuthorizationException extends ApiExceptionInvalidConfigurationException {
    	...
}


Client side code example

Code Block
public class TransactionalClientDemo {

    private static final String CONSUMER_GROUP_ID = "my-group-id";
    private static final String OUTPUT_TOPIC = "output";
    private static final String INPUT_TOPIC = "input";
    private static KafkaConsumer<String, String> consumer;
    private static KafkaProducer<String, String> producer;

    public static void main(String[] args) {
        initializeApplication();

        boolean isRunning = true;
        // Continuously poll for records
        while (isRunning) {
            try {
                try {
                    // Poll records from Kafka for a timeout of 60 seconds
                    ConsumerRecords<String, String> records = consumer.poll(ofSeconds(60));

                    // Process records to generate word count map
                    Map<String, Integer> wordCountMap = new HashMap<>();

                    for (ConsumerRecord<String, String> record : records) {
                        String[] words = record.value().split(" ");
                        for (String word : words) {UnknownProducerIdException                             words) {
						  wordCountMap.merge(word, 1, Integer::sum);
                        }
                    }

                    // Begin transaction
                    producer.beginTransaction();

                    // Produce word count results to output topic
                    wordCountMap.forEach((key, value) ->
                            producer.send(new ProducerRecord<>(OUTPUT_TOPIC, key, value.toString())));

                    // Determine offsets to commit
                    Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>();
                    for (TopicPartition partition : records.partitions()) {
                        List<ConsumerRecord<String, String>> partitionedRecords = records.records(partition);
                        long offset = partitionedRecords.get(partitionedRecords.size() - 1).offset();
                        offsetsToCommit.put(partition, new OffsetAndMetadata(offset + 1));
                    }

                    // Send offsets to transaction for atomic commit
                    producer.sendOffsetsToTransaction(offsetsToCommit, CONSUMER_GROUP_ID);

                    // Commit transaction
                    producer.commitTransaction();
                } catch (TransactionAbortableException e) {
                    // Abortable Exception: Handle Kafka exception by aborting transaction. producer.abortTransaction() should not throw abortable exception.
                    producer.abortTransaction();
                    resetToLastCommittedPositions(consumer);
                }
            } catch (InvalidConfigurationException e) {
                //  Fatal Error: The error is bubbled up to the application layer. The application can decide what to do
                closeAll();
                throw e;
            } catch (KafkaException | ApplicationRecoverableException e) {
                // Application Recoverable: The application must restart
                closeAll();
                initializeApplication();
            }
        }

    }

Full example can be accessed at: https://github.com/apache/kafka/pull/15913/files 


Note: The producer.abortTransaction() method will not throw an abortable exception to avoid creating a loop.

Compatibility, Deprecation, and Migration Plan

...

Currently, the transactional producer.send returns retriable exception types, such as TimeoutException , which poses a risk of duplicates in Kafka. In this KIP, we will update the transactional producer to return TransactionAbortableException  instead of TimeoutException .send path such that all retriable exceptions will be translated to TransactionAbortableException. Older clients that are using the transactional producer and handling TimeoutException  by retrying the produce operation can update to handle TransactionAbortableException .

...

Applications using newer code, as mentioned in Clientsidecodeexample, have the flexibility to adjust their exception handling logic based on the specific type of exception encountered, enabling them to take appropriate actions.

Handling for OutOfOrderSequenceException OutOfOrderSequenceException  requires additional considerations which is out of scope for this KIP.  OutOfOrderSequenceException  and UnknownProducerIdException  OutOfOrderSequenceException andUnknownProducerIdException (which extends OutOfOrderSequenceException  ) will not be categorised under any of the defined exception groups in this KIP. This will not impact applications using newer code.

...