...

Retry the operation without bothering the application (retriable)
Share with the application that an error has occurred, but the client can recover if the transaction is aborted by the client and under the covers, the epoch is bumped (abortable)
Share with the application that an error has occurred, and there is no way to recover except shut down the client (fatal)

Code Block

// Producer-Recoverable
public class AbortableTransactionException extends ApiException {
    public AbortableTransactionException(String message) {
        super(message);
    }
    ...
}

//Producer-Retriable 
public class ProducerRetriableTransactionException extends ApiException {
    public ProducerRetriableTransactionException(String message) {
        super(message);
    }
 	...
}

//Producer-Retriable
public class ProducerImmediateRetriableTransactionExceptionProducerRefreshMetadataRetriableTransactionException extends ApiException {
    public ProducerImmediateRetriableTransactionExceptionProducerRefreshMetadataRetriableTransactionException(String message) {
        super(message);
    }
	...
}

//Application-Recoverable
public class ApplicationRecoverableTransactionException extends ApiException {
    public ApplicationRecoverableTransactionException(String message) {
        super(message);
    }
    ...
}

// Invalid-Configuration
public class InvalidConfiguationTransactionException extends ApiException {
    public InvalidConfiguationTransactionException(String message) {
        super(message);
    }
 	...
}

// Extending exception types example
public class InvalidProducerEpochException extends AbortableTransactionException {
    private static final long serialVersionUID = 1L;
    public InvalidProducerEpochException(String message) {
        super(message);
    }
}

...

While it is good to have a mapping, I think it is also useful to have a general strategy – ie a typical unknown error (not specified by the client to have a type) should probably be application recoverable.

Exception Table

Below table contains list of exceptions with current and expected handing:

...

Cross → Exception handling doesn't match as mentioned in KIP-691

...

AbortableTransactionException

Exception/Error Names

Thrown scenarios during transaction

Current handling		Expected Handling
	Producer API	Transaction API	Producer API	Transaction API

ErrorRetry


TransactionAbortableException ** note – Added via KIP-890	Producer

Retriable

Recoverable

Producer

Retriable

Recoverable

Producer

Retriable

Recoverable

Producer

Retriable

Recoverable

ErrorRefreshMetadataAndRetry

Refresh + Retriable

ErrorFenceClient

Application Recoverable

ErrorBadConfig

Invalid Configuration

TransactionAbortableException
** note – Added via KIP-890

Producer Recoverable

IllegalStateException

Abortable

Sometimes Fatal depending on whether application or Sender caused issue. See: kafka: KAFKA-14831: Illegal state errors should be fatal in transactional producerCLOSED

Application Recoverable (probably not expected) Image Removed

Application RecoverableImage Removed

Au thenticationException

Abortable

Fatal

Invalid Configuration Image Removed

Invalid Configuration (not expected) Image Removed

InvalidPidMappingException

If the TransactionalId does not exist in the TransactionalId mapping or if the mapped PID is different from that in the request, reply with InvalidPidMapping; otherwise proceed to next step.

Sometimes retriable (no longer returned) or abortable

Abortable

Application Recoverable Image Removed

UnknownProducerIdException

Sometimes retriable (no longer returned) or abortable

Abortable

Application Recoverable Image Removed

ClusterAuthorizationException

TransactionalIdAuthorizationException

UnsupportedVersionException

UnsupportedForMessageFormatException

Fatal, except UnsupportedForMessageFormatException which is Abortable

Cluster/Transaction Auth → abortable on InitProducerId, other errors abortable

TransactionAuth → fatal, others abortable for AddPartitions, Find Coordinator, EndTxn, AddOffsets

Transaction Auth, UnssupportedForMessageFormat → fatal for OffsetCommit

Invalid Configuration Image Removed

TransactionAbortableException
** note – Added via KIP-890

Producer Recoverable

ProducerRetriableTransactionException

Exception/Error Names	Current handling		New Handling
	Producer API	Transaction API	Producer API	Transaction API
CorruptRecordException NotEnoughReplicasAfterAppendException NotEnoughReplicasException TimeoutException	Retriable if the error is retriable. Otherwise abortable	Retriable if the error is retriable otherwise abortable	Producer Retriable Image Added	Producer Retriable Image Added
ConcurrentTransactionsException	Retriable	Retriable	Producer Retriable (KIP-890 may see this)	Producer Retriable
CoordinatorLoadInProgressException	Retriable	Retriable	Producer Retriable	Producer Retriable
OutOfOrderSequenceException	Sometimes retriable or abortable	N/A	Producer Retriable Image Added	N/A

ProducerRefreshMetadataRetriableTransactionException

Exception/Error Names

Current handling

New Handling

Producer API

Transaction API

Producer API

Transaction API

UnknownTopicOrPartitionException

NotLeaderOrFollowerException

Retriable if the error is retriable. Otherwise abortable

Retriable if the error is retriable otherwise abortable

Refresh + Retriable Image Added

NotCoordinatorException

CoordinatorNotAvailableException

Retriable

Refresh + Retriable

ApplicationRecoverableTransactionException

CorruptRecordException

NotEnoughReplicasAfterAppendException

NotEnoughReplicasException

TimeoutException

Retriable if the error is retriable. Otherwise abortable

Retriable if the error is retriable otherwise abortable

Producer Retriable Image Removed

UnknownTopicOrPartitionException

NotLeaderOrFollowerException

Retriable if the error is retriable. Otherwise abortable

Retriable if the error is retriable otherwise abortable

Refresh + Retriable Image Removed

InvalidRecordException

InvalidRequiredAcksException

RecordBatchTooLargeException

InvalidTopicException

Retriable if the error is retriable. Otherwise abortable

Retriable if the error is retriable otherwise abortable

Invalid Configuration Image Removed

TopicAuthorizationException

GroupAuthorizationException

Abortable

Invalid Configuration Image Removed

FencedInstanceIdException

CommitFailedException

UnknownMemberIdException

IllegalGenerationExceiption

N/A

Abortable (TxnOffsetCommit Only)

N/A

Application Recoverable Image Removed

InvalidProducerEpochException

If the PID’s epoch number is different from the current TransactionalId PID mapping, reply with the InvalidProducerEpoch error code; otherwise proceed to next step.

Abortable

Fatal

Application Recoverable Image Removed

ProducerFencedException

Fatal

Application Recoverable Image Removed

OutOfOrderSequenceException

Sometimes retriable or abortable

N/A

Producer Retriable Image Removed

N/A

InvalidTxnStateException

When client makes a request which involves change of Txn state. However expected state transition is invalid.
e.g COMPLETE_ABORT to COMMIT

Abortable

Fatal

Producer Recoverable (KIP-890 relies on this – note this is the only one that differs with Produce API) Image Removed

Application Recoverable Image Removed

KafkaException

Abortable (default seems to be abortable)

Fatal in most cases, but abortable when there are partition errors

Exception/Error Names	Current handling		Expected Handling
	Producer API	Transaction API	Producer API	Transaction API
IllegalStateException	Abortable	Sometimes Fatal depending on whether application or Sender caused issue. See: kafka: KAFKA-14831: Illegal state errors should be fatal in transactional producerCLOSED	Application Recoverable (probably not expected) Image Added	Application RecoverableImage Added
UnknownProducerIdException	Sometimes retriable (no longer returned) or abortable	Abortable	Application Recoverable Image Added	Application Recoverable Image Added
ProducerFencedException	Fatal	Fatal	Application Recoverable Image Added	Application Recoverable Image Added
InvalidProducerEpochException	Abortable	Fatal	Application Recoverable Image Added	Application Recoverable Image Added
ProducerFencedException	Fatal	Fatal	Application Recoverable Image Added	Application Recoverable Image Added
InvalidPidMappingException	Sometimes retriable (no longer returned) or abortable	Abortable	Application Recoverable Image Added	Application Recoverable Image Added
UnknownProducerIdException	Sometimes retriable (no longer returned) or abortable	Abortable	Application Recoverable Image Added	Application Recoverable Image Added
FencedInstanceIdException CommitFailedException UnknownMemberIdException IllegalGenerationExceiption	N/A	Abortable (TxnOffsetCommit Only)	N/A	Application Recoverable Image Added
KafkaException	Abortable (default seems to be abortable)	Fatal in most cases, but abortable when there are partition errors

Application Recoverable (not expected) Image Modified	Application Recoverable (not expected) Image Modified
RuntimeException	Abortable (default seems to be abortable)	Fatal – only thrown as this generic type when correlation ID is wrong. This should be updated as KIP-691 suggests	Application Recoverable (not expected) Image Modified	Application Recoverable (not expected) Image Modified

ConcurrentTransactionsException

CorrelationIdMismatchException

Retriable

Application Recoverable

Retriable

Producer Retriable (KIP-890 may see this)

Producer Retriable

CoordinatorLoadInProgressException

Retriable

Producer Retriable

Application Recoverable

InvalidConfiguationTransactionException

Exception/Error Names	Current handling		Expected Handling
	Producer API	Transaction API	Producer API	Transaction API
AuthenticationException	Abortable	Fatal	Invalid Configuration Image Added	Invalid Configuration (not expected) Image Added
ClusterAuthorizationException TransactionalIdAuthorizationException UnsupportedVersionException UnsupportedForMessageFormatException	Fatal, except UnsupportedForMessageFormatException which is Abortable	Cluster/Transaction Auth → abortable on InitProducerId, other errors abortable TransactionAuth → fatal, others abortable for AddPartitions, Find Coordinator, EndTxn, AddOffsets Transaction Auth, UnssupportedForMessageFormat → fatal for OffsetCommit	Invalid Configuration Image Added	Invalid Configuration Image Added
InvalidRecordException InvalidRequiredAcksException RecordBatchTooLargeException InvalidTopicException	Retriable if the error is retriable. Otherwise abortable	Retriable if the error is retriable otherwise abortable	Invalid Configuration Image Added	Invalid Configuration Image Added
TopicAuthorizationException GroupAuthorizationException	Abortable	Abortable	Invalid Configuration Image Added	Invalid Configuration Image Added

InvalidTxnStateException requires special handling where producer API and Transactional API has different handling:

Exception/Error Names	Current handling		Expected Handling
	Producer API	Transaction API	Producer API	Transaction API
InvalidTxnStateException	Abortable	Fatal	Producer Recoverable (KIP-890 relies on this – note this is the only one that differs with Produce API) Image Added	Application Recoverable Image Added

NotCoordinatorException

CoordinatorNotAvailableException

Check whether there is a previous entry with the same TransactionalId and a higher epoch. If so, throw an exception. In particular, this indicates the log is corrupt. All future transactional RPCs to this coordintaor will result in a `NotCoordinatorForTransactionalId` error code, and this partition of the log will be effectively disabled.
Check if it is the assigned transaction coordinator for the TransactionalId, if not reply with the NotCoordinatorForTransactionalId error code.

Retriable

Refresh + Retriable

CorrelationIdMismatchException

Application Recoverable

Compatibility, Deprecation, and Migration Plan

This KIP involves client side changes which only affects the resiliency of new Producer client and Streams. Old clients will continue to be able to use their error handling. Based on New clients have the flexibility to adjust their exception handling logic based on the specific type of exception thrown, user needs to change their exception catching logic encountered, enabling them to take actions against their exception handlingappropriate actions as described in the Clientsidecodeexample section. Existing clients need not make any modifications, as they can continue to utilize their current error handling mechanisms.

Test Plan

We will add additional integration and unit tests to check if client acts as expected.

...

Space shortcuts

Child pages

Versions Compared

Old Version 19

New Version 20

Key

Exception Table

AbortableTransactionException

ProducerRetriableTransactionException

ProducerRefreshMetadataRetriableTransactionException

ApplicationRecoverableTransactionException

InvalidConfiguationTransactionException

Compatibility, Deprecation, and Migration Plan

Test Plan

Space shortcuts

Child pages

Page History

Versions Compared

Old Version 19

New Version 20

Key

Exception Table

AbortableTransactionException

ProducerRetriableTransactionException

ProducerRefreshMetadataRetriableTransactionException

ApplicationRecoverableTransactionException

InvalidConfiguationTransactionException

Compatibility, Deprecation, and Migration Plan

Test Plan