DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
Status
Current state: Under discussion
...
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Kafka Producer supports following transactional APIs
...
We have attempted error handing in accepted KIP: KIP-691: Enhance Transactional Producer Exception Handling - Apache Kafka - Apache Software Foundation which we can perhaps leverage some of the work here.
Public Interfaces
We would new error types which will be extended by existing exceptions mentioned in below table.
| Code Block |
|---|
public class AbortableTransactionException extends ApiException {
public AbortableTransactionException(String message) {
super(message);
}
}
public class ProducerRetriableTransactionException extends ApiException {
public ProducerRecoverableTransactionException(String message) {
super(message);
}
}
public class ProducerImmediateRetriableTransactionException extends ApiException {
public ProducerRecoverableTransactionException(String message) {
super(message);
}
}
public class ApplicationRecoverableTransactionException extends ApiException {
public AbortableTransactionException(String message) {
super(message);
}
}
public class InvalidConfiguationTransactionException extends ApiException {
public ProducerRecoverableTransactionException(String message) {
super(message);
}
}
// Extending exception types example
public class InvalidProducerEpochException extends AbortableTransactionException {
private static final long serialVersionUID = 1L;
public InvalidProducerEpochException(String message) {
super(message);
}
} |
Proposed Changes
We are proposing to group exceptions into four types:
...
Exception/Error Names | Thrown scenarios during transaction | Current handling | Expected Handling | ||
|---|---|---|---|---|---|
Producer API | Transaction API | Producer API | Transaction API | ||
ErrorRetry | Producer Retriable | Producer Retriable | Producer Retriable | Producer Retriable | |
ErrorRefreshMetadataAndRetry | Refresh + Retriable | Refresh + Retriable | Refresh + Retriable | Refresh + Retriable | |
ErrorFenceClient | Application Recoverable | Application Recoverable | Application Recoverable | Application Recoverable | |
ErrorBadConfig | Invalid Configuration | Invalid Configuration | Invalid Configuration | Invalid Configuration | |
TransactionAbortableException | Producer Recoverable | Producer Recoverable | Producer Recoverable | Producer Recoverable | |
IllegalStateException | Abortable | Sometimes Fatal depending on whether application or Sender caused issue. See: kafka: KAFKA-14831: Illegal state errors should be fatal in transactional producerCLOSED | Application Recoverable (probably not expected) | Application Recoverable | |
Au thenticationException | Abortable | Fatal | Invalid Configuration | Invalid Configuration (not expected) | |
InvalidPidMappingException |
| Sometimes retriable (no longer returned) or abortable | Abortable | Application Recoverable | Application Recoverable |
UnknownProducerIdException | Sometimes retriable (no longer returned) or abortable | Abortable | Application Recoverable | Application Recoverable | |
ClusterAuthorizationException TransactionalIdAuthorizationException UnsupportedVersionException UnsupportedForMessageFormatException | Fatal, except UnsupportedForMessageFormatException which is Abortable | Cluster/Transaction Auth → abortable on InitProducerId, other errors abortable TransactionAuth → fatal, others abortable for AddPartitions, Find Coordinator, EndTxn, AddOffsets Transaction Auth, UnssupportedForMessageFormat → fatal for OffsetCommit | Invalid Configuration | Invalid Configuration | |
CorruptRecordException NotEnoughReplicasAfterAppendException NotEnoughReplicasException TimeoutException | Retriable if the error is retriable. Otherwise abortable | Retriable if the error is retriable otherwise abortable | Producer Retriable | Producer Retriable | |
UnknownTopicOrPartitionException NotLeaderOrFollowerException | Retriable if the error is retriable. Otherwise abortable | Retriable if the error is retriable otherwise abortable | Refresh + Retriable | Refresh + Retriable | |
InvalidRecordException InvalidRequiredAcksException RecordBatchTooLargeException InvalidTopicException | Retriable if the error is retriable. Otherwise abortable | Retriable if the error is retriable otherwise abortable | Invalid Configuration | Invalid Configuration | |
TopicAuthorizationException GroupAuthorizationException | Abortable | Abortable | Invalid Configuration | Invalid Configuration | |
FencedInstanceIdException CommitFailedException UnknownMemberIdException IllegalGenerationExceiption | N/A | Abortable (TxnOffsetCommit Only) | N/A | Application Recoverable | |
InvalidProducerEpochException |
| Abortable | Fatal | Application Recoverable | Application Recoverable |
ProducerFencedException | Fatal | Fatal | Application Recoverable | Application Recoverable | |
OutOfOrderSequenceException | Sometimes retriable or abortable | N/A | Producer Retriable | N/A | |
InvalidTxnStateException |
| Abortable | Fatal | Producer Recoverable (KIP-890 relies on this – note this is the only one that differs with Produce API) | Application Recoverable |
KafkaException | Abortable (default seems to be abortable) | Fatal in most cases, but abortable when there are partition errors | Application Recoverable (not expected) | Application Recoverable (not expected) | |
RuntimeException | Abortable (default seems to be abortable) | Fatal – only thrown as this generic type when correlation ID is wrong. This should be updated as KIP-691 suggests | Application Recoverable (not expected) | Application Recoverable (not expected) | |
ConcurrentTransactionsException | Retriable | Retriable | Producer Retriable (KIP-890 may see this) | Producer Retriable | |
CoordinatorLoadInProgressException | Retriable | Retriable | Producer Retriable | Producer Retriable | |
NotCoordinatorException CoordinatorNotAvailableException |
| Retriable | Retriable | Refresh + Retriable | Refresh + Retriable |
CorrelationIdMismatchException | Application Recoverable | Application Recoverable | |||
Compatibility, Deprecation, and Migration Plan
This KIP involves client side changes which only affects the resiliency of new Producer client and Streams. Old clients will continue to be able to use their error handling. Based on type of exception thrown, user needs to change their exception catching logic to take actions against their exception handling.
Test Plan
We will add additional integration and unit tests to check if client acts as expected.
Rejected Alternatives
We discussed another approach where we encode the handling into the response as a separate field from the error code or something similar. Produce responses have space for a code and for a message, so something similar could be done to distinguish the cause/meaning of the error from the action that should be taken. The encoding of the error could be included in the definition of the response spec for other client compatibility. We rejected this approach as it requires a larger overhaul on the APIs as well as the clients.
...