...
This KIP is aimed at improving the error-handling semantics in Kafka Streams when Kafka Steams fails to produce a message to the downstream sink by providing an interface that can provide custom massaging of the error (e.g. report to a custom metrics system) and indicate to Streams whether or not it should re-throw the Exception, thus causing the application to fall over.
Status
Current state: Under Discussion Adopted (1.1.0)
Discussion thread: Click here
...
We'll implement the following error handling logic to the onCompletion handler in RecordCollectorImpl and to the try
statement that wraps it:
- If the Exception that is thrown is a
ProducerFencedException
, behave as we do today and do not invoke theProductionExceptionHandler
as these exceptions are self-healing. - If the Exception that is thrown is fatal will affect all records and should cause Streams to always fail. If so, then do not invoke the
ProductionExceptionHandler
because its result will have to be ignored. We should log that we're ignoring these exceptions at DEBUG level.- The exceptions that meet this classification are: TODO
AuthenticationException
AuthorizationException
SecurityDisabledException
InvalidTopicException
UnknownServerException
IllegalStateException
OffsetMetadataTooLarge
SerializationException
TimeoutException
when it occurs immediately on send due to a full buffer
- The exceptions that meet this classification are: TODO
- If the
Exception
that is thrown meets neither of the above conditions, determine ifsendException
is already set. If so, do not invoke theProductionExceptionHandler
because this would mean that we've already invoked it and decided toFAIL
. Invoking it again would just result in an ignored result. - If none of the conditions above is met, invoke the
handle
method in theProductionExceptionHandler
and check the result.- If the result is
CONTINUE
, log a note atDEBUG
that we received that result and are not failing Streams as a result. This ensures that it's not possible for a client developer to ship code that totally swallows errors without presenting any kind of activity in the log. - If the result is
FAIL
, log a message atERROR
that we received that result and setsendException
so Streams will fail.
- If the result is
The error handler will only be invoked for exceptions that are returned via the producer callback, and will not be invoked for Exceptions thrown directly from send as all of those exceptions should be seen by Streams immediately.
These changes will facilitate a number of error handling scenarios. For example, one could choose to write an interface that always fails, but does some additional logging in the process:
...