...
At MailChimp, we've run into occasional situations where a message that comes into streams just under the size limit on the inbound size (say for the sake of illustration, 950KB with a 1MB max.request.size
on the Producer) and we change it to a different serialization format for producing to the destination topic. In these cases, it's possible that the serialization format we change to comes in as larger than the inbound message. (For example, if we were going from a binary format to JSON we might get something much larger than if the inbound format was a binary format of some kindon the outbound side.)
These cases are rare, but when they occur they cause our entire application to fall over and someone gets woken up in the middle of the night to figure out how to deal with it. Further, solutions that address this issue by hacking around it (increasing the max.request.size or trying to manually commit to the offsets topic to skip the large messages) each have their own problems. It would be preferable for us to be able to optionally provide code to ignore an ApiException
returned from the producer. Such an interface would permit us to provide code that will log an error and instruct Streams to not re-throw the error.
...
The default behavior here will be consistent with existing behavior. Changing that behavior will be opt-in by providing the new config setting and an implementation of the interface. Constructors of RecordCollectorImpl
, StreamThread
, and StreamTask
will need to change, but as those aren't (to my knowledge) part of the public interface, so that should be fine. We could even provide overloaded constructors with the old signatures if we're concerned about binary compatibility of this change.
...