...

Currently, producer-side recoverable errors (the KIP's target category) prevent a record from being added to a batch. They additionally make a transition to an `error` state, which ends up causes the transaction to fail. This KIP provides the possibility to commit a transaction successfully in presence of such errors. Obviously, the problematic records are not added to the batch, but the transition to the `error` state is not done. In other words, with the new `send` API, the transaction does not fail because of a single poison pill record that is not even present in the batch. Obviously, the transaction can still fail due to other types of errors (for example, broker-side errors).

Public Interfaces

If the user 1) is performing a transaction and 2) passes the `TxnSendOption` with the value `IGNORE_SEND_ERRORS` to the `send` method, any poison pill record is excluded from the batch, and the transaction is committed successfully. Note that if the user sets the `TxnSendOption` to `IGNORE_SEND_ERRORS` outside of a transaction, the overloaded `send` method throws an `IllegalStateException`.

...

Unit tests for `KafkaProducer` to show that the new feature works with different `send()` errors and exceptions such as RecordTooLargeException.

Rejected Alternatives

• Add a producer custom exception handler interface: see KIP-1038 and the discussions.
• Identify poison pill records application-side:  It is not efficient and sometimes not even doable. For example for identifying too-large-records the application must be aware of producer configs as well as serialization method, which is not feasible sometimes. More over, checking every single record's size before sending it to catch the bad record is an overhead considering that this check is done by Producer as well.
• Add the feature of clearing errors to `flush()`: `flush()` is not necessarily a transactional method + `AddPartition` is not done successfully in the next `send`.
• Add the feature of clearing errors to `commitTxn()`: `flush()` is not necessarily a transactional method + `AddPartition` is not done successfully in the next `send`.
• `send()` throws ApiException: may break backward compatibility.
• No need of explicit `flush()` before calling `commitTransaction(commitOptions)` :  not safe + `AddPartition` is not done successfully in the next `send`
• The user must use ALOS instead of EOS in case they want to drop the poison pill records: ALOS can not guarantee not having duplicates.