...

• One could attempt to reach from the producer into the consumer background thread to make consumer heartbeats and transactional offset commits mutually exclusive. This would solve the problem, since member epochs are only bumped when a heartbeat is in-flight. This would be more of a workaround, and would likely require a larger change of the Java client APIs.

• We considered tracking a single RevocationEpoch  per member instead of an assignment epoch per assigned partition - which is the last epoch a partition was revoked from the member. While this would have reduced the chance of hitting the race conditions, it turned out there were still cases where you could run into spurious ILLEGAL_GENERATION errors. 

• If we use a consumer rebalance listener to always commit any open transactions in onPartitionsRevoked  and abort any open transactions in onPartitionsLost, we can actually make sure that the ILLEGAL_GENERATION can always be safely retried. The reason is that the member epoch cannot be bumped during the execution of the rebalance handler, and partitions cannot be revoked without executing the rebalance listener. So one option discussed was to simply retry the error, since we know that all partitions we are trying to commit are still assigned to us, just the member epoch was outdated. However, this would make the usage of transactions in Kafka even more complicated. Instead, we want to fix the problem on the broker side and simplify client-side usage of transactions. Furthermore, this would require significant changes in the Java producer, since it can only enter an abortable state, a fatal state or retry a request immediately. Implementing a retry after refreshing the consumer group metadata would have required a new way of interacting with the producer: We’d have to implement an “application-retriable” kind of error, that is handing back control-flow to the application without entering an error state, without retrying the commit immediately and without indicating “success” to the application.

References

• A similar mechanism was already implemented in KIP-1071: Streams Rebalance Protocol