THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- it allows full speed consuming and asynchronous processing from partitions that are _not_ revoked while a (cooperative) rebalance is on-going,
- it becomes easier to use the API in a way that does not cause duplicate messages,
- it becomes easier to process data asynchronously with receiving more records,
- it changes minimally such that existing users need no changes.
...
- While the partition-revoked callback method is waiting, the poll loop is waiting for the poll to complete. As a consequence _all_ partitions can no longer make progress and total throughput plummets. This is especially relevant for cooperative rebalancing where only a few partitions are revoked at a time.
- Keeping track of which records are being processed can be non-trivial. This challenge is compounded by having to share this information with the rebalance listener.
- When an async runtime (for example: Kotlin coroutines, Pekko, ZIO, Cats Effect) is used, the single-thread consumer lock makes it very hard to call the consumer from the callback. (See KIP-944.)
- Even though processing is asynchronous, offsets must be committed in order.
- For better throughput, offsets should be aggregated before committing.
...