THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Future more for pt 2, there are also different requirements for stateless and stateful operations, such as whether key level ordering is required or not. Naturally speaking, with a requirement of key level ordering, the broker needs to allocate the same message key to the same consumer within one generation as a must. For stateless operation which doesn't care about the ordering at all, the design could be much simplified as round robin assignment.
Proposed roadmap
We would start from supporting a new offset acknowledgement semantics as this unblocks the potential to process regardless of partition level ordering. The stateful operation support is of more value in nature, so concurrently we could add supports on key based filtering with Fetch calls. The stateless and transactional supports have to be built on top of the first two steps.
| Stage name | Goal | Dependency |
|---|---|---|
| Individual acknowledgement | Create a generic offset acknowledgement model beyond current partition → offset mapping. | No |
| Key filtering based fetch on topic level | Add capability to FetchRequest with specific hashed key range or specific keys | No |
| Transactional support | Incorporate individual ack into the transaction processing model | Key based filtering |
| Support cooperative fetch on broker level | Round robin assign data to cooperative consumer fetches when there is no key-range specified | Individual ack |
High Level Design
The design will be broken down, aligning with the roadmap defined above.
Individual acknowledgement
| draw.io Diagram | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Public Interfaces
The
Compatibility, Deprecation, and Migration Plan
...