THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
We want to clarify beforehand that this KIP would be a starting point of a transformational change on the Kafka client consumption semantics. It's not possible to have all the design details rolling out in one shot. Instead, instead the hope is to breakdown
Use case scenario
focus is to define a clear roadmap of what things have to be done, and better illustrate the dependency plan while getting some concrete tasks in the step one.
Use case scenario
As stated above, the scaling cap for consumer based application is the number of input partitions. In an extreme scenario when there is single input partition with two consumers, one must be idle. If the single box consumer could not keep up the speed of processing, there is no solution to add more computing capacity. It would be ideal we could co-process data within one partition with two consumers when the partition level order is not required, and we could add consumer instances as many as we want.
So this cooperative consumption model applies with following limitations:
- The bottleneck is on the application processing, not data consumption which could be caused by other roots such as network saturation of broker.
- The processing semantic does not require partition level order, otherwise only one consumer could work on the input sequentially without parallelism.
Future more for pt 2, there are also different requirements for stateless and stateful operations, such as whether key level ordering is required or not. Naturally speaking, with a requirement of key level ordering, the broker needs to allocate the same message key to the same consumer within one generation as a must. For stateless operation which doesn't care about the ordering at all, the design could be much simplified as round robin assignment.
Proposed roadmap
| Stage name | Goal | Dependency |
|---|---|---|
| Individual acknowledgement | Create a generic offset acknowledgement model beyond current partition → offset mapping. | No |
| Support cooperative fetch on broker level | Broker will maintain a state for to track which consumer | Individual ack |
| Key filtering based fetch on topic level | Add capability to FetchRequest with specific hashed key range or specific keys | No |
| Transactional support | Incorporate individual ack into the transaction processing model | |
Public Interfaces
The
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Rejected Alternatives
...
There are a couple of alternatives to this proposal here.
- KIP-253 proposed physical partition expansion, which is a fairly complex implementation and could be hard to reason about correctness.
- Some discussion around making Kafka Streams as a multi-threading model where consumers are completely decoupled from processing thread. This means we have to tackle the concurrent processing challenge and there could be more inherent work to redesign state store semantics too.