Page History

...

Sometimes end user wants to reach a sweet spot between ongoing task transfer and streaming resource free-up. So we want to take a similar approach as KIP-415, where we shall introduce a client config to make sure the scale down is time-bounded. If the time takes to migrate tasks outperforms this config, the leaving member will shut down itself immediately instead of waiting for the final confirmation. And we could simply transfer learner tasks to active because they are now the best shot to own new tasks.

Task Tagging Info

...

Note that to make sure the above resource shuffling could happen as expected, we need to have at least three new the following task status indicator indicators to be provided:

...

Tag Name	Task Type	Explanation	prerequisites
isLearner	standby	Indicate whether standby task is

...

a learner task.	isStateful = True
beingLearned	active	Indicate whether active task is being learned by some other stream worker.	isStateful = True
isReady	standby	Indicate whether standby task is ready to serve as active task.	isLearner = True and isStateful = True
isStateful	both	Indicate whether given task has a state to restore.	N/A

...

Algorithm Walkthrough

The above examples are focusing more on demonstrating expected behaviors with KStream incremental rebalancing.  We also want to define the new learner algorithm for a holistic view.

...