Page History

...

The above examples are focusing more on demonstrating expected behaviors with KStream incremental rebalancing .  We "end picture". However, we also want to define the new learner algorithm for a holistic view.

We have set of workers, where each worker contains:

Workers = []

Worker {

  Set<ActiveTask>;

  Set<StandbyTask>; 

  instanceId;

}

We shall assign tasks in the order of: active, learner and standby. The assignment will be broken down into following roundssteps:

Assign active tasks:

1. Assign all active tasks to learner tasks that indicates "ready"
2. Assign the rest of active tasks to previous owners
3. Assign the rest of active tasks towards unready learner tasks owners
4. Assign the rest of active tasks to resource available hosts

...

1. Keep assigned learner tasks until finished. We don't want to see half way bounce.
2. If the load is not balanced between hosts, assign learner tasks from hosts with heavy loads to hosts with lightweight tasks.
3. As long as the group members/ number of tasks are not changing, there should be a defined balanced stage instead of forever rebalancing.
4. Instances with standby tasks have higher priority to be chosen as learner task assignor. The standby task will convert to learner task immediately.

Assign standby tasks:The assignment of standby tasks will

1. Will mostly remain the same as current, which we will just pick resource abundant hosts to allocate standby tasks. 
2. For tasks that get transferred after learner tasks finish, we could assign standby tasks right to the degraded hosts which hold previous round of active tasks.
   

Version 1.

...

0

We will care more about smooth transition over resource balance for stage one. This is because we do have some historical discussion on marking weight for different types of tasks. If we go ahead to aim for task balance too early, we are potentially in the position of over-optimization.

So we only gonna assign learner task to "new comers", which means every stream thread will denote itself as "new member".

Version 2.0

We will take task balance into consideration. Specifically for each instance, the total number of

...

tasks it owns should be within the range of 0.5 ~ 2 times of the expected number of tasks, which buffers some capacity in order to avoid

...

imbalanc

We could even provide a stream.balancing.factor for the user to configure. The smaller this number sets to, the more strict the assignment will behave.  If the factor is set to r, the number of tasks a host could own is (w/total tasks)

...