Page History

...

Version 1.0

Delivery goal: Stateful Task State declaration, conservative rebalance, new instance only

...

So we only gonna assign learner task only to "new comers", which means every stream worker will denote itself as "new member" when they don't have local information. The assignment, however, will be on the task .

Version 2.0

Delivery goal: Scale down feature

Specifically

1. Add tag for leaving members. 
2. Create new tooling for marking instances as ready to scale down
   
3. Add scale down timeout implementation

Version 3.0

Delivery goal: eager rebalance analysis

We want to have a detailed analysis and benchmark support before fully devoting into this feature, because in user perspective most applications should be able to tolerate minor discrepancy of task replaying. 

Version 4.0

Delivery goal: balancing factor

We will take ending task load balance into consideration. Specifically for each instance, the total number of tasks it owns should be within the range of 0.5 ~ 2 times of the expected number of tasks, which buffers some capacity in order to avoid imbalanc

We could even provide a stream.balancing.factor for the user to configure. The smaller this number sets to, the more strict the assignment will behave.  If the factor is set to r, the number of tasks a host could own is (w/total tasks)

...

Stream Instance Replacement

Sometimes we want to upgrade the current group The right approach for a global application instance replacement is

...