THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Code Block | ||
---|---|---|
| ||
Algorithm incremental-rebalancing Input Set of Tasks, Set of Instances, Set of Workers, Where each worker contains: Set of active Tasks, Set of standby Tasks, owned by which instance Main Function Assign active tasks: (if any) To instances with learner tasks that indicate "ready" To previous owners To unready learner tasks owners To instances with standby tasks To resource available instances Keep existing learner tasks' assignment unchanged Pick new learner tasks out of heaviest loaded instances Assign learner tasks: (if any) To new-coming instances with abundant resource To instances with corresponding standby tasks Assign standby tasks: (if any) To instances without matching active tasks To previous active task owners after learner transfer in this round To resource available instances Based on num.standby.task config, this could take multiple rounds Output Finalized Task Assignment |
Backing up Information on Leader
Since the incremental rebalancing requires certain historical information of last round assignment, the leader worker will need to maintain the knowledge of:
- Who participated in the last round of rebalance. This is required information to track new comers.
- Who will be leaving the consumer group. This is for scaling down support as the replay could take longer time than the scaling down timeout. Under static membership, since we don't send leave group information, we could leverage leader to explicitly trigger rebalance when the scale-down timeout reaches. Maintaining set of leaving members are critical in making the right task shuffle judgement.
These are essential group state knowledges leader wants to memorize. To avoid the severity of leader crash during scaling, we are avoiding backing up too much information on leader for now.
For the smooth delivery of all the features we have discussed so far, an iteration plan of reaching final algorithm is defined as below:
...