THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
As we have already discussed around the “learner” logic, when performing the scale down of stream group, it is also favorable to initiate learner tasks before actually shutting down the instances. Although standby tasks could help in this case, it requires user to pre-set num.standby.tasks which may not be available when administrator performs scaling down. The plan is to use command line tool to tell certain stream members that a shutdown is on the way to be executed. These informed members will send join group request to indicate that they are “leaving soon”. During rebalance assignment, leader will perform the learner assignment among members without intention of who are not leaving. And the leaving member will shut down itself once received the instruction to revoke all its active tasks.
...
- Percentage scaling. Compute targeting scaled down members while end user just needs to provide a %. For example, if the current cluster size is 40 and we choose to scale down to 80%, then the script will attempt to inform 8 of 40 hosts to “prepare leaving” the group.
- Direct Name-based scaling. Name the stream instances that we want to shut down soon. Built This is built for online hot swapping and host replacement.
| Code Block | ||||
|---|---|---|---|---|
| ||||
Group stable state: S1[T1, T2], S2[T3, T4], S3[T5] Scaling down the application, S2 will be leaving. #First Rebalance Member S2 joins the group and claim that it is leaving S1 performs task assignments: S1(assigned: [T1, T2], revoked: [], learning: [T3]) S2(assigned: [T3, T4], revoked: [], learning: []) S3(assigned: [T5], revoked: [], learning: [T4]) #Second Rebalance S3 finishes replay first and trigger another rebalance Member S1~S3 join with following status:(S1 is not ready yet) S1(assigned: [T1, T2], revoked: [], learning: [T3]) S2(assigned: [T3], revoked: [T3, T4], learning: []) S3(assigned: [T5], revoked: [], learning: [T4]) S1 performs task assignments: S1(assigned: [T1, T2], revoked: [], learning: [T3]) S2(assigned: [T3], revoked: [T4], learning: []) S3(assigned: [T4, T5], revoked: [], learning: []) #Third Rebalance S1 finishes replay and trigger rebalance. Member S1~S3 join with following status: S1(assigned: [T1, T2], revoked: [], learning: [T3]) S2(assigned: [], revoked: [T3], learning: []) S3(assigned: [T4, T5], revoked: [], learning: []) S1 performs task assignments: S1(assigned: [T1, T2, T3], revoked: [], learning: []) S2(assigned: [], revoked: [T3], learning: []) S3(assigned: [T4, T5], revoked: [], learning: []) S2 will shutdown itself upon new assignment since there is no assigned task left. |
...