THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- Users actually know how many resources are needed when run a single job, initialize all workers when cluster starts can speed up startup process
- Job schedule in topology order, next operator won't schedule until prior execution slot allocated. The TaskExecutors will start in several batchs in some cases, it might slow down the startup speed
- Flink support FLINK-12122 [Spread out tasks evenly across all available registered TaskManagers], but it will only effect if all TMs are registered. Start all TMs at begining can slove this problem
- For session clusters running short lived-jobs, cluster should have workers all the time to reduce job's E2E latency
...
A public interface is any change to the following:
| Option name | Default Value |
|---|---|
| slotmanager.number-of-slots.min | 0 |
| slotmanager.min-total-resource.cpu | no default value, can be derived from slotmanager.number-of-slots.min |
| slotmanager.min-total-resource.memory | no default value, can be derived from slotmanager.number-of-slots.min |
Proposed Changes
Minimum required resource check in DefaultResourceAllocationStrategy
...
In this case, the min required worker number derived from DefaultResourceAllocationStrategy is 3 but this will exceed the max total resource limit which is 14 slots.
To Avoid this oscillation, we need to check the worker number derived from minimum and maximum resource configuration is consistent before starting the SlotManager.
...
- What impact (if any) will there be on existing users? No
Test Plan
UT & Manually verify that the min resource allocation is working as expected.
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.