THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
For long term plans, the DeclarativeSlotManager will be completely removed in the next release after the default value is changed.
Proposed Changes
Overview of FineGrainedSlotManager and DeclarativeSlotManager
Functionality supports
| functionality | FineGrainedSlotManager | DeclarativeSlotManager |
|---|---|---|
| Allocate new task managers when resource not enough | YES | YES |
| Release idle task managers if there are no tasks/resultPartitions | YES | YES |
| Keep some redundant task managers | NO | YES |
| Max limitations of slots number | YES | YES |
| Filter out blocked resources | YES | YES |
| Track requirements of multiple jobs | YES | YES |
| Fulfill requirements by evenly strategy | NO | YES |
| Reclaim inactive slots when job finished | NO | YES |
| Different slot resources in the same task manager | YES | NO |
sub-components
DeclarativeSlotManager
There are tree subcomponents in DeclarativeSlotManager
- ResourceTracker: Track all resource requirements of multiple jobs
- SlotTracker: Track slot status(free/allocated). It's used for fulfill resource requirements with free slots.
- TaskExecutorManager: Manage the lifecycle of task managers(allocated/release/keep redundant).
The main process flow of DeclarativeSlotManager is:
- When new/updated resource requirements arrived, DeclarativeSlotManager will send it to ResourceTracker.
- When dealing with all resource requirements tracked by ResourceTracker. DeclarativeSlotManager get all free slots from SlotTracker to fulfill requirements.
- If there are also some unfulfilled requirements, DeclarativeSlotManager get all pending task manager slots from TaskExecutorManager to fulfill it.
- And the DeclarativeSlotManager will try to allocate new task managers when pending task manager slot is not enough.
FineGrainedSlotManager
There are four subcomponents in FineGrainedSlotManager
- ResourceTracker: The same as DeclarativeSlotManager#ResourceTracker.
- TaskManagerTracker: Track task managers and their resources.
- SlotStatusSyncer: Sync the slot status with task manager, and communicate with task manager to allocate/free slots.
- ResourceAllocationStrategy: Try fulfill the resource requirements with available/pending resources.
The main process flow of FineGrainedSlotManager is:
- When new/updated resource requirements arrived, FineGrainedSlotManager will send it to ResourceTracker.
- When dealing with all resource requirements tracked by ResourceTracker. FineGrainedSlotManager invoke ResourceAllocationStrategy with TaskManagerTracker.
- The ResourceAllocationStrategy will try to fulfill the resource requirements by available/pending resources orderly. It will create new pending resources if not enough.
- Then FineGrainedSlotManager will allocate new task managers according to the result of ResourceAllocationStrategy.
Add the missing capability of FineGrainedSlotManager
...
The DeclarativeSlotManager and related configs will be completely removed in the next release after the default value is changed.
Test Plan
Our current CI pipeline includes a stage dedicated to fine-grained resource management. To avoid adding too much overhead, this stage only runs tests from the 'flink-runtime' and 'flink-test' modules.
Before we deprecated the declarative slot manager. The FineGrainedSlotManager should be the default SlotManager, and it should pass all the tests in ci.
...