...
Public Interfaces
- ResourceSpec (See Resource Requirements)
- Introduce new configuration "taskmanager.defaultSlotResourceFraction", while deprecate but stay compatible with the configuration "taskmanager.numberOfTaskSlots". (See Unknown Resource Requirements)
Proposed Changes
General Workflow
...
To solve this problem, we need to put different connected components into the same slot sharing group for streaming jobs, while keep them in different slot sharing groups to avoid having large slots with tasks not necessarily scheduled together. We need a parameter scheduleAllSourcesTogether/allVerticesInSameSlotSharingGroupByDefault indicating whether to identify all the sources as in the same pipelined region (imagine a virtual source connected to all the real sources) or not, and passed it into StreamingJobGraphGenerator differently for streaming and batch jobs.
...
- For tasks with specified resource requirements, we add up resource requirements of all the tasks in the slot sharing group, and request a slot with the sum resources.
- For tasks with unknown resource requirements, we request a slot with default resources.
Implementation Steps
Step 1. Introduce
...
option allVerticesInSameSlotSharingGroupByDefault in ExecutionConfig
- Introduce option
allSourcesInSamePipelinedRegion
in allVerticesInSameSlotSharingGroupByDefault inExecutionConfig
- Set it to
true
by default - Set it to
false
for SQL/Table API bounded batch jobs by the Blink planner
...
- Identify pipelined regions, with respect to
allSourcesInSamePipelinedRegion
to allVerticesInSameSlotSharingGroupByDefault - Set slot sharing groups according to pipelined regions
- By default, each pipelined region should go into a separate slot sharing group
- If the user sets operators in multiple pipelined regions into same slot sharing group, it should be respected
...