Status

Current state: Released

Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html

JIRA: https://jira.apache.org/jira/browse/FLINK-14058

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

To solve this problem, we need to put different connected components into the same slot sharing group for streaming jobs, while keep them in different slot sharing groups to avoid having large slots with tasks not necessarily scheduled together. We need a parameter scheduleAllSourcesTogether/allVerticesInSameSlotSharingGroupByDefault indicating whether to identify all the sources as in the same pipelined region (imagine a virtual source connected to all the real sources) or not, and passed it into StreamingJobGraphGenerator differently for streaming and batch jobs.

...

• For tasks with specified resource requirements, we add up resource requirements of all the tasks in the slot sharing group, and request a slot with the sum resources.
• For tasks with unknown resource requirements, we request a slot with default resources.

Implementation Steps

Step 1. Introduce

...

option allVerticesInSameSlotSharingGroupByDefault in ExecutionConfig

• Introduce option allSourcesInSamePipelinedRegion in allVerticesInSameSlotSharingGroupByDefault in ExecutionConfig
• Set it to true by default
• Set it to false for SQL/Table API bounded batch jobs by the Blink planner

...

• Identify pipelined regions, with respect to allSourcesInSamePipelinedRegionto allVerticesInSameSlotSharingGroupByDefault
• Set slot sharing groups according to pipelined regions 
  • By default, each pipelined region should go into a separate slot sharing group
  • If the user sets operators in multiple pipelined regions into same slot sharing group, it should be respected

...