Status

...

...

JIRA: here (<- link to https://issues.apache.org/jira/browse/FLINK-XXXX)

Released: <Flink Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

FineFine-Grained Resource Management is one of Apache Flink’s roadmap features that the community has been trying to deliver. While FLIP-56 delivers the ability to allocate slots with respect to fine-grained resource requirements, the question of how to get those resource requirements remains unanswered. In this FLIP, we will discuss how the runtime interfaces for fine-grained resource requirements should look like, with respect to usability, flexibility and how resources are used in runtime.

...

public class StreamGraphGenerator {
    /**
     * Specify fine-grained resource requirements for slot sharing groups. Not that it is just a
     * hint and will not restrict the scheduling strategy of the scheduler, i.e. the scheduler may
     * not guarantee that the operators within the same slot sharing group will be scheduled to the
     * same slot
     *
     * <p>Note that a slot sharing group hints the scheduler that the grouped operators CAN be
     * deployed into a shared slot. There's no guarantee that the scheduler always deploy the
     * grouped operator together. In cases grouped operators are deployed into separate slots,
     * the slot resources will be derived from the specified group requirement.
     */
    public StreamGraphGenerator setSlotSharingGroupResource(Map<String, ResourceProfile> slotSharingGroupResources);
}

...

We might provide operator-level resource configuration interface interfaces for expert users who know the exact resource usage of each operator and also want to control the scheduling strategy in a fine-grained approachfiner grain.

Default operator resource configuration

In this FLIP, the SSGs with unknown resource requirements will be scheduled with the resource profile of the default slot. We might allow user to configure the default operator resource and schedule those SSGs according to the number of operators inside it.

Known limitations and constraints to be documented

When the fine-grained resource management feature is finally exposed provided to the user. We should document its limitation clearly.user, the following limitations and constraints should be well documented, with potential impacts and suggestions.

• Setting chain-able operators to different SSGs may break operator chaining, thus change the performance.
• Change of data exchange mode (pipelined vs. blocking) with in a SSG may affect the resource requirement of the group.
• Parallelism differences between operators in the same SSG may reduce the resource utilization
• The SSG-based resource requirement might affect the internal optimization logic of Flink, such as chaining.
• Explain the difference between different execution modes and provide some configure suggestions for user.
• The operators of the same SSGs should with the same parallelism, otherwise there might be resource waste.