Page History

...

Pass the blacklist information to cluster ResourceManager

Yarn

We First, nodes' attributes should include machine ip attribute, then we can control containers do not on some mechines by yarn AllocationTags and by yarn PlacementConstraints.

Now Flink use hadoop-2.x and request container by ResourceRequest api, it don't support PlacementConstraints.

So for define Allocation tags and use PlacementConstraints, I will introduce hadoop-3.x SchedulingRequest api by java reflect mechanism.

When the executions are scheduled, we I will add blacklist information of the blacklist to yarn PlacementConstraint. In this way, I can ensure that the yarn container is not on the machines in the blacklist.https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html

Kubernetes

My code only supports Yarn integration. But as far as I know, we could use nodeaffinity or podaffinity to achieve the same goal with Yarn PlacementConstraint in K8s integration. 

For Kubernetes, we could the node selector to control where to assign the pod. However, one difference is that the K8s Apiserver could not help us to persistent this information. We need to store it by Flink internally.

...

We could achieve the same goal with Yarn PlacementConstraint in K8s integration by node-affinity if nodes had attached ip label.

The same as yarn, when the executions are scheduled, I will add blacklist information to k8s PodSpec.

Mesos

According to 

...