Page History

...

I will implement (Job, Host) blacklist for speculative execution feature. In order to implement FLINK-11000  feiendly in the future, my interface also suit other blacklist descripted above.The blacklist module is a thread that maintains the black machines of this job and removes expired elements periodically.Each element in blacklist contains IP and timestamp. The timestamp is used to decide whether the elements of the blacklist is expired or not. 


这里加一个整体逻辑图。

Classes and Interfaces of (JobVertex Host) blacklist

...

Pass the blacklist information to cluster ResourceManager

Yarn

We can control containers do not on some mechines by yarn AllocationTags and PlacementConstraints.

Now Flink use hadoop version is -2.4.1, but if I want to use PlacementConstraints of Yarn, I must 

<hadoop.version>2.4.1</hadoop.version>

x and request container by ResourceRequest api, it don't support PlacementConstraints.

So for define Allocation tags and PlacementConstraints, I will introduce hadoop-3.x SchedulingRequest api by java reflect mechanism.反射，yarn-3.0

When the executions are scheduled, we will add information of the blacklist to yarn PlacementConstraint. In this way, I can ensure that the yarn container is not on the machines in the blacklist.

...