This document describes the strategy behind MXNet CIs fleet auto scaling for slaves.
MXNet CI will make use of multiple slaves, distinguishable by their labels.
Instance-Type: C5.18xlarge (72 vCPUs, 0 GPUs)
Executors: 4 (maybe variable, see “Further plans: Heavy job plugin”)
This instance is capable of building all Unix-based artifacts and execute all CPU-based tests on Ubuntu. The same instance will be used for both tasks because neither guarantees a permanent CPU-utilization but rather generate short spikes. Thus, available resources can be shared with as many tasks as possible and efficiency kept on a high level without sacrificing execution time.
Instance-Type: G3.8xlarge (32 vCPUs, 2 GPUs)
Executors: 1 (see “Further plans: Assign GPUs to jobs”)
This instance is only executing GPU-based tests on Ubuntu. At this stage, only one GPU will be used because no parallelization is possible with nvidia-docker (see “Further plans: Assign GPUs to jobs”). 8xlarge is used due to the amount of vCPUs because at this stage, a GPU-test spends most of the time validating results using NumPy on a CPU (see “Further plans: Cache validation results”).
Instance-Type: C5.18xlarge (72 vCPUs, 0 GPUs)
Executors: 4 (maybe variable, see “Further plans: Heavy job plugin”)
This instance is capable of building all Windows-based artifacts and execute all CPU-based tests on Windows.
Instance-Type: G3.8xlarge (32 vCPUs, 2 GPUs)
Executors: 1 (see “Further plans: Assign GPUs to jobs”)
This instance is only executing GPU-based tests on Windows. At this stage, only one GPU will be used because the current CI setup does not support assigning GPUs to jobs (see “Further plans: Assign GPUs to jobs”).
Instance-Type: P3.2xlarge (8 vCPUs, 1 GPUs)
Executors: 1 (see “Further plans: Assign GPUs to jobs”)
This instance is used to validate features which are only available on the latest GPU generations on Ubuntu.
6. mxnetlinux-gpu-p3-8xlarge
Instance-Type: P3.8xlarge (32 vCPUs, 4 GPUs)
Executors: 1 (see “Further plans: Assign GPUs to jobs”)
This instance is used to validate features which are only available on the latest GPU generations on Ubuntu.
Instance-Type: C5.large (2 vCPUs, 0 GPUs)
Executors: 30
The job of this slave is to execute Jenkins operations that need a node context. This could be loading a groovy file, publishing a GitHub status, checking out the repository and similar things.
This slave is NOT intended to run any workload. If you have to run a proper task, as small as it might seem, it should be run on mxnetlinux-cpu. The intention of this slave is to act as a high-priority queue for lifecycle operations that are required to execute a job properly. That's why this slave has so few CPU cores but so many executors - the intention is not that this slave does any actual work.
1. mxnetwindows-gpu-p3
Instance-Type: P3.2xlarge (8 vCPUs, 1 GPUs)
Executors: 1 (see “Further plans: Assign GPUs to jobs”)
This instance will be used to validate features which are only available on the latest GPU generations on Windows.
2. mxnetwindows-gpu-p3-8xlarge
Instance-Type: P3.8xlarge (32 vCPUs, 4 GPUs)
Executors: 1 (see “Further plans: Assign GPUs to jobs”)
This instance will be used to validate features which are only available on the latest GPU generations on Windows.
System metrics (CPU, GPU, RAM, HDD/IO, network) will not be used as base for scaling decisions. This is due to the fact that the utilization, especially while running multiple jobs, varies extensively and thus does not represent a reliable data source. Instead, more context-sensitive information should be used. These could be the following Jenkins-informations:
In order to generate these metrics, the following two approaches can be used:
The only difference in this case is the location where the script is getting executed. In order to ensure separation of concerns, using AWS Lambda is preferred as it ensures proper execution and debugging possibilities without requiring access to the master. In the end, the Jenkins REST API is used in both cases.
While some metrics may vary heavily during the lifetime of a slave, ongoing high values could indicate bad health of a slave or the GitHub repository. In order to catch these cases, the following Alarms could be used:
Before a slave is being added, the current queue should be evaluated. Due to the fact that Jenkins may require some time to schedule tasks, only jobs which have been in the queue for at least 2 minutes should be considered.
Due to all instances being in the same VPC, anonymous authentication is used for JNLP. This allows to start a slave and let it connect automatically without the necessity to manage credentials. In order to improve debugging, an instance should be able to rename itself in the EC2 console by using the AWS-CLI command modify-instance-attribute http://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html. This step has to be security evaluated. An alternative could be to retrieve the slave’s IP on the master and thus create a mapping. This mapping could be evaluated by Lambda in order to rename the instances.
In order to improve startup times and prevent a cold-start during execution of Jenkins-jobs, an AMI should be created and the usage of a shared Docker cache (See “Further plans: Shared Docker cache) considered.
By default, Jenkins Master declines connection attempts to already occupied slots. An instance could try to connect to every single existing slot and wait until the connection is accepted. It has to be evaluated whether this is a good attempt and how race conditions are handled. An alternative solution could be to access the Jenkins API and retrieve a list of free slots. In order to reduce the number of slaves trying to connect to the same slot, the chosen slot could be determined by randomizing the selection – the instance should just try a different slave slot if the randomly chosen one has been occupied in the meantime.
Before a downscale operation can be executed, a few actions have to be taken before an instance is shut down. Otherwise, a running build could be interrupted and thus marked as failed.
By default, Jenkins distributes jobs evenly across all connected slaves. This results in all slaves only being used partially – shutting an instance down would thus result in an interrupted build. Instead, a way to make Jenkins occupy all executors of a single slave has to be found. In case this is not possible, the Naginator-Plugin https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin can be used to re-schedule a failed build. By evaluating how long a job has been running yet, the slave with the smallest sum of durations can be turned off.
Before a slave is being turned off, it should be marked as offline. This avoids race conditions caused by the Jenkins scheduler assigning a job before the slave is disconnected. After waiting a few seconds, the auto scaling service should validate that no jobs are being executed. Afterwards, the instance can be terminated.
To avoid further race conditions, the job queue should be checked while evaluating how many slaves should be stopped. If a job for that slave-type is queued, this should be considered.
This section describes thoughts which should be evaluated at a later point in time.
This plugin allows to assign weights to jobs. Jobs using many resources may be marked this way. This allows to execute only few heavy or many small jobs on a single machine. This allows a better resource utilization.
More information at https://wiki.jenkins.io/display/JENKINS/Heavy+Job+Plugin
In order to reduce costs, Spot instances could be used. Depending on the region, up to 90% compared to On Demand-Instances can be saved. To avoid builds getting marked as failed because the market price rose above the threshold and the instance getting shut down, the Naginator-Plugin can be used.
To improve parallelisation of tests, all GPUs should be used to execute tests. At the moment, only one GPU is being used per instance due to hardcoded GPU-indizes in the MXNet code and tests. Also, some tests like KVStore require more than one GPU and would fail otherwise.
On Ubuntu, nvidia-docker is being used as wrapper for docker in order to provide access to the GPUs inside the container. By default, nvidia-docker is not able to start multiple containers because GPUs can not be shared in between instances.
TODO: Evaluate
...
Instead of computing validation data in GPU-jobs using NumPy on CPUs, the results should be cached. This can be achieved by mocking the NumPy interface and checking if the passed command chain has been executed previously.
Some operators require the latest GPU-hardware and are being skipped if they are executed on older hardware. In order to test these cases, a new test job can be introduced. Due to the high price of P3-instances, these tests should be kept to a bare minimum.
If the master fails, it has to be restarted and may not get back up. We should evaluate whether it's possible and worth the effort to develop a redundant Jenkins Master setup.
To reduce the cold-start time of jobs on new slaves, the used Docker images should be stored in a shared cache.