THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Fine-grained Scaling Behavior
...
The administrator launches Node Managers
...
with zero capacity
...
(via the REST /api/cluster/flexup command). The Node Managers report zero
...
capacity to the Resource Manager upon registration.
...
When a user submits an application to YARN (for example, a MapReduce job)
...
, the following occurs:
- The application is added to the Resource Manager's scheduling pipeline.
...
- If a Node Manager has a zero profile, the YARN scheduler (for example. FairShareScheduler)
...
- does not allocate any application containers
...
- .
- If
...
- a Node Manager has a non-zero
...
- capacity (low, medium, or high profiles),
...
- containers might be allocated for those Node Managers depending on their free capacity.
- Myriad
...
- receives resource offers from Mesos for slave nodes running zero profile Node Managers.
- The offered resources are projected to YARN's scheduler as available capacity of the zero profile Node Manager. For example, if Mesos offers (10G memory, 4CPU) for a given node, then the capacity of the zero profile Node Managers running on that node increases to (10G memory, 4 CPU).
- The YARN scheduler
...
- allocates a few containers for the zero profile Node Managers.
- For each allocated container, Myriad spins up a placeholder Mesos task that holds on to Mesos resources as long as the corresponding YARN container is alive.
...
- The placeholder tasks are launched in a single shot, corresponding to
...
- the containers that YARN allocates.
...
- Node Managers become aware of container allocations via YARN's HBase mechanism.
- Myriad ensures that Node Managers are made aware of container allocations only after the corresponding placeholder Mesos tasks are launched.
- When Node Managers report to Resource Manager that some of the containers have finished, Myriad sends out finished status updates to Mesos for the corresponding placeholder tasks.
- Mesos takes back the resources from Myriad that were held using placeholder tasks upon receiving the finished status updates.
Trying out Fine-grained Scaling
...