THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
The objective of fine-grained scaling (FGS) is to bring elasticity of resources between Myriad/YARN and YARN and other Mesos frameworks. The concept is to allow YARN to take With fine-grained scaling, YARN takes resource offers from Mesos , run and runs enough containers (YARN tasks) that the offers can hold and release the resources back to Mesos once the containers finish.
- Node Managers that register with RM the Resource Manager with (0 memory, 0 cpuCPU) are eligible for fine-grained scaling, that is, Myriad expands /and shrinks the capacity of such NMs the Node Managers with the resources offered by Mesos. Further, Myriad ensures that YARN containers will be are launched on such NMs the Node Managers only if Mesos offers enough resources on the slave nodes running those NMsNode Managers.
- A new ** zero ** profile is defined in profile, as well as small, medium, and large profiles are defined in the Myriad configuration file,
myriad-config-default.ymlfile to help admins launch NMs with (0mem,0cpu) capacities using the xxx.. A zero profile allows administrators to launch Node Managers with (0 memory, 0 CPU) capacities. - Node Managers that register with RM the Resource Manager with more than than (0 memmemory, 0 cpuCPU) are not eligible for FGSfine-grained scaling. For example, Myriad will does not expand /and shrink the capacity of such NMs. These NMs are typically launched with low/medium/high profile.the Node Managers.
Fine-grained Scaling Behavior
...
The administrator launches Node Managers
...
with zero capacity (via the REST /api/cluster/flexup command) and the Node Managers report
...
zero
...
capacity to the Resource Manager upon registration.
...
When a user submits an application to YARN (for
...
example, a MapReduce job)
...
, the following happens:
- The application is added to
...
- the Resource Manager's scheduling pipeline.
...
- If a Node Manager has a zero profile, the YARN scheduler (for
...
- example. FairShareScheduler)
...
- does not allocate any application containers
...
- .
- If
...
- a Node Manager has a non-zero
...
- profile (low
...
- , medium
...
- , or high profiles),
...
- containers might be allocated for
...
- those Node Managers depending on their free capacity.
- Myriad
...
- receives resource offers from Mesos for slave nodes running
...
- zero
...
- profile
...
- Node Managers.
- The offered resources are projected to the YARN
...
- Scheduler as
...
- available capacity
...
- of the
...
- zero
...
- profile Node Manager. For example, if Mesos offers (10G memory, 4CPU) for a given node, then the capacity of the
...
- zero profile Node Manager running on that node increases to (10G memory,
...
- 4 CPU).
- The YARN
...
- Scheduler allocates a few containers for the
...
- zero
...
- profile Node
...
- Manager.
- For each allocated container, Myriad spins up a
...
- placeholder
...
- Mesos task that holds on to Mesos resources as long as the corresponding YARN container is alive.
...
- The placeholder tasks are launched in a single shot, corresponding to
...
- the containers that YARN allocates.
...
...
- Node Managers become aware of container allocations via YARN's
...
- heart-beat mechanism.
- Myriad ensures
...
- that Node Managers are made aware of container allocations only after the corresponding
...
- placeholder
...
- Mesos tasks are launched.
- When
...
- containers finish, Myriad Executor sends out
...
- finished
...
- status updates to Mesos for the corresponding
...
- placeholder
...
- tasks.
- Mesos takes back the resources from Myriad
...
- after receiving a finished status update.
Trying out Fine-grained Scaling
- Spin up Resource Manager with
...
- Myriad Scheduler
...
- plugged into it.
Flexup a
...
few Node Managers using Myriad Cluster API with zero profile:
Code Block { "instances":3, "profile": "zero" }
...
Info The zero profile Node Managers advertise
...
zero
...
resources to Resource Manager (
...
the Resource Manager Nodes UI shows this).
- Submit a
...
- MapReduce job to the Resource Manager.
- The zero profile Node Managers advertise zero resources to Resource Manager (the Resource Manager Nodes UI shows this).
- When Mesos offers resources to Myriad, the Mesos UI
should show "placeholder" mesos - shows placeholder Mesos tasks (prefixed with "yarn_") for each yarn container allocated using those offers.
- The
RM- Resource Manager's UI
should show - shows these containers allocated to the
**zero** profile NM - zero profile Node Manager nodes.
- The placeholder
mesos - Mesos tasks
should - typically finish
as and - when the YARN containers finish.
- The job should finish successfully (although
some NMs - some Node Managers were originally launched with zero (0) capacities).
Sample Mesos Tasks Screenshot
