Versions Compared

Old Version 8

changes.mady.by.user Ruth Harris

Saved on

compared with

New Version 9

changes.mady.by.user Ruth Harris

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • The Mesos Slave processes advertises all of a node’s resources (8 CPUs, 16 GB RAM) to the Mesos Master.
  • The YARN Node Manager is started as a Mesos Task. This task is allotted (4 CPUs and 8 GB RAM) and the Node Manager is configured to only advertise 3 CPUs and 7 GB RAM.

  • The Node Manager is also configured to mount the YARN containers under the a cgroup hierarchy  which stems from a Mesos task. For example:

    Code Block
    /sys/fs/cgroup/cpu/mesos/node-manager-task-id/container-1

The following diagram shows a node running YARN Node Manager as a Mesos Slave task.

 

 

 

High Level Design

One way to avoid static partitioning and to enable resource sharing when running two resource managers, is to let one resource manager be in absolute control of the datacenter’s data center’s resources. The other resource manager then manages a subset of resources, allocated to it through the primary resource manager.

The following diagram shows a scenario where Mesos is used as the resource manager for the datacenter a data center which allows both Mesos and YARN to schedule tasks on any node.

 

The following diagram gives an overview of how YARN can run along side Mesos:

Image Removed

Each node in the cluster has both daemons, Mesos Slave and YARN Node Manager, installed. By default, the Mesos slave daemon is started on each node and advertises all available resources to the Mesos Master.

 

The following diagram shows how Myriad launches NodeManager a YARN Node Manager as a task under Mesos Slave:

  1. Myriad makes a decision to launch a new NodeManagerNode Manager.
    1. It Myriad passes the required configuration and task launch information to the Mesos Master which forwards that to the Mesos Slave(s).
    2. Mesos Slave launches Myriad Executor which manages the lifecycle life cycle of the NodeManagerNode Manager.
    3. Myriad Executor upon launch, configures Node Manager (for example, specifying CPU and memory to advertise, cgroups hierarchy, and so on) and then launches it. For example: In the previous diagram, the Node Manager is allotted 2.5 CPU and 2.5 GB RAM.
  2. NodeManagerThe YARN Node Manager, upon startup, advertises configured resources to YARN's Resource Manager. In the previous example, 2 CPU and 2 GB RAM are advertised. The rest of the resources are used by the Myriad Executor and NodeManager Node Manager processes to run.
  3. YARN's Resource Manager can launch containers now, via this Node Manager. The launched containers are mounted under the configured cgroup hierarchy. 

Image Added  HTML CommentXREF: See [cgroups doc](cgroups.md) for more information.