Page History

This section provides information for setting up a cluster in a virtual machine. To setup Setting up a cluster in a virtual machine involves the following:

1. Start the cluster.
2. Setup YARN/Hadoop.
3. Build the Myriad Scheduler.
4. Get started.

This section contains the following topics:

Prerequisites

In addition to the standard requirements, the following are also needed:

...

• Virtualbox
• Vagrant

Starting the Cluster

To start the cluster, run the following:

vagrant up

At this point , the VM has will have a single node Mesos cluster running and a single node mesos
HDFS cluster running. You can modify the Mesos and Hadoop versions by editing the
corresponding variables in the `Vagrantfile`.

Check that Mesos and Hadoop are running with a browser:

Hadoop/HDFS namenode: http://10.141.141.20:50070
Mesos Master: http://10.141.141.20:5050/

To ssh into the cluster, run the following:

 vagrant ss

...

ssh

The password for the vagrant user is vagrant.

Setting up YARN/Hadoop

To setup YARN/Hadoop inside VM, run the following yarn setup shell files:

1. Run the first yarn setup shell command from the vagrant directory to create a user hduser in group hadoop. Be sure to remember the password that you provide for this user.

 cd /vagrant
./setup-yarn-1.sh

2. Run the second yarn setup shell command as sudo.

You can switch to the hadoop user when needed to launch hadoop processes

sudo

 sudo su - hduser
cd /vagrant
./setup-yarn-2.sh

The following processes should be running:

9844 Jps
6709 NameNode
6393 JobHistoryServer
6874 DataNode

...

Building the Myriad Scheduler involves:

1. Running ./gradlew build.
2. Copying myriad-scheduler jar files to YARN home.
3. Copying myriad-executor jar files to mesos.
4. Configuring YARN to use Myriad (yarn-site.xml).
5. Configuring Myriad (myriad-config-default.yml).

Configuring Myriad

To configure Myriad:

Step 1: Build Myriad

...

To build the Myriad Scheduler inside a VM, run the gradlew build:

...

Note
If a build failure occurs, the issue is not with the build itself, but a failure to write to disk. This can happen when you build outside the vagrant instance first. To resolve this issue, exit the user hduser (type: exit) and build again as the vagrant user.

...

Step 2:

...

Deploy the Myriad

...

Files

The Myriad Schedule and Executer jar files and all the runtime dependences as well as the Myriad configuration file must be copied to $YARN_HOME.

• The Myriad Scheduler jar files and all the runtime dependencies are

...

• located at:

 /vagrant/myriad-scheduler/build/libs/*

...

Copy these myriad scheduler files to `$YARN_HOME/share/hadoop/yarn/lib/`. The default `$YARN_HOME` is `/usr/local/hadoop/`.

• The Myriad configuration file is located at:

cp /vagrant/myriad-scheduler/buildsrc/libsmain/* /usr/local/hadoop/share/hadoop/yarn/lib/ 

Step 3: Copy the Myriad Executor Jar File

...

resources/myriad-config-default.yml

• The Myriad Executor jar file are located at:

/vagrant/myriad-executorscheduler/build/libs/myriad-executor-runnable-x0.y1.z0.jar

Copy the myriad executor jar file to the /usr/local/libexec/mesos/ directory.

For example, the files are copied to the following locations:

cp /vagrant/myriad-scheduler/build/libs/* $YARN_HOME/share/hadoop/yarn/lib/ 

sudo mkdir -p /usr/local/libexec/mesos/
sudo cp /vagrant/myriad-executor/build/libs/myriad-executor-runnable-0.1.0.1.jar $YARN_HOME/usrshare/localhadoop/libexecyarn/mesoslib/
sudo chown hduser:hadoop -R /usr/local/libexec/mesos/

Step 4: Configure Myriad Defaults

cp /vagrant/myriad-scheduler/src/main/resources/myriad-config-default.yml $YARN_HOME/etc/hadoop/

Step 3: Configure the Myriad Defaults

 As a minimum, the following Myriad configuration parameters must be set:

• mesosMaster
• zkServers
• YARN_HOME

To configure MyriadTo configure Myriad itself, update $YARN_HOME/etc/hadoop/myriad-config-default.yml with the following content:

mesosMaster: 10.141.141.20 <mesos Master IP address>:5050
checkpoint: false
frameworkFailoverTimeout: 43200000
frameworkName: MyriadAlpha
nativeLibrary: /usr/local/lib/libmesos.so
zkServers: localhost:2181
zkTimeout: 20000
profiles:
small:
cpu: 12
mem: 11002048
medium:
cpu: 24
mem: 20484096
large:
cpu: 410
mem: 409612288
rebalancer: true
nodemanager:
jvmMaxMemoryMB: 1024
user: hduser
cpus: 0.2
cgroups: false
executor:
jvmMaxMemoryMB: 256
path: file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.1.0.1.jar

Step

...

4: Configure YARN to use Myriad

To configure YARN to use Myriad, update $YARN_HOME/etc/hadoop/yarn-site.xml with following content:

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>${nodemanager.resource.cpu-vcores}</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>${nodemanager.resource.memory-mb}</value>
</property>
<!-- The following properties enable dynamic port assignment by Mesos -->
<property>
    <name>yarn.nodemanager.address</name>
    <value>${myriad.yarn.nodemanager.address}</value>
</property>
<property>
    <name>yarn.nodemanager.webapp.address</name>
    <value>${myriad.yarn.nodemanager.webapp.address}</value>
</property>
<property>
    <name>yarn.nodemanager.webapp.https.address</name>
    <value>${myriad.yarn.nodemanager.webapp.address}</value>
</property>
<property>
    <name>yarn.nodemanager.localizer.address</name>
    <value>${myriad.yarn.nodemanager.localizer.address}</value>
</property>
<!-- Configure The following properties configure Myriad Scheduler here -->
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>com<value>org.ebayapache.myriad.scheduler.yarn.MyriadFairScheduler</value>
<description>One can configure other scehdulersschedulers as well from following list: comorg.ebayapache.myriad.scheduler.yarn.MyriadCapacityScheduler, comorg.ebayapache.myriad.scheduler.yarn.MyriadFifoScheduler</description>
</property>

Getting Started

Launching

To launch Myriad, run the following:

sudo su - hduser
yarn-daemon.sh start resourcemanager

Verifying Activity

To check that things are running, from a browser on the host check out the following urls:

• Myriad UI - Use the 8192 port. For example: http://<ip address><IP address>:8192/ (http://10.141.141.20:8192/)

• • Without the Mesos-DNS service, this IP is slave node's IP address where the ResourceManager is running.
  • With the Mesos-DNS service, Mesos-DNS discovers the node where the ResourceManager is running by using Mesos DNS FQDN (<app>.<framework>.mesos).

• Mesos UI - Use the 5050 port and Frameworks directory. For example: http://<ip <IP address>:81925050/#/frameworks (http://10.141.141.20:5050/#/frameworks)

Shutting Down

To shut down from the vagrant ssh console, do run the following:

yarn-daemon.sh stop resourcemanager
./vagrant/shutdown.sh
exit
exit
vagrant halt