You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 22 Next »

This section provides information for setting up a cluster in a virtual machine. Setting up a cluster in a virtual machine involves the following:

Prerequisites

In addition to the standard requirements, the following are also needed:

  • Virtualbox
  • Vagrant


Starting the Cluster

To start the cluster, run the following:

vagrant up

 

At this point, the VM has a single node mesos cluster running.

To ssh into the cluster, run the following:

 vagrant ssh

 

The password for the vagrant user is vagrant.

Setting up YARN/Hadoop

To setup YARN/Hadoop inside VM, run the following YARN setup shell files:

1. Run the first yarn setup shell command from the vagrant directory to create a user hduser in group hadoop. Be sure to remember the password that you provide for this user.

 cd /vagrant
./setup-yarn-1.sh

 

2. Run the second yarn setup shell command as sudo.

 sudo su - hduser
cd /vagrant
./setup-yarn-2.sh

 

The following processes should be running:

9844 Jps
6709 NameNode
6393 JobHistoryServer
6874 DataNode

Process IDs are different.

 

Installing Myriad

Installing Myriad involves downloading the Myriad binaries from GitHub. The Myriad location is currently at https://github.com/mesos/myriad.

To install Myriad:

  1. Create a new myriad directory.
  2. Clone the Myriad GitHub.
  3. Pull the contents to the new directory.

 

Configuring Myriad

Configuring the Myriad involves:

  1. Building Myriad Scheduler.
  2. Copying myriad-scheduler files to YARN home.
  3. Copying myriad-executor jar files to Mesos.
  4. Configuring Myriad (myriad-config-default.yml).
  5. Configuring YARN to use Myriad (yarn-site.xml).

Step 1: Run the gradlew build

To build the Myriad Scheduler inside a VM, run the gradlew build:

 cd /vagrant
./gradlew build

If a build failure occurs, the issue is not with the build itself, but a failure to write to disk. This can happen when you build outside the vagrant instance first. To resolve this issue, exit the user hduser (type: exit) and build again as the vagrant user.

 

Step 2: Copy Myriad Scheduler Files

The Myriad Schedule jar files and all the runtime dependences as well as the Myriad configuration file must be copied the $YARN_HOME.

  • The Myriad Scheduler jar files and all the runtime dependencies are located at:

    /vagrant/myriad-scheduler/build/libs/*

  • The Myriad configuration file is located at:

    /vagrant/myriad-scheduler/src/main/resources/myriad-config-default.yml

Copy these Myriad Scheduler files to $YARN_HOME/share/hadoop/yarn/lib/. The default $YARN_HOME is /usr/local/hadoop/.

cp /vagrant/myriad-scheduler/build/libs/* $YARN_HOME/share/hadoop/yarn/lib/ 
cp /vagrant/myriad-scheduler/src/main/resources/myriad-config-default.yml $YARN_HOME/share/hadoop/yarn/lib/

Step 3: Copy Myriad Executor Jar File

The self-contained Myriad Executor jar is available at the following location:

/vagrant/myriad-executor/build/libs/myriad-executor-runnable-x.y.z.jar

 

Copy the Myriad Executor jar file to the /usr/local/libexec/mesos/ directory.

sudo mkdir -p /usr/local/libexec/mesos/
sudo cp /vagrant/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/local/libexec/mesos/
sudo chown hduser:hadoop -R /usr/local/libexec/mesos/

 

Step 4: Configure Myriad Defaults

 As a minimum, the following Myriad configuration parameters must be set:

  • mesosMaster
  • zkServers
  • YARN_HOME

 

Enabling Cgroups involves modifying the yarn-site.xml and myriad-config-default.yml files. If you plan on using Cgroups, you could set that property at this time. See Configuring Cgroups for more information.

 

To configure Myriad itself, update $YARN_HOME/etc/hadoop/myriad-config-default.yml with the following content:

mesosMaster: <mesos Master IP address>:5050
checkpoint: false
frameworkFailoverTimeout: 43200000
frameworkName: MyriadAlpha
nativeLibrary: /usr/local/lib/libmesos.so
zkServers: localhost:2181
zkTimeout: 20000
profiles:
small:
cpu: 1
mem: 1100
medium:
cpu: 2
mem: 2048
large:
cpu: 4
mem: 4096
rebalancer: true
nodemanager:
jvmMaxMemoryMB: 1024
user: hduser
cpus: 0.2
cgroups: false
executor:
jvmMaxMemoryMB: 256
path: file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar

Step 5: Configure YARN to use Myriad

To configure YARN to use Myriad, update $YARN_HOME/etc/hadoop/yarn-site.xml with following content:

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>${nodemanager.resource.cpu-vcores}</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>${nodemanager.resource.memory-mb}</value>
</property>
<!-- Configure Myriad Scheduler here -->
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value>
<description>One can configure other schedulers as well from following list: com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler, com.ebay.myriad.scheduler.yarn.MyriadFifoScheduler</description>
</property>

 

Getting Started

Launching

To launch Myriad, run the following:

sudo su hduser
yarn-daemon.sh start resourcemanager

Verifying Activity

To check that things are running, from a browser on the host check out the following urls:

  • Myriad UI - Use the 8192 port. For example: http://<ip address>:8192/ (http://10.141.141.20:8192/)
  • Mesos UI - Use the 5050 port and Frameworks directory. For example: http://<ip address>:5050/#/frameworks (http://10.141.141.20:5050/#/frameworks)

Shutting Down

To shut down from the vagrant ssh console, run the following:

yarn-daemon.sh stop resourcemanager
./shutdown.sh
exit
exit
vagrant halt
  • No labels