...
This section contains the following topics:
Table of Contents |
---|
Prerequisites
- Virtualbox
- Vagrant
Starting the Cluster
To start the cluster, run following:
...
The password for the vagrant user is vagrant.
Setting up YARN/Hadoop
To setup YARN/Hadoop inside VM, run the following yarn setup shell files:
...
Note |
---|
Process IDs are different. |
Building the Myriad Scheduler
Building the Myriad Scheduler involves:
1. Running ` ./gradlew build`build.
2. Copying myriad-scheduler jar files to YARN home.
3. Copying myriad-executor jar files to mesos.
4. Configuring YARN to use Myriad (yarn-site.xml).
5. Configuring Myriad (myriad-config-default.yml).
Step 1: Run gradlew build
To build the Myriad Scheduler inside a VM, run the gradlew build:
Code Block |
---|
cd /vagrant ./gradlew build |
...
Note |
---|
If a build failure occurs, the issue is not with the build itself, but a failure to write to disk. This can happen when you build outside the vagrant instance first. To resolve this issue, exit the user |
...
hduser (type: exit) and build again as the |
...
vagrant user. |
{% include endnote.html %}
Step 2: Copy the Myriad Scheduler Jar Files
At this point, myriad's scheduler jar and all the runtime dependencies are available in the following location:
...
Code Block |
---|
cp /vagrant/myriad-scheduler/build/libs/* /usr/local/hadoop/share/hadoop/yarn/lib/ |
Step 3: Copy the Myriad Executor Jar File
The self-contained myriad executor jar is available at the following location:
Code Block |
---|
...
/vagrant/myriad-executor/build/libs/myriad-executor-runnable-x.y.z.jar |
jar`
Copy the myriad executor jar file to `the /usr/local/libexec/mesos/` directory.
Code Block |
---|
sudo mkdir -p /usr/local/libexec/mesos/ sudo cp /vagrant/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/local/libexec/mesos/ sudo chown hduser:hadoop -R /usr/local/libexec/mesos/ |
Step 4: Configure YARN to use Myriad
To configure YARN to use Myriad, update ```$YARN$YARN_HOME/etc/hadoop/yarn-site.xml``` xml with following content:
Code Block |
---|
<property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>${nodemanager.resource.cpu-vcores}</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>${nodemanager.resource.memory-mb}</value> </property> <!-- Configure Myriad Scheduler here --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value> <description>One can configure other scehdulers as well from following list: com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler, com.ebay.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property> |
Step 5: Configure Myriad Defaults
To configure Myriad itself, update $YARN_HOME/etc/hadoop/myriad-default-config.yml with the following content:
Code Block |
---|
mesosMaster: 10.141.141.20:5050 checkpoint: false frameworkFailoverTimeout: 43200000 frameworkName: MyriadAlpha nativeLibrary: /usr/local/lib/libmesos.so zkServers: localhost:2181 zkTimeout: 20000 profiles: small: cpu: 1 mem: 1100 medium: cpu: 2 mem: 2048 large: cpu: 4 mem: 4096 rebalancer: true nodemanager: jvmMaxMemoryMB: 1024 user: hduser cpus: 0.2 cgroups: false executor: jvmMaxMemoryMB: 256 path: file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.0.1.jar |
Getting Started
To launch Myriad, run following:
Code Block |
---|
sudo su hduser yarn-daemon.sh start resourcemanager |
Verifying Activity
To check that things are running, from a browser on the host check out the following urls:
- [Myriad UI](http://10.141.141.20:8192/)
- [Mesos UI - Frameworks](http://10.141.141.20:5050/#/frameworks)
Shutting Down
To shut down from the vagrant ssh console, do the following:
...