THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Code Block |
|---|
vagrant up |
At this point , the VM has will have a single node Mesos cluster running and a single node mesos cluster running.
HDFS cluster running. You can modify the Mesos and Hadoop versions by editing the
corresponding variables in the `Vagrantfile`.
Check that Mesos and Hadoop are running with a browser:
Hadoop/HDFS namenode: http://10.141.141.20:50070
Mesos Master: http://10.141.141.20:5050/
To ssh into the cluster, run the following:
| Code Block |
|---|
vagrant ssh |
...
The password for the vagrant user is vagrant.
Setting up YARN/Hadoop
To setup YARN/Hadoop inside VM, run the following YARN setup shell files:
1. Run the first yarn setup shell command from the vagrant directory to create a user hduser in group hadoop. Be sure to remember the password that you provide for this user.
| Code Block |
|---|
cd /vagrant
./setup-yarn-1.sh |
2. Run the second yarn setup shell command as sudo.
You can switch to the hadoop user when needed to launch hadoop processes
| Code Block |
|---|
sudo |
| Code Block |
sudo su - hduser cd /vagrant ./setup-yarn-2.sh |
The following processes should be running:
| Code Block |
|---|
9844 Jps
6709 NameNode
6393 JobHistoryServer
6874 DataNode |
| Note |
|---|
Process IDs are different. |
Installing Myriad
Installing Myriad involves downloading the Myriad binaries from GitHub. The Myriad location is currently at https://github.com/mesos/myriad.
To install Myriad:
- Create a new myriad directory.
- Clone the Myriad GitHub.
- Pull the contents to the new directory.
Configuring Myriad
Configuring the Myriad involves:
- Building Myriad Scheduler.
- Copying myriad-scheduler files to YARN home.
- Copying myriad-executor jar files to Mesos.
- Configuring Myriad (myriad-config-default.yml).
- Configuring YARN to use Myriad (yarn-site.xml).
...
Configuring Myriad
To configure Myriad:
Step 1: Build Myriad
To build the Myriad Scheduler inside a VM, run the gradlew build:
...
| Note |
|---|
If a build failure occurs, the issue is not with the build itself, but a failure to write to disk. This can happen when you build outside the vagrant instance first. To resolve this issue, exit the user hduser (type: exit) and build again as the vagrant user. |
Step 2:
...
Deploy the Myriad
...
Files
The Myriad Schedule and Executer jar files and all the runtime dependences as well as the Myriad configuration file must be copied the to $YARN_HOME.
- The Myriad Scheduler jar files and all the runtime dependencies are located at:
| Code Block |
|---|
/vagrant/myriad-scheduler/build/libs/* |
- The Myriad configuration file is located at:
| Code Block |
|---|
/vagrant/myriad-scheduler/src/main/resources/myriad-config-default.yml |
- The Myriad Executor jar file are located at:
| Code Block |
|---|
/vagrant/myriad-scheduler/build/libs/myriad-executor-0.1.0.jar |
For example, the files are copied to the following locations:Copy these Myriad Scheduler files to $YARN_HOME/share/hadoop/yarn/lib/. The default $YARN_HOME is /usr/local/hadoop/.
| Code Block |
|---|
cp /vagrant/myriad-scheduler/build/libs/* $YARN_HOME/share/hadoop/yarn/lib/ cp /vagrant/myriad-schedulerexecutor/srcbuild/mainlibs/resources/myriad-config-default.ymlexecutor-0.1.0.jar $YARN_HOME/share/hadoop/yarn/lib/ |
Step 3: Copy Myriad Executor Jar File
The self-contained Myriad Executor jar is available at the following location:
| Code Block |
|---|
/vagrant/myriad-executor/build/libs/myriad-executor-runnable-x.y.z.jar |
...
/ |
...
| Code Block |
|---|
sudo mkdir -p /usr/local/libexec/mesos/ sudo cp /vagrant/myriad-executorscheduler/src/buildmain/libsresources/myriad-executorconfig-runnable-0.0.1.jar /usr/local/libexec/mesos/ sudo chown hduser:hadoop -R /usr/local/libexec/mesos/ |
...
default.yml $YARN_HOME/etc/hadoop/
|
Step 3: Configure the Myriad Defaults
As a minimum, the following Myriad configuration parameters must be set:
...
| Note |
|---|
Enabling Cgroups involves modifying the yarn-site.xml and myriad-config-default.yml files. If you plan on using Cgroups, you could set that property at this time. See Configuring Cgroups for more information. |
To configure Myriad itself, update $YARN_HOME/etc/hadoop/myriad-config-default.yml with the following content:
| Code Block |
|---|
mesosMaster: <mesos Master IP address>:5050 checkpoint: false frameworkFailoverTimeout: 43200000 frameworkName: MyriadAlpha nativeLibrary: /usr/local/lib/libmesos.so zkServers: localhost:2181 zkTimeout: 20000 profiles: small: cpu: 12 mem: 11002048 medium: cpu: 24 mem: 20484096 large: cpu: 410 mem: 409612288 rebalancer: true nodemanager: jvmMaxMemoryMB: 1024 user: hduser cpus: 0.2 cgroups: false executor: jvmMaxMemoryMB: 256 path: file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.1.0.1.jar |
Step
...
4: Configure YARN to use Myriad
To configure YARN to use Myriad, update $YARN_HOME/etc/hadoop/yarn-site.xml with following content:
| Code Block |
|---|
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>${nodemanager.resource.cpu-vcores}</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>${nodemanager.resource.memory-mb}</value>
</property>
<!-- The following properties enable dynamic port assignment by Mesos -->
<property>
<name>yarn.nodemanager.address</name>
<value>${myriad.yarn.nodemanager.address}</value>
</property>
<property>
<name>yarn.nodemanager.webapp.address</name>
<value>${myriad.yarn.nodemanager.webapp.address}</value>
</property>
<property>
<name>yarn.nodemanager.webapp.https.address</name>
<value>${myriad.yarn.nodemanager.webapp.address}</value>
</property>
<property>
<name>yarn.nodemanager.localizer.address</name>
<value>${myriad.yarn.nodemanager.localizer.address}</value>
</property>
<!-- The following properties configure Myriad Scheduler -->
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>com<value>org.ebayapache.myriad.scheduler.yarn.MyriadFairScheduler</value>
<description>One can configure other schedulers as well from following list: comorg.ebayapache.myriad.scheduler.yarn.MyriadCapacityScheduler, comorg.ebayapache.myriad.scheduler.yarn.MyriadFifoScheduler</description>
</property> |
...
To launch Myriad, run the following:
| Code Block |
|---|
sudo su - hduser
yarn-daemon.sh start resourcemanager |
...
- Myriad UI - Use the 8192 port. For example: http://<ip address><IP address>:8192/ (http://10.141.141.20:8192/)
- Without the mesosMesos-dns DNS service, this IP is slave node's IP address where the ResourceManager is running.
- With the mesosMesos-dns DNS service, mesosMesos-dns DNS discovers the node where the ResourceManager is running using mesos by using Mesos DNS FQDN (<app>.<framework>.mesos).
- Mesos UI - Use the 5050 port and Frameworks directory. For example: http://<ip <IP address>:5050/#/frameworks (http://10.141.141.20:5050/#/frameworks)
...
| Code Block |
|---|
yarn-daemon.sh stop resourcemanager
./vagrant/shutdown.sh
exit
exit
vagrant halt |