THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
To deploy Myriad Scheduler and Executor , copy the Myriad Scheduler and Executor jar files and the Myriad configuration file, myriad-config-default.yml, into your YARN classpath:
- Copy the Myriad Scheduler jar files from the $PROJECT_HOME/myriad-scheduler/build/libs/ directory to the YARN ResourceManager's class path, $YARN_HOME/share/hadoop/yarn/lib/ directory, on all nodes in your cluster. On each Mesos slave node, create a mesos directory under the /usr/local/libexec directory.
- Copy the Myriad Executor myriad-executor-xxx.jar file from the $PROJECT_HOME/myriad-executor/build/libs/ to directory to each mesos slave's /$YARN_HOME/share/hadoop/yarn/lib/ directory.
- Copy the myriad-config-default.yml file from $PROJECT_HOME/myriad-scheduler/build/src/main/resources/ directory to the $YARN_HOME/etc/hadoop directory.
For example:
| Code Block |
|---|
cp myriad-scheduler/build/libs/*.jar /opt/hadoop-2.7.1/share/hadoop/yarn/lib/ cp myriad-schedulerexecutor/build/src/main/resourceslibs/myriad-config-default.ymlexecutor-0.0.1.jar /opt/hadoop-2.7.1/share/hadoop/yarn/lib/ cp myriad-executorscheduler/build/src/main/libsresources/myriad-executorconfig-0.0.1.jardefault.yml /opt/hadoop-2.7.1/shareetc/hadoop/yarn/lib/ |
| Note |
|---|
For advanced users, you can also copy myriad-executor-xxx.jar to any other directory on a slave filesystem or it can be copied to HDFS as well. In either case, you need to update the executor's path property in the myriad-config-default.yml file and prepend the path with either file:// or hdfs://, as appropriate. |
...
Step 3: Configure the Myriad Defaults
Myriad configuration parameters must be specified before building Myriad. These configuration parameters are specified in the the myriad-config-default.yml file in the $PROJECT_HOME/myriad-scheduler/src/main directory. This is required because the myriad-config-default.yml file is embedded into the Myriad Scheduler jar. As As a minimum, the following Myriad configuration parameters must be set:
- mesosMaster
- zkServers
- YARN_HOME
...
| Note |
|---|
Enabling Cgroups involves modifying the yarn-site.xml and myriad-config-default.yml files. If you plan on using Cgroups, you could set that property at this time. See Configuring Cgroups for more information. |
| Note |
|---|
By copying the myriad-config-default.yml file to the /etc/hadoop directory, you can make changes to the configuration file without having to rebuild Myriad. If you specify the Myriad configuration parameters before building Myriad, you must rebuild Myriad and redeploy the jar files. This is required because the myriad-config-default.yml file is embedded into the Myriad Scheduler jar. |
Step 4: Configure YARN to use Myriad
In order to run Myriad, the following YARN properties must be modified :
...
on
...
YARN Node Manager resource properties to the $YARN_HOME/etc/hadoop/yarn-site.xml file on all nodes.
...
Dynamic port assignment to the mapred-site.xml file on all nodes.
Add MESOS_NATIVE_JAVA_LIBRARY Environment Variable
On each node in the cluster, edit :
Edit the $YARN_HOME/etc/hadoop/hadoop-env.sh file and add the following:
Code Block export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
Add YARN Node Manager Resource Properties
...
Edit the $YARN_HOME/etc/hadoop/yarn-site.xml file and add the following:
Code Block <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>${nodemanager.resource.cpu-vcores}</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>${nodemanager.resource.memory-mb}</value> </property> <!--These options enable dynamic port assignment by mesos --> <property> <name>yarn.nodemanager.address</name> <value>${myriad.yarn.nodemanager.address}</value> </property> <property> <name>yarn.nodemanager.webapp.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.webapp.https.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.localizer.address</name> <value>${myriad.yarn.nodemanager.localizer.address}</value> </property> <!-- Configure Myriad Scheduler here --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value> <description>One can configure other scehdulers as well from following list: com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler, com.ebay.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property>
...
Enable Dynamic Port Assignment
...
- Edit the $YARN_HOME/etc/hadoop
...
- /mapred-site.xml file
...
- and add the dynamic port assignment
...
- properties.
- On each node, change directory to $YARN_HOME/etc/hadoop.
- Copy mapred-site.xml.template to mapred-site.xml.
- Edit and add the following property to the mapred-site.xml file.
| Code Block |
|---|
// Add following to $YARN_HOME/etc/hadoop/mapred-site.xml:
<!--This option enables dynamic port assignment by mesos -->
<property>
<name>mapreduce.shuffle.port</name>
<value>${myriad.mapreduce.shuffle.port}</value>
</property> |
...