Page History

...

./gradlew build

or to specify your exact hadoop version

./gradlew -PhadoopVer=X.Y.Z build (replace X.Y.Z with your hadoop version)

At this point, the jars are located in the following directories (relative to $PROJECT_HOME).

...

./gradlew build

Step 2: Deploy the Myriad Jar and Configuration Files

To deploy Myriad Scheduler and Executor Common, Executor, and Scheduler jar files:

1. Copy the Myriad Scheduler jar files from the $PROJECT_HOME/myriad-scheduler/build/libs/ directory to the $YARN_HOME/share/hadoop/yarn/lib/ directory on all nodes in your cluster. 
2. Copy the Myriad Executor myriad-executor-xxx.jar file from the $PROJECT_HOME/myriad-executor/build/libs/ directory to each mesos slave's $YARN_HOME/share/hadoop/yarn/lib/ directory.

To deploy the Myriad configuration file:

1. Copy the myriad-config-default.yml file from $PROJECT_HOME/myriad-scheduler/build/src/main/resources/ directory to the $YARN_HOME/etc/hadoop directory.

...

• Edit the $YARN_HOME/etc/hadoop/yarn-site.xml file and add the following:

  Code Block

  <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>${nodemanager.resource.cpu-vcores}</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>${nodemanager.resource.memory-mb}</value> </property> <!--These options enable dynamic port assignment by mesos --> <property> <name>yarn.nodemanager.address</name> <value>${myriad.yarn.nodemanager.address}</value> </property> <property> <name>yarn.nodemanager.webapp.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.webapp.https.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.localizer.address</name> <value>${myriad.yarn.nodemanager.localizer.address}</value> </property> <!-- Configure Myriad Scheduler here --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value> <description>One can configure other scehdulers as well from following list: org.apache.myriad.scheduler.yarn.MyriadCapacityScheduler, org.apache.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property> <!-- Disable PMem/VMem checks for Hadoop 2.7.2 --> <property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property>

• Edit the $YARN_HOME/etc/hadoop/mapred-site.xml file and add the dynamic port assignment properties.

...

// Add following to $YARN_HOME/etc/hadoop/mapred-site.xml:

<!--This option enables dynamic port assignment by mesos -->
<property>
    <name>mapreduce.shuffle.port</name>
    <value>${myriad.mapreduce.shuffle.port}</value>
</property>

Step 5: Create Hadoop Deployment Tar Files

There are two tar files uploaded to Mesos when the ResourceManager is started: (1) configuration–all of the hadoop configuration files in the $HADOOP_CONF_DIR/ and (2) binary–contains all files of the $HADOOP_HOME directory.

In order to prepare and generate the configuration and binary tar files, do the following:

• Navigate to the parent directory of $HADOOP_HOME (usually  /usr/local or /opt) and then generate the binary tar gz file as follows:
  
  

cd /opt
tar czf binary.tgz hadoop-2.6.0

• Navigate to the $HADOOP_CONF_DIR and generate the config tar gz file as follows:
  
  

tar czf config.tgz hadoop

Step 6: Configure Resource Manager for Deployment to Mesos

Now that the binary.tgz and config.tgz files are all set, specify in the myriad-config-default.yml the relative path to each file from where the Resource Manager is to be launched.  A common pattern is to create a dist directory within $HADOOP_HOME. The configuration would be as follows:

servedConfigPath: dist/config.tgz
servedBinaryPath: dist/binary.tgz

Starting the Resource Manager

Myriad Scheduler runs inside the Resource Manager as a plug-in. To start Start the Resource Manager within the $HADOOP_HOME directory with either of the two following commands:

./sbin/yarn-daemon.sh start resourcemanager  resourcemanager 
OR
bin/yarn resourcemanager

Running Myriad Executor and Node Managers

...