...
Code Block |
---|
./gradlew build |
or to specify your exact hadoop version
Code Block |
---|
./gradlew -PhadoopVer=X.Y.Z build (replace X.Y.Z with your hadoop version) |
At this point, the jars are located in the following directories (relative to $PROJECT_HOME).
...
Before building Myriad Scheduler, modify the myriad-config-default.yml file with the appropriate configuration properties. The build process ./grandlew gradlew build
command builds the myriad-x.x.x.jar file, downloads the runtime jars, and places them inside the ./build/libs/ directory (relative to the $PROJECT_HOME/myriad-scheduler directory).
To build Myriad Scheduler, from $PROJECT_HOME/myriad-scheduler, run:
...
Building Myriad Executor Only
The ./grandlew gradlew build
command builds the myriad-executor-xxx.jar (as a self-contained executor jar file) and places it inside the PROJECT_HOME/myriad-executor/build/libs/ directory.
To build Myriad Executor individually , as a self-contained executor jar, from $PROJECT_HOME/myriad-executor, run:
Code Block | ||
---|---|---|
| ||
./gradlew build |
Step 2: Deploy the Myriad Jar and Configuration Files
To deploy Myriad Scheduler and Executor Common, Executor, and Scheduler jar files:
- Copy the Myriad Scheduler jar files from the $PROJECT_HOME/myriad-scheduler/build/libs/ directory to the $YARN_HOME/share/hadoop/yarn/lib/ directory on all nodes in your cluster.
- Copy the Myriad Executor myriad-executor-xxx.jar file from the $PROJECT_HOME/myriad-executor/build/libs/ directory to each mesos slave's $YARN_HOME/share/hadoop/yarn/lib/ directory.
To deploy the Myriad configuration file:
- Copy the myriad-config-default.yml file from $PROJECT_HOME/myriad-scheduler/build/src/main/resources/ directory to the $YARN_HOME/etc/hadoop directory.
...
Code Block |
---|
cp myriad-scheduler/build/libs/*.jar /opt/hadoop-2.7.10/share/hadoop/yarn/lib/ cp myriad-executor/build/libs/myriad-executor-0.1.0.1.jar /opt/hadoop-2.7.10/share/hadoop/yarn/lib/ cp myriad-scheduler/build/src/main/resources/myriad-config-default.yml /opt/hadoop-2.7.10/etc/hadoop/ |
Note |
---|
For advanced users, you can also copy myriad-executor-xxx.jar to any other directory on a slave filesystem or it can be copied to HDFS as well. In either case, you need to update the executor's path property in the myriad-config-default.yml file and prepend the path with either file:// or hdfs://, as appropriate. |
...
Edit the $YARN_HOME/etc/hadoop/yarn-site.xml file and add the following:
Code Block <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>${nodemanager.resource.cpu-vcores}</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>${nodemanager.resource.memory-mb}</value> </property> <!--These options enable dynamic port assignment by mesos --> <property> <name>yarn.nodemanager.address</name> <value>${myriad.yarn.nodemanager.address}</value> </property> <property> <name>yarn.nodemanager.webapp.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.webapp.https.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.localizer.address</name> <value>${myriad.yarn.nodemanager.localizer.address}</value> </property> <!-- Configure Myriad Scheduler here --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>com<value>org.ebayapache.myriad.scheduler.yarn.MyriadFairScheduler</value> <description>One can configure other scehdulers as well from following list: comorg.ebayapache.myriad.scheduler.yarn.MyriadCapacityScheduler, comorg.ebayapache.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property> <!-- Disable PMem/VMem checks for Hadoop 2.7.2 --> <property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property>
- Edit the $YARN_HOME/etc/hadoop/mapred-site.xml file and add the dynamic port assignment properties.
...
Code Block |
---|
// Add following to $YARN_HOME/etc/hadoop/mapred-site.xml: <!--This option enables dynamic port assignment by mesos --> <property> <name>mapreduce.shuffle.port</name> <value>${myriad.mapreduce.shuffle.port}</value> </property> |
Step 5: Create Hadoop Deployment Tar Files
There are two tar files uploaded to Mesos when the ResourceManager is started: (1) configuration–all of the hadoop configuration files in the $HADOOP_CONF_DIR/ and (2) binary–contains all files of the $HADOOP_HOME directory.
In order to prepare and generate the configuration and binary tar files, do the following:
- Navigate to the parent directory of $HADOOP_HOME (usually /usr/local or /opt) and then generate the binary tar gz file as follows:
Code Block |
---|
cd /opt
tar czf binary.tgz hadoop-2.6.0 |
- Navigate to the $HADOOP_CONF_DIR and generate the config tar gz file as follows:
Code Block |
---|
tar czf config.tgz hadoop |
Step 6: Configure Resource Manager for Deployment to Mesos
Now that the binary.tgz and config.tgz files are all set, specify in the myriad-config-default.yml the relative path to each file from where the Resource Manager is to be launched. A common pattern is to create a dist directory within $HADOOP_HOME. The configuration would be as follows:
Code Block |
---|
servedConfigPath: dist/config.tgz
servedBinaryPath: dist/binary.tgz |
Starting the Resource Manager
Myriad Scheduler runs inside the Resource Manager as a plug-in. To start Start the Resource Manager within the $HADOOP_HOME directory with either of the two following commands:
Code Block | ||
---|---|---|
| ||
./sbin/yarn-daemon.sh start resourcemanager resourcemanager OR bin/yarn resourcemanager |
Running Myriad Executor and Node Managers
...