Editor Support
You are welcome to use any editor you like, however, many of the Myriad developers use IntelliJ IDEA. The build process uses Gradle (https://gradle.org/). The build script is configured with the Idea Plugin (https://docs.gradle.org/current/userguide/idea_plugin.html) to create an IntelliJ project structure. This is accomplished with the following command:
./gradlew idea // to open on a mac open myriad.ipr
If you are interested in using Eclipse, there is a plugin for that. Either send in a pull request, or send a message on the dev mail list that you would make use of it.
Step 1: Build Myriad
To build Myriad (both Myriad Scheduler and Executor), from the $PROJECT_HOME, run:
./gradlew build
At this point, the jars are located in the following directories (relative to $PROJECT_HOME).
# scheduler jars ./myriad-scheduler/build/libs/ # executor jar ./myriad-executor/build/libs/
Building Myriad Scheduler Only
Before building Myriad Scheduler, modify the myriad-config-default.yml file with the appropriate configuration properties. The build process ./grandlew build
command builds the myriad-x.x.x.jar file, downloads the runtime jars, and places them inside the ./build/libs/ directory (relative to the $PROJECT_HOME/myriad-scheduler directory).
To build Myriad Scheduler, from $PROJECT_HOME/myriad-scheduler, run:
./gradlew build
Building Myriad Executor Only
The ./grandlew build
command builds the myriad-executor-xxx.jar (as a self-contained executor jar file) and places it inside the PROJECT_HOME/myriad-executor/build/libs/ directory.
To build Myriad Executor individually, self-contained executor jar, from $PROJECT_HOME/myriad-executor, run:
./gradlew build
Step 2: Deploy Myriad
To deploy Myriad Scheduler and Executor, copy the Myriad Scheduler and Executor jar files and the Myriad configuration file, myriad-config-default.yml, into your YARN classpath:
- Copy the Myriad Scheduler jar files from $PROJECT_HOME/myriad-scheduler/build/libs/ directory to the YARN ResourceManager's class path on all nodes in your cluster. For example, copy all jars to the $YARN_HOME/share/hadoop/yarn/lib/ directory.
- On each Mesos slave node, create a mesos directory under the /usr/local/libexec directory.
- Copy the Myriad Executor myriad-executor-xxx.jar file from $PROJECT_HOME/myriad-executor/build/libs/ to each mesos slave's /$YARN_HOME/share/hadoop/yarn/lib/s
cp myriad-executor/build/libs/myriad-executor-
0.0
.
1
.jar /opt/hadoop-
2.7
.
1
/share/hadoop/yarn/lib/
directory.
For example:
cp myriad-scheduler/build/libs/*.jar /opt/hadoop-2.7.1/share/hadoop/yarn/lib/ cp myriad-executor/build/libs/myriad-executor-0.0.1.jar /opt/hadoop-2.7.1/share/hadoop/yarn/lib/ cp myriad-scheduler/build/src/main/resources/myriad-config-default.yml /opt/hadoop-2.7.1/share/hadoop/yarn/lib/
For advanced users, you can also copy myriad-executor-xxx.jar to any other directory on a slave filesystem or it can be copied to HDFS as well. In either case, you need to update the executor's path property in the myriad-config-default.yml file and prepend the path with either file:// or hdfs://, as appropriate.
Step 3: Modify Myriad Configuration
Myriad configuration parameters must be specified before building Myriad. These configuration parameters are specified in the the myriad-config-default.yml file in the $PROJECT_HOME/myriad-scheduler/src/main directory. This is required because the myriad-config-default.yml file is embedded into the Myriad Scheduler jar. As a minimum, the following Myriad configuration parameters must be set:
- mesosMaster
- zkServers
- YARN_HOME
Enabling Cgroups involves modifying the yarn-site.xml and myriad-config-default.yml files. If you plan on using Cgroups, you could set that property at this time. See Configuring Cgroups for more information.
Step 4: Configure YARN
In order to run Myriad, the following YARN properties must be modified:
MESOS_NATIVE_JAVA_LIBRARY environment variable to the $YARN_HOME/etc/hadoop/hadoop-env.sh file on all nodes.
YARN Node Manager resource properties to the $YARN_HOME/etc/hadoop/yarn-site.xml file on all nodes.
Dynamic port assignment to the mapred-site.xml file on all nodes.
Add MESOS_NATIVE_JAVA_LIBRARY Environment Variable
On each node in the cluster, edit the $YARN_HOME/etc/hadoop/hadoop-env.sh file and add the following:
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
Add YARN Node Manager Resource Properties
On each node in the cluster, edit the $YARN_HOME/etc/hadoop/yarn-site.xml file and add the following:
<property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>${nodemanager.resource.cpu-vcores}</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>${nodemanager.resource.memory-mb}</value> </property> <!--These options enable dynamic port assignment by mesos --> <property> <name>yarn.nodemanager.address</name> <value>${myriad.yarn.nodemanager.address}</value> </property> <property> <name>yarn.nodemanager.webapp.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.webapp.https.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.localizer.address</name> <value>${myriad.yarn.nodemanager.localizer.address}</value> </property> <!-- Configure Myriad Scheduler here --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value> <description>One can configure other scehdulers as well from following list: com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler, com.ebay.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property>
Enable Dynamic Port Assignment
On each node in the cluster, from the $YARN_HOME/etc/hadoop directory, update the mapred-site.xml file by enabling dynamic port assignment by Mesos.
- On each node, change directory to $YARN_HOME/etc/hadoop.
- Copy mapred-site.xml.template to mapred-site.xml.
- Edit and add the following property to the mapred-site.xml file.
// Add following to $YARN_HOME/etc/hadoop/mapred-site.xml: <!--This option enables dynamic port assignment by mesos --> <property> <name>mapreduce.shuffle.port</name> <value>${myriad.mapreduce.shuffle.port}</value> </property>
Starting the Resource Manager
Myriad Scheduler runs inside the Resource Manager as a plug-in. To start the Resource Manager:
./sbin/yarn-daemon.sh start resourcemanager
Running Myriad Executor and Node Managers
Myriad Executor and Node Managers are launched automatically by Myriad Scheduler as a response to flexup and flexdown behavior. See the Myriad Cluster API.