Installing for Developers

Editor Support

You are welcome to use any editor you like, however, many of the Myriad developers use IntelliJ IDEA. The build process uses Gradle (https://gradle.org/). The build script is configured with the Idea Plugin (https://docs.gradle.org/current/userguide/idea_plugin.html) to create an IntelliJ project structure. This is accomplished with the following command:

./gradlew idea
// to open on a mac
open myriad.ipr

Step 1: Build Myriad

To build Myriad (both Myriad Scheduler and Executor), from the $PROJECT_HOME, run:

./gradlew build

At this point, the jars are located in the following directories (relative to $PROJECT_HOME).

# scheduler jars
./myriad-scheduler/build/libs/

# executor jar
./myriad-executor/build/libs/

Building Myriad Scheduler Only

Before building Myriad Scheduler, modify the myriad-config-default.yml file with the appropriate configuration properties. The build process ./grandlew build command builds the myriad-x.x.x.jar file, downloads the runtime jars, and places them inside the ./build/libs/ directory (relative to the $PROJECT_HOME/myriad-scheduler directory).

To build Myriad Scheduler, from $PROJECT_HOME/myriad-scheduler, run:

./gradlew build

Building Myriad Executor Only

The ./grandlew build command builds the myriad-executor-xxx.jar (as a self-contained executor jar file) and places it inside the PROJECT_HOME/myriad-executor/build/libs/ directory.

To build Myriad Executor individually, self-contained executor jar, from $PROJECT_HOME/myriad-executor, run:

./gradlew build

Step 2: Deploy Myriad

To deploy Myriad Scheduler and Executor, copy the Myriad Scheduler and Executor jar files and the Myriad configuration file, myriad-config-default.yml, into your YARN classpath:

1. Copy the Myriad Scheduler jar files from $PROJECT_HOME/myriad-scheduler/build/libs/ directory to the YARN ResourceManager's class path on all nodes in your cluster. For example, copy all jars to the $YARN_HOME/share/hadoop/yarn/lib/ directory.
2. On each Mesos slave node, create a mesos directory under the /usr/local/libexec directory.
3. Copy the Myriad Executor myriad-executor-xxx.jar file from $PROJECT_HOME/myriad-executor/build/libs/ to each mesos slave's /$YARN_HOME/share/hadoop/yarn/lib/s cp myriad-executor/build/libs/myriad-executor-0.0.1.jar /opt/hadoop-2.7.1/share/hadoop/yarn/lib/directory.

For example:

cp myriad-scheduler/build/libs/*.jar /opt/hadoop-2.7.1/share/hadoop/yarn/lib/
cp myriad-executor/build/libs/myriad-executor-0.0.1.jar /opt/hadoop-2.7.1/share/hadoop/yarn/lib/
cp myriad-scheduler/build/src/main/resources/myriad-config-default.yml /opt/hadoop-2.7.1/share/hadoop/yarn/lib/

Step 3: Modify Myriad Configuration

Myriad configuration parameters must be specified before building Myriad. These configuration parameters are specified in the the myriad-config-default.yml file in the $PROJECT_HOME/myriad-scheduler/src/main directory.  This is required because the myriad-config-default.yml file is embedded into the Myriad Scheduler jar. As a minimum, the following Myriad configuration parameters must be set:

• mesosMaster
• zkServers
• YARN_HOME

Step 4: Configure YARN

In order to run Myriad, the following YARN properties must be modified:

• MESOS_NATIVE_JAVA_LIBRARY environment variable to the $YARN_HOME/etc/hadoop/hadoop-env.sh file on all nodes.

• YARN Node Manager resource properties to the $YARN_HOME/etc/hadoop/yarn-site.xml file on all nodes.

• Dynamic port assignment to the mapred-site.xml file on all nodes.

Add MESOS_NATIVE_JAVA_LIBRARY Environment Variable

On each node in the cluster, edit the $YARN_HOME/etc/hadoop/hadoop-env.sh file and add the following:

export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so

Add YARN Node Manager Resource Properties

On each node in the cluster, edit the $YARN_HOME/etc/hadoop/yarn-site.xml file and add the following:

<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>${nodemanager.resource.cpu-vcores}</value>
</property>
<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>${nodemanager.resource.memory-mb}</value>
</property>
<!--These options enable dynamic port assignment by mesos -->
<property>
    <name>yarn.nodemanager.address</name>
    <value>${myriad.yarn.nodemanager.address}</value>
</property>
<property>
    <name>yarn.nodemanager.webapp.address</name>
    <value>${myriad.yarn.nodemanager.webapp.address}</value>
</property>
<property>
    <name>yarn.nodemanager.webapp.https.address</name>
    <value>${myriad.yarn.nodemanager.webapp.address}</value>
</property>
<property>
    <name>yarn.nodemanager.localizer.address</name>
    <value>${myriad.yarn.nodemanager.localizer.address}</value>
</property>
<!-- Configure Myriad Scheduler here -->
<property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value>
    <description>One can configure other scehdulers as well from following list: com.ebay.myriad.scheduler.yarn.MyriadCapacityScheduler, com.ebay.myriad.scheduler.yarn.MyriadFifoScheduler</description>
</property>

Enable Dynamic Port Assignment

On each node in the cluster, from the $YARN_HOME/etc/hadoop directory, update the mapred-site.xml file by enabling dynamic port assignment by Mesos.

1. On each node, change directory to $YARN_HOME/etc/hadoop.
2. Copy mapred-site.xml.template to mapred-site.xml.
3. Edit and add the following property to the mapred-site.xml file.

// Add following to $YARN_HOME/etc/hadoop/mapred-site.xml:

<!--This option enables dynamic port assignment by mesos -->
<property>
    <name>mapreduce.shuffle.port</name>
    <value>${myriad.mapreduce.shuffle.port}</value>
</property>

Starting the Resource Manager

Myriad Scheduler runs inside the Resource Manager as a plug-in. To start the Resource Manager:

./sbin/yarn-daemon.sh start resourcemanager 

Running Myriad Executor and Node Managers

Myriad Executor and Node Managers are launched automatically by Myriad Scheduler as a response to flexup and flexdown behavior. See the Myriad Cluster API.