Versions Compared

Old Version 20

changes.mady.by.user Ruth Harris

Saved on

compared with

New Version 21

changes.mady.by.user Darin J

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Updated the steps to reflect merger of NodeManager and Executor, mostly this involves packaging the executer in the tarball and removing the path variable from myriad-config-default.ml

...

  1. Running ./gradlew build.
  2. Copying the Myriad Scheduler and files to your YARN classpath.Putting the Myriad Executor jar files in HDFS.
  3. Modifying the Myriad default configuration file (myriad-config-default.yml).
  4. Modifying the YARN XML file (yarn-site.xml).
  5. Creating the tarball.

...

Code Block
./gradlew build

 

Step 2: Copy Myriad

...

Scheduler and executor Files

Copy the jar files and configuration .yml file into your YARN classpath:

Code Block
cp myriad-scheduler/build/libs/*.jar $YARN_HOME/share/hadoop/yarn/lib/
cp myriad-scheduler/srcbuild/mainlibs/resources/myriad-config-default.ymlexecutor-0.0.1.jar $YARN_HOME/share/hadoop/yarn/lib/

 

Step 3: Put Myriad Executor Jar File

Put the myriad-executor-runnable-x.x.x.jar file in HDFS.

Code Block
hadoop fs -put
cp myriad-executorscheduler/build/libssrc/main/resources/myriad-executorconfig-runnable-0.0.1.jar /dist

...

default.yml $YARN_HOME/share/hadoop/yarn/lib/
 


Step 3: Configure the Myriad Defaults

Edit the $YARN_HOME/etc/hadoop/myriad-config-default.yml file to configure the default parameters. See the sample Myriad configuration file for more information. To enable remote binary distribution, you must set the following options:

 

Code Block
frameworkSuperUser: admin              # Must be root or have passwordless sudo on all nodes!
frameworkUser: hduser                  # Should be the same user running the resource manager.
                                       # Must exist on all nodes and be in the 'hadoop' group
executor: 
  nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.7.1.tar.gz 
path: hdfs://namenode:port/dist/myriad-executor-runnable-0.0.1.jar
yarnEnvironment: 
YARN_HOME: hadoop-2.7.1                # This should be relative if nodeManagerUri is set

 

Step

...

4: Configure YARN to use Myriad

Modify the $YARN_HOME/etc/hadoop/yarn-site.xml file as instructed in Sample: yarn-site.xml file.

Step

...

5: Create the Tarball

The tarball has all of the files needed for the Node Managers and  Resource Managers. The following shows how to create the tarball and place it in HDFS:

...