Page History

...

// Add following to $YARN_HOME/etc/hadoop/mapred-site.xml:

<!--This option enables dynamic port assignment by mesos -->
<property>
    <name>mapreduce.shuffle.port</name>
    <value>${myriad.mapreduce.shuffle.port}</value>
</property>

Step 5: Create

...

Hadoop Deployment Tar Files

There are two tar files uploaded to Mesos when the ResourceManager is started: (1) configuration–all of the hadoop configuration files in the $HADOOP_CONF_DIR/ and (2) binary–contains all files of the $HADOOP_HOME directory. Both of these files are placed within a dist directory, and the paths to both are specified in the myriad-default-conf.yml file.

In order to prepare and generate the configuration and binary tar files, do the following:

• Navigate to the parent directory containing the of $HADOOP_HOME directory (i.e., usually  /usr/local or /opt) and then generate the binary tar gz file as follows:
  
  

cd /opt
tar czf hadoopbinary.tgz hadoop-2.6.0

• Create a dist directory and a config subdirectory within Navigate to the $HADOOP_HOME directory, copy the contents of /etc/hadoop into the newly created dist/config directory, _CONF_DIR and generate the config tar gz file as follows:
  
  

mkdirtar dist
mkdirczf dist/config
cp /etc/hadoop/*.tgz hadoop

Step 6: Configure Resource Manager for Deployment to Mesos

Now that the binary.tgz and config.tgz files are all set, specify in the myriad-config-default.yml the relative path to each file from where the Resource Manager is to be launched.  A common pattern is to create a dist directory within $HADOOP_HOME. The configuration would be as follows:

servedConfigPath: dist/config.tgz
cdservedBinaryPath: dist
tar czf config/binary.tgz config

Starting the Resource Manager

Myriad Scheduler runs inside the Resource Manager as a plug-in. To start Start the Resource Manager within the $HADOOP_HOME directory with either of the two following commands:

./sbin/yarn-daemon.sh start resourcemanager  resourcemanager 
OR
bin/yarn resourcemanager

 Next, creat

Running Myriad Executor and Node Managers

...