...
Code Block |
---|
// Add following to $YARN_HOME/etc/hadoop/mapred-site.xml: <!--This option enables dynamic port assignment by mesos --> <property> <name>mapreduce.shuffle.port</name> <value>${myriad.mapreduce.shuffle.port}</value> </property> |
Step 5: Create
...
Hadoop Deployment Tar Files
There are two tar files uploaded to Mesos when the ResourceManager is started: (1) configuration–all of the hadoop configuration files in the $HADOOP_CONF_DIR/ and (2) binary–contains all files of the $HADOOP_HOME directory. Both of these files are placed within a dist directory, and the paths to both are specified in the myriad-default-conf.yml file.
In order to prepare and generate the configuration and binary tar files, do the following:
- Navigate to the parent directory containing the of $HADOOP_HOME directory (i.e., usually /usr/local or /opt) and then generate the binary tar gz file as follows:
Code Block |
---|
cd /opt tar czf hadoopbinary.tgz hadoop-2.6.0 |
- Create a dist directory and a config subdirectory within Navigate to the $HADOOP_HOME directory, copy the contents of /etc/hadoop into the newly created dist/config directory, _CONF_DIR and generate the config tar gz file as follows:
Code Block |
---|
mkdirtar dist mkdirczf dist/config cp /etc/hadoop/*.tgz hadoop |
Step 6: Configure Resource Manager for Deployment to Mesos
Now that the binary.tgz and config.tgz files are all set, specify in the myriad-config-default.yml the relative path to each file from where the Resource Manager is to be launched. A common pattern is to create a dist directory within $HADOOP_HOME. The configuration would be as follows:
Code Block |
---|
servedConfigPath: dist/config.tgz cdservedBinaryPath: dist tar czf config/binary.tgz config |
Starting the Resource Manager
Myriad Scheduler runs inside the Resource Manager as a plug-in. To start Start the Resource Manager within the $HADOOP_HOME directory with either of the two following commands:
Code Block | ||
---|---|---|
| ||
./sbin/yarn-daemon.sh start resourcemanager resourcemanager OR bin/yarn resourcemanager |
Next, creat
Running Myriad Executor and Node Managers
...