THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
The Myriad Scheduler can be configured to automatically download and run the hadoop yarn binaries and get the hadoop
configuration from the resource manager. This means you won't have to install and configure hadoop yarn on each machine.NOTE:
| Note |
|---|
This is a very new feature and the configuration options may change dramatically in the future. |
Assumptions
The following are assumptions about your environment:
...
- Running `./gradlew build`.
- Copying the Myriad Scheduler jar files in your YARN classpath.
- Placing the Myriad Executor jar files in HDFS.
- Configuring the Myriad default configuration file.
- Configuring the YARN XML file.
- Creating the tarball.
Step 1: Run
...
gradlew build
From the project root, build Myriad with the following command:
...
Step 3: Put the Myriad Executor Jar File
Put the `myriadmyriad-executor-runnable-x.x.x.jar` jar file in HDFS.```
| Code Block |
|---|
hadoop fs -put myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /dist |
...
Step 4: Configure the Myriad Configuration File
Edit `/opt/hadoop/etc/hadoop/myriad-config-default` default to configure the default parameters. To enable remote binary distribution, you must set the following options:
| HTML Comment |
|---|
XREF: For a standard configuration, see [myriad-configuration]({{site.baseurl}}/docs/myriad-configuration-properties/myriad-configuration.md). |
| Code Block |
|---|
frameworkSuperUser: admin # Must be root or have passwordless sudo on all nodes! |
...
frameworkUser: hduser # Should be the same user running the resource manager. |
...
# Must exist on all nodes and be in the 'hadoop' group |
...
executor: |
...
nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.5.0.tar.gz |
...
path: hdfs://namenode:port/dist/myriad-executor-runnable-0.0.1.jar |
...
yarnEnvironment: |
...
YARN_HOME: hadoop-2.5.0 # This should be relative if nodeManagerUri is set |
```
Step 5: Configure the YARN XML File
Configure `/opt/hadoop-2.5.0/etc/hadoop/yarn-site.xml` xml as instructed in : [myriad-configuration]({{site.baseurl}}/docs/myriad-configuration-properties/myriad-configuration.md). Myriad Configuration Properties.
Step 6: Create the Tarball
Create the tarball and place it in HDFS:```
| Code Block |
|---|
cd ~ |
...
sudo cp -rp /opt/hadoop-2.5.0 . |
...
sudo rm hadoop-2.5.0/etc/hadoop/*.xml |
...
sudo tar -zcpf ~/hadoop-2.5.0.tar.gz hadoop-2.5.0 |
...
hadoop fs -put ~/hadoop-2.5.0.tar.gz /dist |
```
You can now start the resource manager and attempt to flex up the cluster!