THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
The Myriad environment is established by running an initial application for the resource manager. This initial application name is specified in the yarn-site.xml file with the the yarn.resourcemanager.hostname property and the value, <app-ID>.<framework>.mesos.
For example, if the value of yarn.resourcemanager.hostname property is rm.marathon.mesos:
rm - The ID given to the Resource Manager when launched using Marathon. Mesos-DNS constructs the Resource Manager hostname using the ID.
marathon - The Mesos framework.
If you are using Marathon, launch Marathon and run an initial Resource Manager application. The Resource Manager can be launched or stopped from either the command line or the Marathon UI.
Launching from the Command Line
To start the Resource Manager, run the YARN daemon from the command line:
yarn-daemon.sh start resourcemanager
To shut down he Resource Manager, run the YARN daemon from the command line:
yarn-daemon.sh stop resourcemanager
Launching from Marathon
Alternatively, start and stop Myriad from the Marathon UI. See Marathon: Application Basics for more information. For example, create an application to start the Resource Manager:
cd hadoop-2.7.0/sbin && yarn-daemon.sh start resourcemanager
Alternatively, when launching the Resource Manager in an HA environment, specify value for the yarn.resourcemanager.hostname property. The hostname is the ID field specified when launching a Marathon application.
To initially launch the Resource Manager from Marathon:
Launch Marathon with <host>:8080. For example: http://10.10.100.16:8080
Click on New App.
Create a new application for the resource manager and specify:
ID
CPU
Memory
Instances
Command.
For example
| Parameter | Example Value | Description |
|---|---|---|
ID | rm | The ID for the Resource Manager when launched using Marathon. Mesos-DNS constructs the Resource Manager hostname using the ID. |
CPU | 0.2 | Amount of CPU allocated. |
Memory | 2048 | Amount of memory allocated |
Instances | 1 | Number of instances to be launched. |
Command | env && yarn resourcemanager | Command to launch the resource manager. |
If the yarn.resourcemanager.hostname property is not specified in the yarn-site.xml file, then the -DYARN_RESOURCEMANAGER_OPTS option must be specified. When the -Dyarn.resourcemanager.hostname property is specified when launching Resource Manager, Myriad propagates this value to the Node Managers launched via the flexup API. The Node Managers can discover back the Resource Manager using the hostname specified in -Dyarn.resourcemanager.hostname. If that hostname happens to be a Mesos DNS hostname, then when the Resource Manager moves to another node during failover, Node Managerss can still connect to the new Resource Manager using the same hostname. Marathon helps move the Resource Manager to a different node whenever it detects the failure of Resource Manager.
For example
env && export YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=rm.marathon.mesos && yarn resourcemanager
Some applications might require the yarn.resourcemanager.hostname property to be explicitly specified as a command line option.
1 Comment
Santosh Marella
We should perhaps describe more about how HA should be configured using Mesos-DNS and -Dyarn.resourcemanager.hostname property. Essentially, when -Dyarn.resourcemanager.hostname property is set when launching RM, myriad propagates this value to the Node Managers launched via the flexup API. The NMs can discover back the RM using the hostname specified in
Dyarn.resourcemanager.hostname. If that hostname happens to be a Mesos DNS hostname, then when RM moves to another node during failover, NMs can still connect to the new RM using the same hostname. Marathon helps move the RM to a different node whenever it detects RM's failure.Also, it might be easier to indicate using a screenshot of Marathon's UI that the ID used to launch the RM should be used as a prefix for the -Dyarn.resourcemanager.hostname value. For e.g. if the ID used was "myyarncluster", then the value for -Dyarn.resourcemanager.hostname would have to be "myyarncluster.marathon.mesos" and Mesos DNS would have to be configured for ".mesos" domain.