...
A Myriad HA environment allows the Node Manager Managers to connect reconnect to the new Resource Manager upon instance upon failover.
On failover, the following occurs:
...
Note |
---|
All clients that are connected to Resource Manager continue to work as long as the FQDN (for example, rmapprm.marathon.mesos) is used to connect to the Resource Manager. |
...
- Deploy mesos-master, mesos-slave (per node), zookeeper, marathon, and mesos-dns on your cluster.Add the mesos-dns nameserver to all cluster nodes and clients. For example, clients running RM UI, Myriad UI, and so on.
Setting up Mesos-DNS
Mesos-DNS is available on the Mesosphere GitHub. For an online version of the Mesos-DNS documents, see https://mesosphere.github.io/mesos-dns.
- Create a directory for Mesos-DNS. For example, /etc/mesos-dns.
- Install Mesos-DNS on one node in your cluster.
Configure Mesos-DNS by providing the required parameters in the /etc/mesos-dns/config.json file. See the Mesos-DNS configuration documentation for more information. The following example parameters represent a minimum configuration.
Code Block { "zk": "zk:10.10.100.19:2181/mesos", "refreshSeconds": 60, "ttl": 60, "domain": "mesos", "port": 53, "resolvers": ["10.10.1.10"], "timeout": 5, }
If you are on Linux, add the following
...
Mesos-DNS name server to the /etc/resolv.conf file
...
(at the top of the file)
...
on all cluster nodes and clients. For example, clients running RM UI, Myriad UI, and so on.
Code Block nameserver
...
<mesos-dnsIP address>
Note |
---|
Add the entries at the top (in the beginning) of the /etc/resolv.conf file. If the entries are not at the top, Mesos-DNS may not work correctly. |
Configuring HA
Configuring Myriad for HA involves adding HA configuration properties to the $YARN_HOME/etc/hadoop/yarn-site.xml file and the $YARN_HOME/etc/hadoop/myriad-config-default.yml file.
To the $YARN_HOME/etc/hadoop/yarn-site.xml file file, add the following properties:
Code Block |
---|
<!-- HA configuration properties --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle,myriad_executor</value> <!-- If using MapR distro, please use the following value <value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value> --> </property> <property> <name>yarn.nodemanager.aux-services.myriad_executor.class</name> <value>com.ebay.myriad.executor.MyriadExecutorAuxService</value> </property> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore</value> </property> <property> <name>yarn.resourcemanager.fs.state-store.uri</name> <!-- Path on HDFS, MapRFS etc --> <value>/var/mapr/cluster/yarn/rm/system</value> </property> <property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> <!-- If using MapR distro <property> <name>yarn.resourcemanager.ha.custom-ha-enabled</name> <value>false</value> </property> --> |
Launching Resource Manager
Launch the Resource Manager using Marathon. When launching, specify the yarn.resourcemanager.hostname
property. The hostname is the ID field specified when launching a Marathon application.
Code Block |
---|
env && export YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=rmapp.marathon.mesos && yarn resourcemanager |
...
To the $YARN_HOME/etc/hadoop/myriad-config-default.yml file, modify the following values:
Code Block |
---|
frameworkFailoverTimeout: <non-zero value>
haEnabled: true |
Note |
---|
The Myriad Mesos |