The Myriad high availability (HA) feature provides no job failure or downtime in case of failure. In addition, self recovery from a failure is provided to restore it back to a highly available state after the failure.
A Myriad HA environment allows the Node Managers to reconnect to the new Resource Manager instance upon failover.
On failover, the following occurs:
- Marathon re-launches the Resource Manager as a new task.
- Mesos-DNS updates the IP address for the Resource Manager Mesos task to the new IP address.
All clients that are connected to Resource Manager continue to work as long as the FQDN (for example, rmapp.marathon.mesos) is used to connect to the Resource Manager.
Prerequisites
- Deploy mesos-master, mesos-slave (per node), zookeeper, marathon, mesos-dns on your cluster.
- Add the mesos-dns nameserver to all cluster nodes and clients. For example, clients running RM UI, Myriad UI, and so on.
If you are on Linux, add the following entry to the /etc/resolv.conf file (at the top of the file):
nameserver <mesos-dnsIP address>
Add the entries at the top (in the beginning) of the /etc/resolv.conf file. If the entries are not at the top, Mesos-DNS may not work correctly.
Configuring HA
Configuring Myriad for HA involves adding HA configuration properties to the $YARN_HOME/etc/hadoop/yarn-site.xml file.
To the yarn-site.xml file, add the following properties:
<!-- HA configuration properties --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle,myriad_executor</value> <!-- If using MapR distro, please use the following value <value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value> --> </property> <property> <name>yarn.nodemanager.aux-services.myriad_executor.class</name> <value>com.ebay.myriad.executor.MyriadExecutorAuxService</value> </property> <property> <name>yarn.resourcemanager.store.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore</value> </property> <property> <name>yarn.resourcemanager.fs.state-store.uri</name> <!-- Path on HDFS, MapRFS etc --> <value>/var/mapr/cluster/yarn/rm/system</value> </property> <property> <name>yarn.resourcemanager.recovery.enabled</name> <value>true</value> </property> <!-- If using MapR distro <property> <name>yarn.resourcemanager.ha.custom-ha-enabled</name> <value>false</value> </property> -->
Launching Resource Manager
Launch the Resource Manager using Marathon. When launching, specify the yarn.resourcemanager.hostname
property. The hostname is the ID field specified when launching a Marathon application.
env && export YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=rmapp.marathon.mesos && yarn resourcemanager
Some applications might require the yarn.resourcemanager.hostname
property to be specified explicitly as a command line option.