Configuring for HA

The Myriad high availability (HA) feature provides no job failure or downtime in case of failure. In addition, self recovery from a failure is provided to restore it back to a highly available state after the failure.

A Myriad HA environment allows the Node Manager to connect to the Resource Manager upon failover.

On failover, the following occurs:

• Marathon re-launches the Resource Manager as a new task.

• Mesos-DNS updates the IP address for the Resource Manager Mesos task to the new IP address.

Prerequisites

• Deploy mesos-master, mesos-slave (per node), zookeeper, marathon, mesos-dns on your cluster.

• Add the mesos-dns nameserver to all cluster nodes and clients. For example, clients running RM UI, Myriad UI, and so on.

• If you are on Linux, add the following entry to the /etc/resolv.conf file  (at the top of the file): nameserver <mesos-dnsIP address>

  Add the entries at the top (in the beginning) of the /etc/resolv.conf file. If the entries are not at the top, Mesos-DNS may not work correctly. 

Configuring HA

Configuring Myriad for HA involves adding HA configuration properties to the $YARN_HOME/etc/hadoop/yarn-site.xml file.

To the yarn-site.xml file, add the following properties:

<!--  HA configuration properties -->
<property>
	<name>yarn.resourcemanager.scheduler.class</name>
	<value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value>
</property>
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle,myriad_executor</value>
	<!-- If using MapR distro, please use the following value
	<value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value> -->
</property>
<property>
	<name>yarn.nodemanager.aux-services.myriad_executor.class</name>
	<value>com.ebay.myriad.executor.MyriadExecutorAuxService</value>
</property>
<property>
	<name>yarn.resourcemanager.store.class</name>
	<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.MyriadFileSystemRMStateStore</value>
</property>
<property>
	<name>yarn.resourcemanager.fs.state-store.uri</name>
           <!-- Path on HDFS, MapRFS etc -->
	<value>/var/mapr/cluster/yarn/rm/system</value>
</property>
<property>
	<name>yarn.resourcemanager.recovery.enabled</name>
	<value>true</value>
</property>
<!-- If using MapR distro
 <property>
	<name>yarn.resourcemanager.ha.custom-ha-enabled</name>
	<value>false</value>
 </property> -->

Launching Resource Manager

Launch the Resource Manager using Marathon. When launching, specify the yarn.resourcemanager.hostname property. The hostname is the ID field specified when launching a Marathon application.

env && export YARN_RESOURCEMANAGER_OPTS=-Dyarn.resourcemanager.hostname=rmapp.marathon.mesos && yarn resourcemanager