Page History

...

Summary

As of Ambari 2.0

...

Summary

Ambari 2.0 adds support for deploying clusters with a Blueprint that utilize various Hadoop High-Availability (HA) features.  This allows for HA clusters to be deployed and setup initially.  This is slightly different than the current HA model in Ambari, where a user will create a cluster and then use the Ambari Web HA Wizards to enable HA features.  The Blueprints feature will now allow for the deployment of HA clusters without any need to use a Wizard to enable HA after the fact.  A cluster created with Blueprints HA will be configured for high-availability from the start.  

While supporting HA-enabled clusters is a new feature in Ambari Blueprints, it should be noted that no changes are required in the process to deploy a Blueprint or a cluster based on that Blueprint.  A user can configure a Blueprint to describe an HA cluster for HDFS, Yarn, and HBase RegionServers.  Support may be added for other Hadoop technologies in later releases.  

Supported HA Cluster Types

Supported HA scenarios for Blueprints in Ambari 2.0:

1. HDFS NameNode HA

2. Yarn ResourceManager HA

3. HBase RegionServer HA

Compatibility with Ambari UI

While this feature does not require the Ambari UI to function, the Blueprints HA feature is completely compatible with the Ambari UI.  An HA cluster created via Blueprints can be monitored and configured via the Ambari UI, just as any other Blueprints cluster would function.  

Expert-Mode Configuration

In Ambari 2.0, the support for HA availability is meant to handle “expert mode” configuration.  In this mode, the creator of the Blueprint sets all the configuration required for a given technology to support HA.  For example: In HDFS NameNode HA, the user must specify the nameservice name, namenode logical names, and other configuration items as well.  This gives the Blueprint creator fine-grained control of the exact HA configuration desired for a given cluster.  In future releases, we hope to also provide a higher-level mode of operations, so that HA can be enabled in a more coarse-grained way.  

Supported Stack Versions

This feature is enabled for the HDP 2.1 stack, as well as future versions of the stack.  Previous versions of HDP have not been verified for this feature, and may not function as desired.  In addition, earlier HDP versions may not include the HA support for the required technology.  

Getting Started with Blueprints HA

The simplest way to get started with setting up an HA cluster using Blueprints is to use the sample Blueprints in this documentation as a starting point.  This will likely be the most efficient way to obtain the full list of required properties for a given HA cluster, and can then be used as a starting point for customization.  While the samples provided describe smaller clusters (2-3 nodes) in order to provide simpler documentation, it should be straightforward to customize this initial Blueprint to suit larger clusters as well.  It is also possible to create an HA cluster with the Ambari UI, and export a Blueprint based on that cluster.  

Blueprint Example: HDFS NameNode HA Cluster

Summary

HDFS NameNode High-Availability allows a cluster to be configured such that a NameNode is not a single point of failure for a cluster.  In an HDFS NameNode HA cluster, two NameNodes are deployed: an “active” namenode, and a “passive” namenode.  If the active NameNode should stop functioning properly, the passive node’s Zookeeper client will detect this case, and the passive node will become the new active node.  HDFS relies on Zookeeper to manage the details of failover in these cases.  

By setting a series of properties in the “hdfs-site” configuration file, a user can configure HDFS NameNode HA to use at most two NameNodes in a cluster.  These NameNodes are typically referenced via a logical name, the “nameservice”.  These properties must be set in a Blueprint if NameNode HA is to be enabled.  

The following Apache Hadoop documentation link describes the steps required to setup HDFS NameNode HA manually:

http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html

The Blueprints HA feature will automatically invoke all required commands and setup steps for an HDFS NameNode HA cluster, provided that the correct configuration is provided in the Blueprint.  The shared edit logs of each NameNode are managed by the Quorum Journal Manager, rather than NFS shared storage.  The use of NFS shared storage in an HDFS HA setup is not supported by Ambari Blueprints, and is generally not encouraged.  

The following HDFS stack components should be included in any host group in a Blueprint that supports an HA HDFS NameNode:

1. NAMENODE

2. ZKFC

3. ZOOKEEPER_SERVER

4. JOURNALNODE

Configuring Active and Standby NameNodes

The HDFS “NAMENODE” component must be assigned to two servers, either via two separate host groups, or to a host group that maps to two physical servers in the Cluster Creation Template for this cluster.  

By default, the Blueprint processor will assign the “active” NameNode to one host, and the “standby” NameNode to another.  The user of an HA Blueprint does not need to configure the initial status of each NameNode, since this can be assigned automatically.  

If desired, the user can configure the initial state of each NameNode by adding the following configuration properties in the “hadoop-env” namespace:

1. dfs_ha_initial_namenode_active - This property should contain the hostname for the “active” NameNode in this cluster.

2. dfs_ha_initial_namenode_standby - This property should contain the host name for the “passive” NameNode in this cluster.

Example Blueprint

This is a minimal blueprint with HDFS HA: hdfs_ha_blueprint.json 

These are the base configurations required. See the blueprint above for more details:

, Blueprints are able to deploy the following components with HA:

•  HDFS NameNode HA

Blueprint Example: HDFS NameNode HA Cluster

HDFS NameNode HA allows a cluster to be configured such that a NameNode is not a single point of failure.

For more details on HDFS NameNode HA see the Apache Hadoop documentation.

In an Ambari-deployed HDFS NameNode HA cluster:

• 2 NameNodes are deployed: an “active” and a “passive” namenode.
• If the active NameNode should stop functioning properly, the passive node’s Zookeeper client will detect this case, and the passive node will become the new active node.
• HDFS relies on Zookeeper to manage the details of failover in these cases.
• The Blueprints HA feature will automatically invoke all required commands and setup steps for an HDFS NameNode HA cluster, provided that the correct configuration is provided in the Blueprint.  The shared edit logs of each NameNode are managed by the Quorum Journal Manager, rather than NFS shared storage.  The use of NFS shared storage in an HDFS HA setup is not supported by Ambari Blueprints, and is generally not encouraged.  

How

The Blueprints HA feature will automatically invoke all required commands and setup steps for an HDFS NameNode HA cluster, provided that the correct configuration is provided in the Blueprint.  The shared edit logs of each NameNode are managed by the Quorum Journal Manager, rather than NFS shared storage.  The use of NFS shared storage in an HDFS HA setup is not supported by Ambari Blueprints, and is generally not encouraged.  

The following HDFS stack components should be included in any host group in a Blueprint that supports an HA HDFS NameNode:

1. NAMENODE

2. ZKFC

3. ZOOKEEPER_SERVER

4. JOURNALNODE

Configuring Active and Standby NameNodes

The HDFS “NAMENODE” component must be assigned to two servers, either via two separate host groups, or to a host group that maps to two physical servers in the Cluster Creation Template for this cluster.  

By default, the Blueprint processor will assign the “active” NameNode to one host, and the “standby” NameNode to another.  The user of an HA Blueprint does not need to configure the initial status of each NameNode, since this can be assigned automatically.  

If desired, the user can configure the initial state of each NameNode by adding the following configuration properties in the “hadoop-env” namespace:

1. dfs_ha_initial_namenode_active - This property should contain the hostname for the “active” NameNode in this cluster.

2. dfs_ha_initial_namenode_standby - This property should contain the host name for the “passive” NameNode in this cluster.

Example Blueprint

This is a minimal blueprint with HDFS HA: hdfs_ha_blueprint.json 

These are the base configurations required. See the blueprint above for more details:

  "configurations": [
    { "core-site": {
        "properties" : {
          "fs.defaultFS" : "hdfs://mycluster",
          "ha.zookeeper.quorum" : "%HOSTGROUP::master_1%:2181,%HOSTGROUP::master_2%:2181,%HOSTGROUP::master_3%:2181"
      }}
    },
    { "hdfs-site": {
        "properties" : {
          "dfs.client.failover.proxy.provider.mycluster" : "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
          "dfs.ha.automatic-failover.enabled" : "true",
          "dfs.ha.fencing.methods" : "shell(/bin/true)",
          "dfs.ha.namenodes.mycluster" : "nn1,nn2",
          "dfs.namenode.http-address" : "%HOSTGROUP::master_1%:50070",
          "dfs.namenode.http-address.mycluster.nn1" : "%HOSTGROUP::master_1%:50070",
          "dfs.namenode.http-address.mycluster.nn2" : "%HOSTGROUP::master_3%:50070",
          "dfs.namenode.https-address" : "%HOSTGROUP::master_1%:50470",
          "dfs.namenode.https-address.mycluster.nn1" : "%HOSTGROUP::master_1%:50470",

  "configurations": [
    { "core-site": {
        "properties" : {
          "fs.defaultFS"dfs.namenode.https-address.mycluster.nn2" : "hdfs://mycluster%HOSTGROUP::master_3%:50470",
          "ha.zookeeper.quorumdfs.namenode.rpc-address.mycluster.nn1" : "%HOSTGROUP::master_1%:2181,%HOSTGROUP::master_2%:2181,8020",
          "dfs.namenode.rpc-address.mycluster.nn2" : "%HOSTGROUP::master_3%:21818020",
      }}
    },
    { "hdfs-site": {
"dfs.namenode.shared.edits.dir" : "qjournal://%HOSTGROUP::master_1%:8485;%HOSTGROUP::master_2%:8485;%HOSTGROUP::master_3%:8485/mycluster",
          "propertiesdfs.nameservices" : {"mycluster"
      }}
    "dfs.client.failover.proxy.provider.mycluster" : "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
          "dfs.ha.automatic-failover.enabled" : "true",
          "dfs.ha.fencing.methods" : "shell(/bin/true)",
          "dfs.ha.namenodes.mycluster" : "nn1,nn2",
          "dfs.namenode.http-address" : "%HOSTGROUP::master_1%:50070",
          "dfs.namenode.http-address.mycluster.nn1" : "%HOSTGROUP::master_1%:50070"}
  ]

HostName Topology Substitution in Configuration Property Values

The host-related properties should be set using the “HOSTGROUP” syntax to refer to a given Blueprint’s host group, in order to map each NameNode’s actual host (defined in the Cluster Creation Template) to the properties in hdfs-site that require these host mappings.  

The syntax for these properties should be:

“%HOSTGROUP::HOST_GROUP_NAME%:PORT””

For example, the following property from the snippet above:

"dfs.namenode.http-address.mycluster.nn1":

"%HOSTGROUP::master_1%:50070"

This property value is interpreted by the Blueprint processor to refer to the host that maps to the “master_1” host group, which should include a “NAMENODE” among its list of components.  The address property listed above should map to the host for “master_1”, at the port “50070”.  

Using this syntax is the most portable way to define host-specific properties within a Blueprint, since no direct host name references are used.  This will allow a Blueprint to be applied in a wider variety of cluster deployments, and not be tied to a specific set of hostnames.  

Register Blueprint with Ambari Server  

Post the blueprint to the "blueprint-hdfs-ha" resource to the Ambari Server.

POST /api/v1/blueprints/blueprint-hdfs-ha
 
...
[ Request Body is the example blueprint defined above ]
...
 
201 - Created

Example Cluster Creation Template

{
  "blueprint": "blueprint-hdfs-ha",
  "default_password": "changethis",
  "host_groups": [
    { "hosts": [,
        { "fqdn": "dfsc6401.namenodeambari.http-address.mycluster.nn2" : "%HOSTGROUP::master_3%:50070",apache.org" }
      ], "name": "gateway"
    },
    {  "dfs.namenode.https-address" : "%HOSTGROUP::master_1%:50470","hosts": [
        { "fqdn": "dfsc6402.namenodeambari.https-address.mycluster.nn1" : "%HOSTGROUP::master_1%:50470",apache.org" }
      ],    "dfs.namenode.https-address.mycluster.nn2" "name": "%HOSTGROUP::master_3%:504701",
    },
     { "dfs.namenode.rpc-address.mycluster.nn1" : "%HOSTGROUP::master_1%:8020",hosts": [
        { "fqdn": "dfsc6403.namenodeambari.rpc-address.mycluster.nn2" apache.org" }
      ], "name": "%HOSTGROUP::master_3%:80202",
    },
    {  "dfs.namenode.shared.edits.dir" : "qjournal://%HOSTGROUP::master_1%:8485;%HOSTGROUP::master_2%:8485;%HOSTGROUP::master_3%:8485/mycluster","hosts": [
        {  "dfs.nameservicesfqdn" : "mycluster"
c6404.ambari.apache.org" }
       }}], "name": "master_3"
    },
  ]

HostName Topology Substitution in Configuration Property Values

The host-related properties should be set using the “HOSTGROUP” syntax to refer to a given Blueprint’s host group, in order to map each NameNode’s actual host (defined in the Cluster Creation Template) to the properties in hdfs-site that require these host mappings.  

The syntax for these properties should be:

“%HOSTGROUP::HOST_GROUP_NAME%:PORT””

For example, the following property from the snippet above:

"dfs.namenode.http-address.mycluster.nn1":

"%HOSTGROUP::master_1%:50070"

This property value is interpreted by the Blueprint processor to refer to the host that maps to the “master_1” host group, which should include a “NAMENODE” among its list of components.  The address property listed above should map to the host for “master_1”, at the port “50070”.  

Using this syntax is the most portable way to define host-specific properties within a Blueprint, since no direct host name references are used.  This will allow a Blueprint to be applied in a wider variety of cluster deployments, and not be tied to a specific set of hostnames.  

Register Blueprint with Ambari Server  

Post the blueprint to the "blueprint-hdfs-ha" resource to the Ambari Server.

POST /api/v1/blueprints/blueprint-hdfs-ha
 
...
[ Request Body is the example blueprint defined above ]
...
 
201 - Created

Example Cluster Creation Template

  { "hosts": [
        { "fqdn": "c6405.ambari.apache.org" }
      ],
      "name": "slave_1"     
    }
  ]
}

Post the cluster to the Ambari Server to provision the cluster.

POST /api/v1/clusters/my-hdfs-ha-cluster
 
...
[ Request Body is above Cluster Creation Template ]
...
 
202 - Accepted
{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/my-hdfs-ha-cluster/requests/1",
  "Requests" : {
    "id" : 1,
    "status" : "InProgress"
  }
}
 
...
[ Client can then monitor the URL in the 202 response to check the status of the cluster deployment. ]
...
 

Blueprint Example: Yarn ResourceManager HA Cluster

Summary

Yarn ResourceManager High Availability (HA) adds support for deploying two Yarn ResourceManagers in a given Yarn cluster.  This support removes the single point of failure that occurs when single ResourceManager is used.  

The Yarn ResourceManager support for HA is somewhat similar to HDFS NameNode HA in that an “active/standby” architecture is adopted, with Zookeeper used to handle the failover scenarios between the two ResourceManager instances.  

The following Apache Hadoop documentation describes the steps required in order to setup Yarn ResourceManager HA manually:

http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

  The following stack components should be included in any host group in a Blueprint that supports an HA Yarn ResourceManager

1. RESOURCEMANAGER

2. ZOOKEEPER_SERVER

Initial setup of Active and Standby ResourceManagers

The Yarn ResourceManager HA feature depends upon Zookeeper in order to manage the details of  the “active” or “standby” status of a given ResourceManager.  When the two ResourceManagers are starting up initially, the first ResourceManager instance to acquire a Zookeeper lock, called a “znode”, will become the “active” ResourceManager for the cluster, with the other instance assuming the role of the “standby” ResourceManager.  

Example Blueprint

The following link includes an example Blueprint for a 3-node Yarn ResourceManager HA Cluster:

yarn_ha_blueprint.json

{
  "Blueprints": {
    "stack_name": "HDP",
    "stack_version": "2.2"
  },
  "host_groups": [
    {
      "name": "gateway",
      "cardinality" : "1",
      "components": [
        { "name": "HDFS_CLIENT" },
        { "name": "MAPREDUCE2_CLIENT" },
        { "name": "METRICS_COLLECTOR" },
        { "name": "METRICS_MONITOR" },
        { "name": "TEZ_CLIENT" },
        { "name": "YARN_CLIENT" },
        { "name": "ZOOKEEPER_CLIENT" }
      ]
    },
    {
      "name": "master_1",
      "cardinality" : "1",
      "components": [
        { "name": "HISTORYSERVER" },
        { "name": "JOURNALNODE" },
        { "name": "METRICS_MONITOR" },
        { "name": "NAMENODE" },
        { "name": "ZOOKEEPER_SERVER" }
      ]
    },
    {
      "name": "master_2",
      "cardinality" : "1",
      "components": [
        { "name": "APP_TIMELINE_SERVER" },
        { "name": "JOURNALNODE" },
        { "name": "METRICS_MONITOR" },
        { "name": "RESOURCEMANAGER" },
        { "name": "ZOOKEEPER_SERVER" }
      ]
    },

{
  "blueprint" : "blueprint-hdfs-ha",
  "default_password" : "default",
  "host_groups" :[
    {
      "name" : "hostmaster_group_13", 
      "hostscardinality" : ["1",
         "components": [
        {
          "fqdnname" : "c6401.ambari.apache.org"JOURNALNODE" },
        { "name": "METRICS_MONITOR" },
      ]
  { "name": "RESOURCEMANAGER" },
    {
     { "name" : "hostSECONDARY_group_2NAMENODE" },
  
      { "hostsname" : [ "ZOOKEEPER_SERVER" }
        ]
    },
    {
          "fqdnname" : "c6402.ambari.apache.org"
slave_1",
      "components": [
   }
      ]
   { "name": "DATANODE" },
   {
     { "name" : "hostMETRICS_group_3MONITOR" },
   
     { "name": "hostsNODEMANAGER" :}
 [     ]
    }
  ],
  "configurations": [
    {
      "core-site": {
      "fqdn" : "c6403.ambari.apache.org"
  properties" : {
      }
    "fs.defaultFS"  ]: "hdfs://%HOSTGROUP::master_1%:8020"
    } 
  ]
}

Post the cluster to the Ambari Server to provision the cluster.

POST /api/v1/clusters/my-hdfs-ha-cluster
 
...
[ Request Body is above Cluster Creation Template ]
...
 
202 - Accepted
{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/my-hdfs-ha-cluster/requests/1",
  "Requests" : {
    "id" : 1,
    "status" : "InProgress"
  }
}
 
...
[ Client can then monitor the URL in the 202 response to check the status of the cluster deployment. ]
...
 

Blueprint Example: Yarn ResourceManager HA Cluster

Summary

Yarn ResourceManager High Availability (HA) adds support for deploying two Yarn ResourceManagers in a given Yarn cluster.  This support removes the single point of failure that occurs when single ResourceManager is used.  

The Yarn ResourceManager support for HA is somewhat similar to HDFS NameNode HA in that an “active/standby” architecture is adopted, with Zookeeper used to handle the failover scenarios between the two ResourceManager instances.  

The following Apache Hadoop documentation describes the steps required in order to setup Yarn ResourceManager HA manually:

http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

  The following stack components should be included in any host group in a Blueprint that supports an HA Yarn ResourceManager

1. RESOURCEMANAGER

2. ZOOKEEPER_SERVER

Initial setup of Active and Standby ResourceManagers

The Yarn ResourceManager HA feature depends upon Zookeeper in order to manage the details of  the “active” or “standby” status of a given ResourceManager.  When the two ResourceManagers are starting up initially, the first ResourceManager instance to acquire a Zookeeper lock, called a “znode”, will become the “active” ResourceManager for the cluster, with the other instance assuming the role of the “standby” ResourceManager.  

...

The Blueprints HA feature does not support configuring the initial “active” or “standby” status of the ResourceManagers deployed in a Yarn HA cluster.  The first instance to obtain the Zookeeper lock will become the “active” node.  This allows the user to specify the host groups that contain the 2 ResourceManager instances, but also shields the user from the need to select the first “active” node.  

After the cluster has started up initially, the state of the “active” and “standby” ResourceManagers may change over time.  The initial “active” server is not guaranteed to remain the “active” node over the lifetime of the cluster.  During a failover event, the “standby” node may be required to fulfill the role of the “active” server. 

The active or standby status of a ResourceManager is not recorded or expressed when a Yarn cluster is being exported to a Blueprint, using the Blueprint REST API endpoint.  Since clusters change over time, this state is only accurate in the initial startup of the cluster. 

Example Blueprint

The following link includes an example Blueprint for a 3-node Yarn ResourceManager HA Cluster:

...

}}
    },{
      "yarn-site" : {
        "properties" : {
          "hadoop.registry.rm.enabled" : "false",
          "hadoop.registry.zk.quorum" : "%HOSTGROUP::master_3%:2181,%HOSTGROUP::master_2%:2181,%HOSTGROUP::master_1%:2181",
          "yarn.log.server.url" : "http://%HOSTGROUP::master_2%:19888/jobhistory/logs",
          "yarn.resourcemanager.address" : "%HOSTGROUP::master_2%:8050",
          "yarn.resourcemanager.admin.address" : "%HOSTGROUP::master_2%:8141",
          "yarn.resourcemanager.cluster-id" : "yarn-cluster",
          "yarn.resourcemanager.ha.automatic-failover.zk-base-path" : "/yarn-leader-election",
          "yarn.resourcemanager.ha.enabled" : "true",
          "yarn.resourcemanager.ha.rm-ids" : "rm1,rm2",
          "yarn.resourcemanager.hostname" : "%HOSTGROUP::master_2%",
          "yarn.resourcemanager.recovery.enabled" : "true",
          "yarn.resourcemanager.resource-tracker.address" : "%HOSTGROUP::master_2%:8025",
          "yarn.resourcemanager.scheduler.address" : "%HOSTGROUP::master_2%:8030",
          "yarn.resourcemanager.store.class" : "org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore",
          "yarn.resourcemanager.webapp.address" : "%HOSTGROUP::master_2%:8088",
          "yarn.resourcemanager.webapp.https.address" : "%HOSTGROUP::master_2%:8090",
          "yarn.timeline-service.address" : "%HOSTGROUP::master_2%:10200",
          "yarn.timeline-service.webapp.address" : "%HOSTGROUP::master_2%:8188",
          "yarn.timeline-service.webapp.https.address" : "%HOSTGROUP::master_2%:8190"
        }
      }
    }
  ]
}

Register Blueprint with Ambari Server

...

Example Cluster Creation Template

{
  "blueprint": "blueprint-yarn-ha",
  "default_password": "changethis",
  "configurations": [
    { "yarn-site" : {
      "blueprint" : "blueprint-yarn.resourcemanager.zk-haaddress",
  "default_password" : "default",
  "host_groups" :[: "c6402.ambari.apache.org:2181,c6403.ambari.apache.org:2181,c6404.ambari.apache.org:2181”,
    {
      "name”yarn.resourcemanager.hostname.rm1" : "host_group_1c6403.ambari.apache.org",
 
       "hostsyarn.resourcemanager.hostname.rm2" : [ "c6404.ambari.apache.org"
     }}
  ],
  "host_groups": [
    {    {"hosts": [
        {  "fqdn" : "c6401.ambari.apache.org" }
      ],  }"name": "gateway"
    },
  ]
  {  },"hosts": [
    {
    {  "namefqdn" : "host_group_2", c6402.ambari.apache.org" }
      ], "hostsname" : ["master_1"
         },
    {    {"hosts": [
        {  "fqdn" : "c6402c6403.ambari.apache.org" }
      ],  }
"name": "master_2"
    },
   ]
 {   },"hosts": [
   {
     { "namefqdn" : "host_group_3", 
c6404.ambari.apache.org" }
      ], "hostsname" : ["master_3"
         },
    {    {"hosts": [
         { "fqdn" : "c6403c6405.ambari.apache.org" }
      ],
      }
 "name": "slave_1"     ]
    } 
  ]
}

...

Blueprint Example: HBase RegionServer HA Cluster

Summary

HBase provides a High Availability feature for reads across HBase Region Servers.  

...