THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Support for deploying HA Clusters with Blueprints has been added in Ambari 2.0.
Summary
Ambari 2.0 adds support for deploying clusters with a Blueprint that utilize various Hadoop High-Availability (HA) features. This allows for HA clusters to be deployed and setup initially. This is slightly different than the current HA model in Ambari, where a user will create a cluster and then use the Ambari Web HA Wizards to enable HA features. The Blueprints feature will now allow for the deployment of HA clusters without any need to use a Wizard to enable HA after the fact. A cluster created with Blueprints HA will be configured for high-availability from the start.
While supporting HA-enabled clusters is a new feature in Ambari Blueprints, it should be noted that no changes are required in the process to deploy a Blueprint or a cluster based on that Blueprint. A user can configure a Blueprint to describe an HA cluster for HDFS, Yarn, and HBase RegionServers. Support may be added for other Hadoop technologies in later releases.
Supported HA Cluster Types
Supported HA scenarios for Blueprints in Ambari 2.0:
HDFS NameNode HA
Yarn ResourceManager HA
HBase RegionServer HA
Compatibility with Ambari UI
While this feature does not require the Ambari UI to function, the Blueprints HA feature is completely compatible with the Ambari UI. An HA cluster created via Blueprints can be monitored and configured via the Ambari UI, just as any other Blueprints cluster would function.
Expert-Mode Configuration
In Ambari 2.0, the support for HA availability is meant to handle “expert mode” configuration. In this mode, the creator of the Blueprint sets all the configuration required for a given technology to support HA. For example: In HDFS NameNode HA, the user must specify the nameservice name, namenode logical names, and other configuration items as well. This gives the Blueprint creator fine-grained control of the exact HA configuration desired for a given cluster. In future releases, we hope to also provide a higher-level mode of operations, so that HA can be enabled in a more coarse-grained way.
Supported Stack Versions
This feature is enabled for the HDP 2.1 stack, as well as future versions of the stack. Previous versions of HDP have not been verified for this feature, and may not function as desired. In addition, earlier HDP versions may not include the HA support for the required technology.
Getting Started with Blueprints HA
The simplest way to get started with setting up an HA cluster using Blueprints is to use the sample Blueprints in this documentation as a starting point. This will likely be the most efficient way to obtain the full list of required properties for a given HA cluster, and can then be used as a starting point for customization. While the samples provided describe smaller clusters (2-3 nodes) in order to provide simpler documentation, it should be straightforward to customize this initial Blueprint to suit larger clusters as well. It is also possible to create an HA cluster with the Ambari UI, and export a Blueprint based on that cluster.
Blueprint Example: HDFS NameNode HA Cluster
Summary
HDFS NameNode High-Availability allows a cluster to be configured such that a NameNode is not a single point of failure for a cluster. In an HDFS NameNode HA cluster, two NameNodes are deployed: an “active” namenode, and a “passive” namenode. If the active NameNode should stop functioning properly, the passive node’s Zookeeper client will detect this case, and the passive node will become the new active node. HDFS relies on Zookeeper to manage the details of failover in these cases.
By setting a series of properties in the “hdfs-site” configuration file, a user can configure HDFS NameNode HA to use at most two NameNodes in a cluster. These NameNodes are typically referenced via a logical name, the “nameservice”. These properties must be set in a Blueprint if NameNode HA is to be enabled.
The following Apache Hadoop documentation link describes the steps required to setup HDFS NameNode HA manually:
http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html
Ambari Blueprints will handle much of the details of server setup listed in this documentation, but the user of Ambari will need to define the various configuration properties listed in this article (dfs.nameservices, dfs.ha.namenodes.$NAME_OF_NAMENODE, etc). The example Blueprints listed below will demonstrate the configuration properties that must be included in the Blueprint for this feature to startup the HA cluster properly.
The Blueprints HA feature will automatically invoke all required commands and setup steps for an HDFS NameNode HA cluster, provided that the correct configuration is provided in the Blueprint. The shared edit logs of each NameNode are managed by the Quorum Journal Manager, rather than NFS shared storage. The use of NFS shared storage in an HDFS HA setup is not supported by Ambari Blueprints, and is generally not encouraged.
The following HDFS stack components should be included in any host group in a Blueprint that supports an HA HDFS NameNode:
NAMENODE
ZKFC
ZOOKEEPER_SERVER
JOURNALNODE
Configuring Active and Standby NameNodes
The HDFS “NAMENODE” component must be assigned to two servers, either via two separate host groups, or to a host group that maps to two physical servers in the Cluster Creation Template for this cluster.
By default, the Blueprint processor will assign the “active” NameNode to one host, and the “standby” NameNode to another. The user of an HA Blueprint does not need to configure the initial status of each NameNode, since this can be assigned automatically.
If desired, the user can configure the initial state of each NameNode by adding the following configuration properties in the “hadoop-env” namespace:
dfs_ha_initial_namenode_active - This property should contain the hostname for the “active” NameNode in this cluster.
dfs_ha_initial_namenode_standby - This property should contain the host name for the “passive” NameNode in this cluster.
These properties should only be used when the initial state of the active or standby NameNodes needs to be configured to a specific node. This setting is only guaranteed to be accurate in the initial state of the cluster. Over time, the active/standby state of each NameNode may change as failover events occur in the cluster.
The active or standby status of a NameNode is not recorded or expressed when an HDFS HA Cluster is being exported to a Blueprint, using the Blueprint REST API endpoint. Since clusters change over time, this state is only accurate in the initial startup of the cluster.
Generally, it is assumed that most users will not need to choose the active or standby status of each NameNode, so the default behavior in Blueprints HA is to assign the status of each node automatically.
Example Blueprint
The following link includes an example Blueprint for a 3-node HDFS NameNode HA Cluster:
The following snippet from the example demonstrates the required properties in hdfs-site to enable HDFS NameNode HA:
{
"configurations" : [
{
"hdfs-site" : {
...
"properties" : {
...
"dfs.client.failover.proxy.provider.mycluster" : "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.ha.automatic-failover.enabled" : "true",
"dfs.ha.fencing.methods" : "shell(/bin/true)",
"dfs.ha.namenodes.mycluster" : "nn1,nn2",
"dfs.journalnode.edits.dir" : "/hadoop/hdfs/journal",
"dfs.namenode.http-address.mycluster.nn1" : "%HOSTGROUP::host_group_1%:50070",
"dfs.namenode.http-address.mycluster.nn2" : "%HOSTGROUP::host_group_3%:50070",
"dfs.namenode.https-address" : "%HOSTGROUP::host_group_1%:50470",
"dfs.namenode.https-address.mycluster.nn1" : "%HOSTGROUP::host_group_1%:50470",
"dfs.namenode.https-address.mycluster.nn2" : "%HOSTGROUP::host_group_3%:50470",
"dfs.namenode.name.dir" : "/hadoop/hdfs/namenode",
"dfs.namenode.rpc-address.mycluster.nn1" : "%HOSTGROUP::host_group_1%:8020",
"dfs.namenode.rpc-address.mycluster.nn2" : "%HOSTGROUP::host_group_3%:8020",
"dfs.namenode.safemode.threshold-pct" : "0.99f",
"dfs.namenode.shared.edits.dir" : "qjournal://%HOSTGROUP::host_group_1%:8485;%HOSTGROUP::host_group_2%:8485;%HOSTGROUP::host_group_3%:8485/mycluster",
"dfs.nameservices" : "mycluster",
"dfs.replication" : "3",
"dfs.replication.max" : "50",
}
}
}
}
}
The JSON snippet above is not a complete Blueprint, but is used to highlight the properties that are required for HDFS NameNode HA.
HostName Topology Substitution in Configuration Property Values
The host-related properties should be set using the “HOSTGROUP” syntax to refer to a given Blueprint’s host group, in order to map each NameNode’s actual host (defined in the Cluster Creation Template) to the properties in hdfs-site that require these host mappings.
The syntax for these properties should be:
“%HOSTGROUP::HOST_GROUP_NAME%:PORT””
For example, the following property from the snippet above:
"dfs.namenode.http-address.mycluster.nn1":
"%HOSTGROUP::host_group_1%:50070"
This property value is interpreted by the Blueprint processor to refer to the host that maps to the “host_group_1” host group, which should include a “NAMENODE” among its list of components. The address property listed above should map to the host for “host_group_1”, at the port “50070”.
Using this syntax is the most portable way to define host-specific properties within a Blueprint, since no direct host name references are used. This will allow a Blueprint to be applied in a wider variety of cluster deployments, and not be tied to a specific set of hostnames.
Please refer to the link below for the full Blueprint example for HDFS NameNode HA:
Register Blueprint with Ambari Server
Post the blueprint to the "blueprint-hdfs-ha" resource to the Ambari Server.
POST /api/v1/blueprints/blueprint-hdfs-ha ... [ Request Body is the example blueprint defined above ] ... 201 - Created
Example Cluster Creation Template
{
"blueprint" : "blueprint-hdfs-ha",
"default_password" : "default",
"host_groups" :[
{
"name" : "host_group_1",
"hosts" : [
{
"fqdn" : "c6401.ambari.apache.org"
}
]
},
{
"name" : "host_group_2",
"hosts" : [
{
"fqdn" : "c6402.ambari.apache.org"
}
]
},
{
"name" : "host_group_3",
"hosts" : [
{
"fqdn" : "c6403.ambari.apache.org"
}
]
}
]
}
Create Cluster Instance
Post the cluster to the Ambari Server to provision the cluster.
POST /api/v1/clusters/my-hdfs-ha-cluster
...
[ Request Body is above Cluster Creation Template ]
...
202 - Accepted
{
"href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/my-hdfs-ha-cluster/requests/1",
"Requests" : {
"id" : 1,
"status" : "InProgress"
}
}
...
[ Client can then monitor the URL in the 202 response to check the status of the cluster deployment. ]
...
Blueprint Example: Yarn ResourceManager HA Cluster
Summary
Yarn ResourceManager High Availability (HA) adds support for deploying two Yarn ResourceManagers in a given Yarn cluster. This support removes the single point of failure that occurs when single ResourceManager is used.
The Yarn ResourceManager support for HA is somewhat similar to HDFS NameNode HA in that an “active/standby” architecture is adopted, with Zookeeper used to handle the failover scenarios between the two ResourceManager instances.
The following Apache Hadoop documentation describes the steps required in order to setup Yarn ResourceManager HA manually:
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html
Ambari Blueprints will handle much of the details of server setup listed in this documentation, but the user of Ambari will need to define the various configuration properties listed in this article (yarn.resourcemanager.ha.enabled, yarn.resourcemanager.hostname.$NAME_OF_RESOURCE_MANAGER, etc). The example Blueprints listed below will demonstrate the configuration properties that must be included in the Blueprint for this feature to startup the HA cluster properly.
The following stack components should be included in any host group in a Blueprint that supports an HA Yarn ResourceManager
RESOURCEMANAGER
ZOOKEEPER_SERVER
Initial setup of Active and Standby ResourceManagers
The Yarn ResourceManager HA feature depends upon Zookeeper in order to manage the details of the “active” or “standby” status of a given ResourceManager. When the two ResourceManagers are starting up initially, the first ResourceManager instance to acquire a Zookeeper lock, called a “znode”, will become the “active” ResourceManager for the cluster, with the other instance assuming the role of the “standby” ResourceManager.
The Blueprints HA feature does not support configuring the initial “active” or “standby” status of the ResourceManagers deployed in a Yarn HA cluster. The first instance to obtain the Zookeeper lock will become the “active” node. This allows the user to specify the host groups that contain the 2 ResourceManager instances, but also shields the user from the need to select the first “active” node.
After the cluster has started up initially, the state of the “active” and “standby” ResourceManagers may change over time. The initial “active” server is not guaranteed to remain the “active” node over the lifetime of the cluster. During a failover event, the “standby” node may be required to fulfill the role of the “active” server.
The active or standby status of a ResourceManager is not recorded or expressed when a Yarn cluster is being exported to a Blueprint, using the Blueprint REST API endpoint. Since clusters change over time, this state is only accurate in the initial startup of the cluster.
Example Blueprint
The following link includes an example Blueprint for a 3-node Yarn ResourceManager HA Cluster:
Register Blueprint with Ambari Server
Post the blueprint to the "blueprint-yarn-ha" resource to the Ambari Server.
POST /api/v1/blueprints/blueprint-yarn-ha ... [ Request Body is the example blueprint defined above ] ... 201 - Created
Example Cluster Creation Template
{
"blueprint" : "blueprint-yarn-ha",
"default_password" : "default",
"host_groups" :[
{
"name" : "host_group_1",
"hosts" : [
{
"fqdn" : "c6401.ambari.apache.org"
}
]
},
{
"name" : "host_group_2",
"hosts" : [
{
"fqdn" : "c6402.ambari.apache.org"
}
]
},
{
"name" : "host_group_3",
"hosts" : [
{
"fqdn" : "c6403.ambari.apache.org"
}
]
}
]
}
Create Cluster Instance
Post the cluster to the Ambari Server to provision the cluster.
POST /api/v1/clusters/my-yarn-ha-cluster
...
[ Request Body is above Cluster Creation Template ]
...
202 - Accepted
{
"href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/my-yarn-ha-cluster/requests/1",
"Requests" : {
"id" : 1,
"status" : "InProgress"
}
}
...
[ Client can then monitor the URL in the 202 response to check the status of the cluster deployment. ]
...
Blueprint Example: HBase RegionServer HA Cluster
Summary
HBase provides a High Availability feature for reads across HBase Region Servers.
The following link to the Apache HBase documentation provides more information on HA support in HBase:
http://hbase.apache.org/book.html#arch.timelineconsistent.reads
The documentation listed here explains how to deploy an HBase RegionServer HA cluster via a Blueprint, but there are separate application-specific steps that must be taken in order to enable this feature for a specific table in HBase. A table must be created with replication enabled, so that multiple Region Servers can handle the keys for this table.
For more information on how to define an HBase table with replication enabled (after the cluster has been created), please refer to the following HBase documentation:
http://hbase.apache.org/book.html#_creating_a_table_with_region_replication
The following stack components should be included in any host group in a Blueprint that supports an HA HBase RegionServer:
HBASE_REGIONSERVER
At least two “HBASE_REGIONSERVER” components must be deployed in order to enable this feature, so that table information can be replicated across more than one Region Server.
Example Blueprint
The following link includes an example Blueprint for a 2-node HBase RegionServer HA Cluster:
The following JSON snippet includes the “hbase-site” configuration typically required for a cluster that utilizes the HBase RegionServer HA feature:
{
"configurations" : [
{
"hbase-site" : {
...
"hbase.regionserver.global.memstore.lowerLimit" : "0.38",
"hbase.regionserver.global.memstore.upperLimit" : "0.4",
"hbase.regionserver.handler.count" : "60",
"hbase.regionserver.info.port" : "60030",
"hbase.regionserver.storefile.refresh.period" : "20",
"hbase.rootdir" : "hdfs://%HOSTGROUP::host_group_1%:8020/apps/hbase/data",
"hbase.security.authentication" : "simple",
"hbase.security.authorization" : "false",
"hbase.superuser" : "hbase",
"hbase.tmp.dir" : "/hadoop/hbase",
"hbase.zookeeper.property.clientPort" : "2181",
"hbase.zookeeper.quorum" : "%HOSTGROUP::host_group_1%,%HOSTGROUP::host_group_2%",
"hbase.zookeeper.useMulti" : "true",
"hfile.block.cache.size" : "0.40",
"zookeeper.session.timeout" : "30000",
"zookeeper.znode.parent" : "/hbase-unsecure"
}
}
]
}
The JSON example above is not a complete set of “hbase-site” configurations, but rather shows the configuration settings that are relevant to HBase RegionServer HA. In particular, the “hbase.regionserver.storefile.refresh.period” setting is the most relevant to HBase RegionServer HA, since this property must be set to a value greater than zero in order for the HA feature to be enabled.
Register Blueprint with Ambari Server
Post the blueprint to the "blueprint-hbase-rs-ha" resource to the Ambari Server.
POST /api/v1/blueprints/blueprint-hbase-rs-ha ... [ Request Body is the example blueprint defined above ] ... 201 - Created
Example Cluster Creation Template
{
"blueprint" : "blueprint-hbase-rs-ha",
"default_password" : "default",
"host_groups" :[
{
"name" : "host_group_1",
"hosts" : [
{
"fqdn" : "c6401.ambari.apache.org"
}
]
},
{
"name" : "host_group_2",
"hosts" : [
{
"fqdn" : "c6402.ambari.apache.org"
}
]
}
]
}
Create Cluster Instance
Post the cluster to the Ambari Server to provision the cluster.
POST /api/v1/clusters/my-hbase-rs-ha-cluster
...
[ Request Body is above Cluster Creation Template ]
...
202 - Accepted
{
"href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/my-hbase-rs-ha-cluster/requests/1",
"Requests" : {
"id" : 1,
"status" : "InProgress"
}
}
...
[ Client can then monitor the URL in the 202 response to check the status of the cluster deployment. ]
...