Versions Compared

Old Version 8

changes.mady.by.user Bob Nettleton

Saved on

compared with

New Version 9

changes.mady.by.user Sean Roberts

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Note

These properties should only be used when the initial state of the active or standby NameNodes needs to be configured to a specific node.  This setting is only guaranteed to be accurate in the initial state of the cluster.  Over time, the active/standby state of each NameNode may change as failover events occur in the cluster. 

The active or standby status of a NameNode is not recorded or expressed when an HDFS HA Cluster is being exported to a Blueprint, using the Blueprint REST API endpoint.  Since clusters change over time, this state is only accurate in the initial startup of the cluster.  

Generally, it is assumed that most users will not need to choose the active or standby status of each NameNode, so the default behavior in Blueprints HA is to assign the status of each node automatically.  

 

Example Blueprint

The following link includes an example Blueprint for a 3-node HDFS NameNode HA Cluster: 

This is a minimal blueprint with HDFS HA: hdfs_ha_blueprint.json 

The following snippet from the example demonstrates the required properties in hdfs-site to enable HDFS NameNode HAThese are the base configurations required. See the blueprint above for more details:

Code Block
{
  "configurations" : [
    {
      "hdfscore-site" : {
        ...
        "properties" : {
         ... "fs.defaultFS" : "hdfs://mycluster",
          "ha.zookeeper.quorum" : "%HOSTGROUP::master_1%:2181,%HOSTGROUP::master_2%:2181,%HOSTGROUP::master_3%:2181"
      }}
   "dfs.client.failover.proxy.provider.mycluster" : "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
  },
    { "hdfs-site": {
         "dfs.ha.automatic-failover.enabled"properties" : "true",{
          "dfs.ha.fencing.methods" : "shell(/bin/true)"dfs.client.failover.proxy.provider.mycluster" : "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
          "dfs.ha.namenodesautomatic-failover.myclusterenabled" : "nn1,nn2true",
          "dfs.journalnodeha.editsfencing.dirmethods" : "shell(/hadoop/hdfs/journalbin/true)",
          "dfs.namenodeha.http-addressnamenodes.mycluster.nn1" : "%HOSTGROUP::host_group_1%:50070nn1,nn2",
          "dfs.namenode.http-address.mycluster.nn2" : "%HOSTGROUP::hostmaster_group_3%1%:50070",
          "dfs.namenode.httpshttp-address.mycluster.nn1" : "%HOSTGROUP::hostmaster_group_1%:5047050070",
          "dfs.namenode.httpshttp-address.mycluster.nn1nn2" : "%HOSTGROUP::hostmaster_group_1%3%:5047050070",
          "dfs.namenode.https-address.mycluster.nn2" : "%HOSTGROUP::hostmaster_group_3%1%:50470",
          "dfs.namenode.https-address.namemycluster.dirnn1" : "/hadoop/hdfs/namenode%HOSTGROUP::master_1%:50470",
          "dfs.namenode.rpchttps-address.mycluster.nn1nn2" : "%HOSTGROUP::hostmaster_group_1%3%:802050470",
          "dfs.namenode.rpc-address.mycluster.nn2nn1" : "%HOSTGROUP::hostmaster_group_3%1%:8020",
          "dfs.namenode.safemode.threshold-pctrpc-address.mycluster.nn2" : "0.99f%HOSTGROUP::master_3%:8020",
          "dfs.namenode.shared.edits.dir" : "qjournal://%HOSTGROUP::host_groupmaster_1%:8485;%HOSTGROUP::hostmaster_group_2%:8485;%HOSTGROUP::host_group_3%:8485/mycluster",
          "dfs.nameservices" : "mycluster",
          "dfs.replication" : "3%HOSTGROUP::master_3%:8485/mycluster",
          "dfs.replication.maxnameservices" : "50mycluster",
        }
      }
    }
  }
}

 

 

Note

The JSON snippet above is not a complete Blueprint, but is used to highlight the properties that are required for HDFS NameNode HA.  

...

]

HostName Topology Substitution in Configuration Property Values

...