Blueprints

Ambari Blueprints are a declarative definition of a cluster. With a Blueprint, you specify a Stack, the Component layout and the Configurations to materialize a Hadoop cluster instance (via a REST API) without having to use the Ambari Cluster Install Wizard.

Notable JIRAs

Basic API Resources

Blueprint Usage Overview

Step 0: Prepare Ambari Server and Agents 

Install the Ambari Server, run setup and start. Install the Ambari Agents on all hosts and perform manual registration.

Step 1: Create Blueprint

A blueprint can be created by hand or can be created by exporting a blueprint from an existing cluster.

Step 2: Register Blueprint with Ambari

POST /api/v1/blueprints/:blueprintName

Request body is blueprint created in Step 1.

To disable topology validation and register a blueprint:

POST /api/v1/blueprints/:blueprintName?validate_topology=false

Disabling topology validation allows a user to force registration of a blueprint that fails topology validation.

Step 3: Create Cluster Creation Template

Map Physical Hosts to Blueprint: Create the mapping between blueprint host groups and physical hosts.

Provide Cluster Specific Configuration Overrides : Configuration can be applied at the cluster and host group scope and overrides any configurations specified in the blueprint.

Step 4: Create Cluster

POST /api/v1/clusters/:clusterName

Request body includes blueprint name, host mappings and configurations from Step 3.

Request is asynchronous and returns a /requests URL which can be used to monitor progress.

Step 5: Monitor Cluster Creation Progress

Using the /requests URL returned in Step 4, monitor the progress of the tasks associated with cluster creation.

Blueprint Details

Prepare Ambari Server and Agents

1. Perform your Ambari Server install and setup.
   yum install ambari-server
ambari-server setup
   
   
2. After setup completes, start your Ambari Server.
   ambari-server start
   
   
3. Install Ambari Agents on all of the hosts you plan to include in your cluster.
   yum install ambari-agent
   
   
4. Set the Ambari Server on the Ambari Agents.
   vi /etc/ambari-agent/conf/ambari-agent.ini
   
   
5. Set hostname= to the Fully Qualified Domain Name for the Ambari Server. Save and exit.
   hostname=c6401.ambari.apache.org
   
   
6. Start the Agents to initiate registration to Server.
   ambari-agent start
   
   
7. Confirm the Agent hosts are registered with the Server.
   http://your.ambari.server:8080/api/v1/hosts

Blueprint Structure

A blueprint document is in JSON format and has the following structure:

{
  "configurations" : [
    {
      "configuration-type" : {
          "property-name"  : "property-value",
          "property-name2" : "property-value"
      }
    },
    {
      "configuration-type2" : {
          "property-name" : "property-value"
      }
    }
    ...
  ],
  "host_groups" : [
    {
      "name" : "host-group-name",
      "components" : [
        {
          "name" : "component-name"
        },
        {
          "name" : "component-name2"
        }
        ...
      ],
      "configurations" : [
	    {
          "configuration-type" : {
            "property-name" : "property-value"
          }
        }
        ...
      ],
      "cardinality" : "1"
    }
  ],
  "Blueprints" : {
    "stack_name" : "HDP",
    "stack_version" : "2.1"
  }
}

Blueprint Field Descriptions

• configurations : A list of configuration maps keyed by configuration type. An example of a configuration type is "core-site".  When specified at the top level, configurations are applied at cluster scope and override default values for the cluster.  When specified within a "host_groups" element, configurations are applied at the host level for all hosts mapped to the host group.  Host scoped configurations override cluster scoped configuration for all hosts mapped to the host group.  The configurations element is optional at both levels.
• host_groups : A list of host groups which define topology (components) and configuration for all hosts which are mapped to the host group.  At least one host group must be specified.
  • name : The name of the host group.  Mandatory field which is referred to in the cluster creation body when mapping physical hosts to host groups.
  • components : A list of components which will run on all hosts that are mapped to the host group.  At least one component must be specified for each host group.
  • cardinality : Optional field which may be referred to by the cluster deployer when deciding how many hosts to map to the host group.  This field has no affect during deployment and is only meta-data for the deployer.  When a blueprint is exported for an existing cluster, this field will indicate the number of hosts that correspond the the host group in that cluster.
• Blueprints : Blueprint and stack information
  • stack_name : The name of the stack.  All stacks currently shipped with Ambari have the name "HDP".  This is a required field.
  • stack_version : The version of the stack.  For example: "1.3.2" or "2.1".  This is a required field.  When deploying a cluster using a blueprint, the stack definition identified in the blueprint must be available to the Ambari instance in the new cluster.
  • blueprint_name : Optional field which specifies that name of the blueprint.  Typically the name of the blueprint is specified in the URL of the REST invocation.  The only reason to specify the name in the blueprint is when creating multiple blueprints via a single REST invocation.

Cluster Creation Template Structure

A Cluster Creation Template is in JSON format and has the following structure:

{
  "blueprint" : "blueprint-name",
  "default_password" : "super-secret-password",
  "configurations" : [
    {
      "configuration-type" : {
        "property-name" : "property-value"
      }
    }
    ...
  ],     
  "host_groups" :[
    { 
      "name" : "host-group-name",  
      "configurations" : [
        {
          "configuration-type" : {
            "property-name" : "property-value"
          }
        }
      ],     
      "hosts" : [          
        { 
          "fqdn" : "host.domain.com"
        },
        { 
          "fqdn" : "host2.domain.com"
        }
        ...
      ]
    }
    ...
  ]
}

 Cluster Creation Template Structure: Host Mappings and Configuration Field Descriptions

• blueprint : Name of the blueprint that defines the cluster to be deployed.  Blueprint must already exist.  Required field.
• default_password : Optional field which specifies a default password for all required passwords which are not specified in the blueprint or cluster creation template configurations.
• configurations : A list of configuration maps keyed by configuration type. An example of a configuration type is "core-site".  When specified at the top level, configurations are applied at cluster scope and override default values for the cluster.  When specified within a "host_groups" element, configurations are applied at the host level for all hosts mapped to the host group.  Host scoped configurations override cluster scoped configuration for all hosts mapped to the host group.  All cluster scoped and host group scoped configurations specified here override configurations specified in the corresponding blueprint.  The configurations element is optional at both levels.  
• host_groups : A list of host groups being deployed to the cluster.  At least one host group must be specified.
  • name : Required field which must correspond to a name of a host group in the associated blueprint.
  • hosts : List of host mapping information
    • fqdn : Fully qualified domain name for each host that is being mapped to the host group.  At least one host must be specified

Configurations

Default Values and Overrides

• Stack Defaults: Each Stack provides configurations for all included services which serve as defaults for all clusters deployed via Blueprints. 
• Blueprint Cluster Scoped: Configurations provided at the top level of a Blueprint override the corresponding default values for the entire cluster.
• Blueprint Host Group Scoped: Configurations provided within a host_group element of a Blueprint override both the corresponding default values and blueprint cluster scoped values only for hosts mapped to the host group.
• Cluster Creation Template Cluster Scoped: Configurations provided at the top level of the Cluster Creation Template override both the corresponding default and blueprint cluster scoped values for the entire cluster.
• Cluster Creation Template Host Group Scoped: Configurations provided within a host_group element of a Cluster Creation Template override all other values for hosts mapped to the host group.

Required Configurations

• Not all configuration properties have valid defaults
• Required properties must be specified by the Blueprint user
• Come in two categories, passwords and non passwords
• Non password required properties are validated at Blueprint creation time
• Required password properties are validated at cluster creation time
• For required password properties, they may be explicitly set in either the Blueprint or Cluster Creation Template configurations or a default password may be specified in the Cluster Creation Template which will be applied to all passwords which have not been explicitly set
  • "default_password" : "super-secret-password"
• If required configuration validation fails, a 400 response is returned indicating which properties must be specified

Blueprint Example: Single-Node HDP 2.0 Cluster

• Single-node cluster (c6401.ambari.apache.org)
• HDP 2.0 Stack
• Install Core Hadoop Services (HDFS, YARN, MapReduce2, ZooKeeper) + Ganglia (metrics) and Nagios (alerts)

Example Blueprint

{
  "configurations" : [
    {
      "global" : {
        "nagios_contact": "me@my-awesome-domain.example"
      }
    }
  ],
  "host_groups" : [
    {
      "name" : "host_group_1",      
      "components" : [
        {
          "name" : "NAMENODE"
        },
        {
          "name" : "SECONDARY_NAMENODE"
        },        
        {
          "name" : "DATANODE"
        },
        {
          "name" : "HDFS_CLIENT"
        },
        {
          "name" : "RESOURCEMANAGER"
        },
        {
          "name" : "NODEMANAGER"
        },
        {
          "name" : "YARN_CLIENT"
        },
        {
          "name" : "HISTORYSERVER"
        },
        {
          "name" : "MAPREDUCE2_CLIENT"
        },
        {
          "name" : "ZOOKEEPER_SERVER"
        },
        {
          "name" : "ZOOKEEPER_CLIENT"
        },
        {
          "name" : "GANGLIA_SERVER"
        },
        {
          "name" : "NAGIOS_SERVER"
        }
      ],
      "cardinality" : "1"
    }
  ],
  "Blueprints" : {
    "stack_name" : "HDP",
    "stack_version" : "2.0"
  }
}

Register blueprint with Ambari Server

Post the blueprint to the "single-node-hdfs-yarn" resource to the Ambari Server.

POST /api/v1/blueprints/single-node-hdfs-yarn
 
... 
[ Request Body is the example blueprint defined above ]
...

201 - Created

Example Cluster Creation Template

We are performing a single-node install and the blueprint above has one host group. Therefore, for our cluster instance, we define one host in host_group_1 and reference the single-node-hdfs-yarn blueprint.

{
  "blueprint" : "single-node-hdfs-yarn",
  "default_password" : "my-super-secret-password",
  "host_groups" :[
    { 
      "name" : "host_group_1",  
      "hosts" : [          
        { 
          "fqdn" : "c6401.ambari.apache.org"
        }
      ]
    }
  ]
}

Create Cluster Instance

Post the cluster to the Ambari Server to provision the cluster.

POST /api/v1/clusters/MySingleNodeCluster

...
[ Request Body is above Cluster Creation Template ]
...

202 - Accepted
{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/MyCluster/requests/1",
  "Requests" : {
    "id" : 1,
    "status" : "InProgress"
  }
}

Blueprint Example: Multi-Node HDP 2.0 Cluster

Example Blueprint

The blueprint ("multi-node-hdfs-yarn") below defines with two host groups (a "master" and the "slaves") which hosts the various Service components (masters, slaves and clients).

{
  "host_groups" : [
    {
      "name" : "master",
      "configurations" : [
        {
          "global" : {
            "nagios_contact": "me@my-awesome-domain.example"
          }
        }
      ],
      "components" : [
        {
          "name" : "NAMENODE"
        },
        {
          "name" : "SECONDARY_NAMENODE"
        },        
        {
          "name" : "RESOURCEMANAGER"
        },
        {
          "name" : "HISTORYSERVER"
        },
        {
          "name" : "NAGIOS_SERVER"
        },
        {
          "name" : "GANGLIA_SERVER"
        },
        {
          "name" : "ZOOKEEPER_SERVER"
        }
      ],
      "cardinality" : "1"
    },
    {
      "name" : "slaves",
      "components" : [
        {
          "name" : "DATANODE"
        },
        {
          "name" : "HDFS_CLIENT"
        },
        {
          "name" : "NODEMANAGER"
        },
        {
          "name" : "YARN_CLIENT"
        },
        {
          "name" : "MAPREDUCE2_CLIENT"
        },
        {
          "name" : "ZOOKEEPER_CLIENT"
        }
      ],
      "cardinality" : "2"
    }
  ],
  "Blueprints" : {
    "stack_name" : "HDP",
    "stack_version" : "2.0"
  }
}

Register blueprint with Ambari Server

Post the blueprint to the "single-node-hdfs-yarn" resource to the Ambari Server.

POST /api/v1/blueprints/multi-node-hdfs-yarn
... 
[ Request Body is the example blueprint defined above ]
...
 
201 - Created

Example Cluster Creation Template

We are performing a multi-node install and the blueprint above has two host groups. Therefore, for our cluster instance, we define one host in masters, two hosts in slaves and reference the multi-node-hdfs-yarn blueprint.

{
  "blueprint" : "multi-node-hdfs-yarn",
  "host_groups" :[
    { 
      "name" : "master",  
      "hosts" : [          
        { 
          "fqdn" : "c6401.ambari.apache.org" 
        }
      ]
    },
    { 
      "name" : "slaves",  
      "hosts" : [
        { 
          "fqdn" : "c6402.ambari.apache.org" 
        },
        { 
          "fqdn" : "c6403.ambari.apache.org"
        }
      ]
    }
  ]
}

Create Cluster Instance

Post the cluster to the Ambari Server to provision the cluster.

POST /api/v1/clusters/MyThreeNodeCluster
...
[ Request Body is above Cluster Creation Template ]
...

202 - Accepted
{
  "href" : "http://c6401.ambari.apache.org:8080/api/v1/clusters/MyCluster/requests/1",
  "Requests" : {
    "id" : 1,
    "status" : "InProgress"
  }
}