Services MUST provide the main metainfo.xml file which provides important metadata about the service.
Apart from that, other files can be provided to give more information about the service. More details about these files are provided below.

A service may also inherit from either a previous stack version or common services. For more information see the Service Inheritance page.

metainfo.xml

In the metainfo.xml service descriptor, one can first define the service and its components.

...

config-type - String representing a group of configurations. Example: core-site, hdfs-site, yarn-site, etc. When configurations are saved in Ambari, they are persisted within a version of config-type which is immutable. If you change and save HDFS core-site configs 4 times, you will have 4 versions of config-type core-site. Also, when a service's configs are saved, only the changed config-types are updated.
configFiles - lists the config-files handled by the enclosing component
configFile - represents one config-file of a certain type
- type - type of file based on which contents are generated differently
  - xml - XML file generated in Hadoop friendly format. Ex: hdfs-site.xml
  - env - Generally used for scripts where the content value is used as a template. The template has config-tags whose values are populated at runtime during file generation. Ex: hadoop-env.sh
  - properties - Generates property files where entries are in key=value format. Ex: falcon-runtime.properties
- dictionaryName - Name of the config-type as which key/values of this config file will be stored
configuration-dependencies - Lists the config-types on which this component or service depends on. One of the implications of this dependency is that whenever the config-type is updated, Ambari automatically marks the component or service as requiring restart. From the code section above, whenever core-site is updated, both HDFS service as well as HDFS_CLIENT component will be marked as requiring restart.
configuration-dir - Directory where files listed in configFiles will be. Optional. Default value is configuration.

Adding new configs in a config-type

There are a number of different parameters that can be specified to a config item when it is added to a config-type. These have been covered here.

UI - Categories

Configurations defined above show up in the service's Configs page.

...

Read more about Ambari Alerts framework here in the Alerts wiki page and the alerts.json format in the Alerts definition documentation.

Kerberos

Ambari is capable of enabling and disabling Kerberos for a cluster. To inform Ambari of the identities and configurations to be used for the service and its components, each service can provide a kerberos.json file.

Read more about the Kerberos support and the Kerberos Descriptor in the Automated Kerberization wiki page and the Kerberos descriptor in the Kerberos Descriptor documentation.

Metrics

Ambari provides the Ambari Metrics System ("AMS") service for collecting, aggregating and serving Hadoop and system metrics in Ambari-managed clusters.

Each service can define which metrics AMS should collect and provide by defining a metrics.json file.
You can read about the metrics.json file format in the Stack Defined Metrics page.

...

Quick Links

A service can add a list of quick links to the Ambari web UI by adding metainfo to a text file following a predefined JSON format. Ambari server parses the quicklink JSON file and provides its content to the UI. So that Ambari web UI can calculate quick link URLs based on the information and populate the quicklinks drop-down list accordingly.

Read more about quick links JSON file design in the Quick Links page.

Widgets

Each service can define which widgets and heatmaps show up by default on the service Each service can define which widgets and heatmaps show up by default on the service summary page by defining a widgets.json file.

...

From Ambari 2.4, each service can choose to define it its own service advisor rather than define the details of its configuration and layout in the stack advisor. This is particularly useful for custom services which are not defined in the stack. Ambari provides the Service Advisor capability where a service can write a Python script named service-advisor.py in their service folder. This folder can be in the stack's services directory where the service is defined or can be inherited from the service definition in common-services or elsewhere. Example: common-services/HAWQ/2.0.0.

...

Define Stack

A stack is a versioned collection of services. Each stack is a folder is defined in ambari-server/src/main/resources/stacks source. Once installed, these stack definitions are available on the ambari-server machine at /var/lib/ambari-server/resources/stacks.

Each stack folder contains one sub-folder per version of the stack. Some of these stack-versions are active while some are not. Each stack-version includes services which are either referenced from common-services, or defined inside the stack-version's services folder.

Image Removed

Example: HDP stack. HDP-2.4 stack version.

Stack-Version Descriptor

Each stack-version should provide a metainfo.xml (Example: HDP-2.3, HDP-2.4) descriptor file which describes the following about this stack-version:

Code Block
<metainfo> <versions> <active>true</active> </versions> <extends>2.3</extends> <minJdk>1.7</minJdk> <maxJdk>1.8</maxJdk> </metainfo>

versions/active - Whether this stack-version is still available for install. If not available, this version will not show up in UI during install.
extends - The stack-version in this stack that is being extended. Extended stack-versions inherit services along with almost all aspects of the parent stack-version.
minJdk - Minimum JDK with which this stack-version is supported. Users are warned during installer wizard if the JDK used by Ambari is lower than this version.
maxJdk - Maximum JDK with which this stack-version is supported. Users are warned during installer wizard if the JDK used by Ambari is greater than this version.

Services

Each stack-version includes services which are either referenced from common-services, or defined inside the stack-version's services folder.
Services are defined in common-services if they will be shared across multiple stacks. If they will never be shared, then they can be defined inside the stack-version.

Reference common-services

To reference a service from common-services, the service descriptor file should use the <extends> element. (Example: HDFS in HDP-2.0.6)

Code Block
<metainfo> <schemaVersion>2.0</schemaVersion> <services> <service> <name>HDFS</name> <extends>common-services/HDFS/2.1.0.2.0</extends> </service> </services> </metainfo>

Define Service

In exactly the same format as services defined in common-services, a new service can be defined inside the services folder.

Examples:

Extend Service

When a stack-version extends another stack-version, it inherits all details of the parent service. It is also free to override and remove any portion of the inherited service definition.

Examples:

Service Upgrade

From Ambari 2.4, each service can now define its upgrade within its service definition. This is particularly useful for custom services which no longer need to modify the stack's upgrade-packs in order to integrate themselves into the cluster upgrade.

Each service can define upgrade-packs, which are XML files describing the upgrade process of that particular service and how the upgrade pack relates to the overall stack upgrade-packs. These upgrade-pack XML files are placed in the service's upgrades/ folder in separate sub-folders specific to the stack-version they are meant to extend. Some examples of this can be seen in the testing code.

Examples

Each upgrade-pack that the service defines should match the file name of the service defined by a particular stack version. For example in the testing code, HDP 2.2.0 had an upgrade_test_15388.xml upgrade-pack. The HDFS service defined an extension to that upgrade pack HDP/2.0.5/services/HDFS/upgrades/HDP/2.2.0/upgrade_test_15388.xml. In this case the upgrade-pack was defined in the HDP/2.0.5 stack. The upgrade-pack is an extension to HDP/2.2.0 because it is defined in upgrade/HDP/2.2.0 directory. Finally the name of the service's extension to the upgrade-pack upgrade_test_15388.xml matches the name of the upgrade-pack in HDP/2.2.0/upgrades.

The file format for the service is much the same as that of the stack. The target, target-stack and type attributes should all be the same as the stack's upgrade-pack. The service is able to add its own prerequisite checks.

Code Block

title	General Attributes and Prerequisite Checks

<upgrade xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <target>2.4.*</target>
  <target-stack>HDP-2.4.0</target-stack>
  <type>ROLLING</type>
  <prerequisite-checks>
    <check>org.apache.ambari.server.checks.FooCheck</check>
  </prerequisite-checks>

The order section of the upgrade-pack, consists of group elements just like the stack's upgrade-pack. The key difference is defining how these groups relate to groups in the stack's upgrade pack or other service upgrade-packs. In the first example we are referencing the PRE_CLUSTER group and adding a new execute-stage for the service FOO. The entry is supposed to be added after the execute-stage for HDFS based on the <add-after-group-entry> tag.

Code Block

title	Order Section - Add After Group Entry

  <order>
    <group xsi:type="cluster" name="PRE_CLUSTER" title="Pre {{direction.text.proper}}">
      <add-after-group-entry>HDFS</add-after-group-entry>
      <execute-stage service="FOO" component="BAR" title="Backup FOO">
        <task xsi:type="manual">
          <message>Back FOO up.</message>
        </task>
      </execute-stage>
    </group>

The same syntax can be used to order other sections like service check priorities and group services.

Code Block

title	Order Section - Further Add After Group Entry Examples

    <group name="SERVICE_CHECK1" title="All Service Checks" xsi:type="service-check">
      <add-after-group-entry>ZOOKEEPER</add-after-group-entry>
      <priority>
        <service>HBASE</service>
      </priority>
    </group>

    <group name="CORE_MASTER" title="Core Masters">
      <add-after-group-entry>YARN</add-after-group-entry>
      <service name="HBASE">
        <component>HBASE_MASTER</component>
      </service>
    </group>

It is also possible to add new groups and order them after other groups in the stack's upgrade-packs. In the following example, we are adding the FOO group after the HIVE group using the add-after-group tag.

Code Block

title	Order Section - Add After Group

    <group name="FOO" title="Foo">
      <add-after-group>HIVE</add-after-group>
      <skippable>true</skippable>
      <allow-retry>false</allow-retry>
      <service name="FOO">
        <component>BAR</component>
      </service>
    </group>

You could also include both the add-after-group and the add-after-group-entry tags in the same group. This will create a new group if it doesn't already exist and will order it after the add-after-group's group name. The add-after-group-entry will determine the internal ordering of that group's services, priorities or execute stages.

Code Block

title	Order Section - Add After Group

    <group name="FOO" title="Foo">
      <add-after-group>HIVE</add-after-group>
      <add-after-group-entry>FOO</add-after-group-entry>
      <skippable>true</skippable>
      <allow-retry>false</allow-retry>
      <service name="FOO2">
        <component>BAR2</component>
      </service>
    </group>

The processing section of the upgrade-pack remains the same as what it would be in the stack's upgrade-pack.

Code Block

title	Processing Section

   <processing>
    <service name="FOO">
      <component name="BAR">
        <upgrade>
          <task xsi:type="restart-task" />
        </upgrade>
      </component>
      <component name="BAR2">
        <upgrade>
          <task xsi:type="restart-task" />
        </upgrade>
      </component>
    </service>
  </processing>

Define Stack

A stack is a versioned collection of services. Each stack is a folder is defined in ambari-server/src/main/resources/stacks source. Once installed, these stack definitions are available on the ambari-server machine at /var/lib/ambari-server/resources/stacks.

Each stack folder contains one sub-folder per version of the stack. Some of these stack-versions are active while some are not. Each stack-version includes services which are either referenced from common-services, or defined inside the stack-version's services folder.

Image Added

Example: HDP stack. HDP-2.4 stack version.

Stack-Version Descriptor

Each stack-version should provide a metainfo.xml (Example: HDP-2.3, HDP-2.4) descriptor file which describes the following about this stack-version:

Code Block
<metainfo> <versions> <active>true</active> </versions> <extends>2.3</extends> <minJdk>1.7</minJdk> <maxJdk>1.8</maxJdk> </metainfo>

versions/active - Whether this stack-version is still available for install. If not available, this version will not show up in UI during install.
extends - The stack-version in this stack that is being extended. Extended stack-versions inherit services along with almost all aspects of the parent stack-version.
minJdk - Minimum JDK with which this stack-version is supported. Users are warned during installer wizard if the JDK used by Ambari is lower than this version.
maxJdk - Maximum JDK with which this stack-version is supported. Users are warned during installer wizard if the JDK used by Ambari is greater than this version.

Stack Properties

The stack must contain or inherit a properties directory which contains two files: stack_features.json and stack_tools.json. This directory is new in Ambari 2.4.

The stack_features.json contains a list of features that are included in Ambari and allows the stack to specify which versions of the stack include those features. The list of features are determined by the particular Ambari release. The reference list for a particular Ambari version should be found in the HDP/2.0.6/properties/stack_features.json in the branch for that Ambari release. Each feature has a name and description and the stack can provide the minimum and maximum version where that feature is supported.

{

"stack_features": [

{

"name": "snappy",

"description": "Snappy compressor/decompressor support",

"min_version": "2.0.0.0",

"max_version": "2.2.0.0"

},

...

}

The stack_tools.json includes the name and location where the stack_selector and conf_selector tools are installed.

{

"stack_selector": ["hdp-select", "/usr/bin/hdp-select", "hdp-select"],

"conf_selector": ["conf-select", "/usr/bin/conf-select", "conf-select"]

}

Any custom stack must include these two JSON files. For further information see the Stack Properties wiki page.

Services

Each stack-version includes services which are either referenced from common-services, or defined inside the stack-version's services folder.
Services are defined in common-services if they will be shared across multiple stacks. If they will never be shared, then they can be defined inside the stack-version.

Reference common-services

To reference a service from common-services, the service descriptor file should use the <extends> element. (Example: HDFS in HDP-2.0.6)

Code Block
<metainfo> <schemaVersion>2.0</schemaVersion> <services> <service> <name>HDFS</name> <extends>common-services/HDFS/2.1.0.2.0</extends> </service> </services> </metainfo>

Define Service

In exactly the same format as services defined in common-services, a new service can be defined inside the services folder.

Examples:

Extend Service

When a stack-version extends another stack-version, it inherits all details of the parent service. It is also free to override and remove any portion of the inherited service definition.

Examples:

HDP-2.3 / HDFS - Adding NFS_GATEWAY component, updating service version and OS specific packages
HDP-2.2 / Storm - Deleting STORM_REST_API component, updating
HDP-2.3 / HDFS - Adding NFS_GATEWAY component, updating service version and OS specific packages
HDP-2.2 / Storm - Deleting STORM_REST_API component, updating service version and OS specific packages
HDP-2.3 / YARN - Deleting YARN node-label configuration from capacity-scheduler.xml
HDP-2.3 / Kafka - Add Kafka Broker Process alert

...

INSTALL
UNINSTALL
START
RESTART
STOP
EXECUTE
ABORT
UPGRADE
SERVICE_CHECK
CUSTOM_COMMAND
ACTIONEXECUTE

Examples

Role Command Order	Explanation
"HIVE_METASTORE-START": ["MYSQL_SERVER-START", "NAMENODE-START"]	Start MySQL and NameNode components before starting Hive Metastore
"MAPREDUCE_SERVICE_CHECK-SERVICE_CHECK": ["NODEMANAGER-START", "RESOURCEMANAGER-START"],	MapReduce service check needs ResourceManager and NodeManagers started
"ZOOKEEPER_SERVER-STOP" : ["HBASE_MASTER-STOP", "HBASE_REGIONSERVER-STOP", "METRICS_COLLECTOR-STOP"],	Before stopping ZooKeeper servers, make sure HBase Masters, HBase RegionServers and AMS Metrics Collector are stopped.

...

Code Block

language	py
title	stack-advisor.py Interface
linenumbers	true

class StackAdvisor(object):
  """
  Abstract class implemented by all stack advisors. Stack advisors advise on stack specific questions. 


  Currently stack advisors provide following abilities:
  - Recommend where services should be installed in cluster
  - Recommend configurations based on host hardware
  - Validate user selection of where services are installed on cluster
  - Validate user configuration values 

  Each of the above methods is passed in parameters about services and hosts involved as described below.


    @type services: dictionary
    @param services: Dictionary containing all information about services selected by the user. 
      Example: {
      "services": [
        {
          "StackServices": {
            "service_name" : "HDFS",
            "service_version" : "2.6.0.2.2",
          },
          "components" : [ 
            {
              "StackServiceComponents" : {
                "cardinality" : "1+",
                "component_category" : "SLAVE",
                "component_name" : "DATANODE",
                "display_name" : "DataNode",
                "service_name" : "HDFS",
                "hostnames" : []
              },
              "dependencies" : []
            }, {
              "StackServiceComponents" : {
                "cardinality" : "1-2",
                "component_category" : "MASTER",
                "component_name" : "NAMENODE",
                "display_name" : "NameNode",
                "service_name" : "HDFS",
                "hostnames" : []
              },
              "dependencies" : []
            },
            ...
          ]
        },
        ...
      ]
    }
  @type hosts: dictionary
  @param hosts: Dictionary containing all information about hosts in this cluster
    Example: {
      "items": [
        {
          Hosts: {
            "host_name": "c6401.ambari.apache.org",
            "public_host_name" : "c6401.ambari.apache.org",
            "ip": "192.168.1.101",
            "cpu_count" : 1,
            "disk_info" : [
              {
              "available" : "4564632",
              "used" : "5230344",
              "percent" : "54%",
              "size" : "10319160",
              "type" : "ext4",
              "mountpoint" : "/"
              },
              {
              "available" : "1832436",
              "used" : "0",
              "percent" : "0%",
              "size" : "1832436",
              "type" : "tmpfs",
              "mountpoint" : "/dev/shm"
              }
            ],
            "host_state" : "HEALTHY",
            "os_arch" : "x86_64",
            "os_type" : "centos6",
            "total_mem" : 3664872
          }
        },
        ...
      ]
    }


    Each of the methods can either return recommendations or validations.

    Recommendations are made in a Ambari Blueprints friendly format. 
    Validations are an array of validation objects.
  """


  def recommendComponentLayout(self, services, hosts):
    """
    Returns recommendation of which hosts various service components should be installed on.

    This function takes as input all details about services being installed, and hosts
    they are being installed into, to generate hostname assignments to various components
    of each service.


    @type services: dictionary
    @param services: Dictionary containing all information about services selected by the user.
    @type hosts: dictionary
    @param hosts: Dictionary containing all information about hosts in this cluster
    @rtype: dictionary
    @return: Layout recommendation of service components on cluster hosts in Ambari Blueprints friendly format. 
        Example: {
          "resources" : [
            {
              "hosts" : [
                "c6402.ambari.apache.org",
                "c6401.ambari.apache.org"
              ],
              "services" : [
                "HDFS"
              ],
              "recommendations" : {
                "blueprint" : {
                  "host_groups" : [
                    {
                      "name" : "host-group-2",
                      "components" : [
                        { "name" : "JOURNALNODE" },
                        { "name" : "ZKFC" },
                        { "name" : "DATANODE" },
                        { "name" : "SECONDARY_NAMENODE" }
                      ]
                    },
                    {
                      "name" : "host-group-1",
                      "components" : 
                        { "name" : "HDFS_CLIENT" },
                        { "name" : "NAMENODE" },
                        { "name" : "JOURNALNODE" },
                        { "name" : "ZKFC" },
                        { "name" : "DATANODE" }
                      ]
                    }
                  ]
                },
                "blueprint_cluster_binding" : {
                  "host_groups" : [
                    {
                      "name" : "host-group-1",
                      "hosts" : [ { "fqdn" : "c6401.ambari.apache.org" } ]
                    },
                    {
                      "name" : "host-group-2",
                      "hosts" : [ { "fqdn" : "c6402.ambari.apache.org" } ]
                    }
                  ]
                }
              }
            }
          ]
        }
    """
    pass

  def validateComponentLayout(self, services, hosts):
    """
    Returns array of Validation issues with service component layout on hosts


    This function takes as input all details about services being installed along with
    hosts the components are being installed on (hostnames property is populated for 
    each component).  

    @type services: dictionary
    @param services: Dictionary containing information about services and host layout selected by the user.
    @type hosts: dictionary
    @param hosts: Dictionary containing all information about hosts in this cluster
    @rtype: dictionary
    @return: Dictionary containing array of validation items
        Example: {
          "items": [
            {
              "type" : "host-group",
              "level" : "ERROR",
              "message" : "NameNode and Secondary NameNode should not be hosted on the same machine",
              "component-name" : "NAMENODE",
              "host" : "c6401.ambari.apache.org" 
            },
            ...
          ]
        }  
    """
    pass


  def recommendConfigurations(self, services, hosts):
    """
    Returns recommendation of service configurations based on host-specific layout of components.

    This function takes as input all details about services being installed, and hosts
    they are being installed into, to recommend host-specific configurations.


    @type services: dictionary
    @param services: Dictionary containing all information about services and component layout selected by the user.
    @type hosts: dictionary
    @param hosts: Dictionary containing all information about hosts in this cluster
    @rtype: dictionary
    @return: Layout recommendation of service components on cluster hosts in Ambari Blueprints friendly format. 
        Example: {
         "services": [
          "HIVE", 
          "TEZ", 
          "YARN"
         ], 
         "recommendations": {
          "blueprint": {
           "host_groups": [], 
           "configurations": {
            "yarn-site": {
             "properties": {
              "yarn.scheduler.minimum-allocation-mb": "682", 
              "yarn.scheduler.maximum-allocation-mb": "2048", 
              "yarn.nodemanager.resource.memory-mb": "2048"
             }
            }, 
            "tez-site": {
             "properties": {
              "tez.am.java.opts": "-server -Xmx546m -Djava.net.preferIPv4Stack=true -XX:+UseNUMA -XX:+UseParallelGC", 
              "tez.am.resource.memory.mb": "682"
             }
            }, 
            "hive-site": {
             "properties": {
              "hive.tez.container.size": "682", 
              "hive.tez.java.opts": "-server -Xmx546m -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC", 
              "hive.auto.convert.join.noconditionaltask.size": "238026752"
             }
            }
           }
          }, 
          "blueprint_cluster_binding": {
           "host_groups": []
          }
         }, 
         "hosts": [
          "c6401.ambari.apache.org", 
          "c6402.ambari.apache.org", 
          "c6403.ambari.apache.org" 
         ] 
        }
    """
    pass

  def validateConfigurations(self, services, hosts):
    """"
    Returns array of Validation issues with configurations provided by user
    This function takes as input all details about services being installed along with
    configuration values entered by the user. These configurations can be validated against
    service requirements, or host hardware to generate validation issues. 


    @type services: dictionary
    @param services: Dictionary containing information about services and user configurations.
    @type hosts: dictionary
    @param hosts: Dictionary containing all information about hosts in this cluster
    @rtype: dictionary
    @return: Dictionary containing array of validation items
        Example: {
         "items": [
          {
           "config-type": "yarn-site", 
           "message": "Value is less than the recommended default of 682", 
           "type": "configuration", 
           "config-name": "yarn.scheduler.minimum-allocation-mb", 
           "level": "WARN"
          }
         ]
       }
    """
    pass

Examples

Stack Advisor interface
Default Stack Advisor implementation - for all stacks
HDP (2.0.6) Default Stack Advisor implementation
YARN container size calculated
Recommended configurations - HDFS, YARN, MapReduce2, HBase (HDP-2.0.6), HBase (HDP-2.3)
Delete HBase Bucket Cache configs on smaller machines

Specify maximum value for Tez config

YARN container size calculated
Recommended configurations - HDFS, YARN, MapReduce2, HBase (HDP-2.0.6), HBase (HDP-2.3)
Delete HBase Bucket Cache configs on smaller machines
Specify maximum value for Tez config

Properties

Similar to stack configurations, most properties are defined at the service level, however there are global properties which can be defined at the stack-version level affecting across all services.
Some examples are: stack-selector and conf-selector specific names or what stack versions certain stack features are supported by. Most of these properties were introduced in Ambari 2.4 version in the effort of parameterize stack information and facilitate the reuse of common-services code by other distributions.

Such properties can be defined in .json format in the properties folder of the stack.

More details about stack properties can be found on Stack Properties section.

Widgets

At the stack-version level one can contribute heatmap entries to the main dashboard of the cluster.
Generally these heatmaps would be ones which apply to all services - like host level heatmaps.
Example: HDP-2.0.6 contributes host level heatmaps

...

Space shortcuts

Child pages

Versions Compared

Old Version 51

New Version Current

Key

metainfo.xml

Adding new configs in a config-type

UI - Categories

Kerberos

Metrics

Quick Links

Widgets

Define Stack

Stack-Version Descriptor

Services

Reference common-services

Define Service

Extend Service

Service Upgrade

Examples

Define Stack

Stack-Version Descriptor

Stack Properties

Services

Reference common-services

Define Service

Extend Service

Examples

Examples

Properties

Widgets

Space shortcuts

Child pages

Page History

Versions Compared

Old Version 51

New Version Current

Key

metainfo.xml

Adding new configs in a config-type

UI - Categories

Kerberos

Metrics

Quick Links

Widgets

Define Stack

Stack-Version Descriptor

Services

Reference common-services

Define Service

Extend Service

Service Upgrade

Examples

Define Stack

Stack-Version Descriptor

Stack Properties

Services

Reference common-services

Define Service

Extend Service

Examples

Examples

Properties

Widgets