...
- Service Advisor interface
- HAWQ 2.0.0 Service Advisor implementation
- PXF 3.0.0 Service Advisor implementation
Service Upgrade
From Ambari 2.4, each service can now define its upgrade within its service definition. This is particularly useful for custom services which no longer need to modify the stack's upgrade-packs in order to integrate themselves into the cluster upgrade.
Each service can define upgrade-packs, which are XML files describing the upgrade process of that particular service and how the upgrade pack relates to the overall stack upgrade-packs. These upgrade-pack XML files are placed in the service's upgrades/ folder in separate sub-folders specific to the stack-version they are meant to extend. Some examples of this can be seen in the testing code.
Examples
Each upgrade-pack that the service defines should match the file name of the service defined by a particular stack version. For example in the testing code, HDP 2.2.0 had an upgrade_test_15388.xml upgrade-pack. The HDFS service defined an extension to that upgrade pack HDP/2.0.5/services/HDFS/upgrades/HDP/2.2.0/upgrade_test_15388.xml. In this case the upgrade-pack was defined in the HDP/2.0.5 stack. The upgrade-pack is an extension to HDP/2.2.0 because it is defined in upgrade/HDP/2.2.0 directory. Finally the name of the service's extension to the upgrade-pack upgrade_test_15388.xml matches the name of the upgrade-pack in HDP/2.2.0/upgrades.
The file format for the service is much the same as that of the stack. The target, target-stack and type attributes should all be the same as the stack's upgrade-pack. The service is able to add its own prerequisite checks.
Code Block | ||
---|---|---|
| ||
<upgrade xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<target>2.4.*</target>
<target-stack>HDP-2.4.0</target-stack>
<type>ROLLING</type>
<prerequisite-checks>
<check>org.apache.ambari.server.checks.FooCheck</check>
</prerequisite-checks> |
The order section of the upgrade-pack, consists of group elements just like the stack's upgrade-pack. The key difference is defining how these groups relate to groups in the stack's upgrade pack or other service upgrade-packs. In the first example we are referencing the PRE_CLUSTER group and adding a new execute-stage for the service FOO. The entry is supposed to be added after the execute-stage for HDFS based on the <add-after-group-entry> tag.
Code Block | ||
---|---|---|
| ||
<order>
<group xsi:type="cluster" name="PRE_CLUSTER" title="Pre {{direction.text.proper}}">
<add-after-group-entry>HDFS</add-after-group-entry>
<execute-stage service="FOO" component="BAR" title="Backup FOO">
<task xsi:type="manual">
<message>Back FOO up.</message>
</task>
</execute-stage>
</group> |
The same syntax can be used to order other sections like service check priorities and group services.
Code Block | ||
---|---|---|
| ||
<group name="SERVICE_CHECK1" title="All Service Checks" xsi:type="service-check">
<add-after-group-entry>ZOOKEEPER</add-after-group-entry>
<priority>
<service>HBASE</service>
</priority>
</group>
<group name="CORE_MASTER" title="Core Masters">
<add-after-group-entry>YARN</add-after-group-entry>
<service name="HBASE">
<component>HBASE_MASTER</component>
</service>
</group>
|
It is also possible to add new groups and order them after other groups in the stack's upgrade-packs. In the following example, we are adding the FOO group after the HIVE group using the add-after-group tag.
Code Block | ||
---|---|---|
| ||
<group name="FOO" title="Foo">
<add-after-group>HIVE</add-after-group>
<skippable>true</skippable>
<allow-retry>false</allow-retry>
<service name="FOO">
<component>BAR</component>
</service>
</group>
|
You could also include both the add-after-group and the add-after-group-entry tags in the same group. This will create a new group if it doesn't already exist and will order it after the add-after-group's group name. The add-after-group-entry will determine the internal ordering of that group's services, priorities or execute stages.
Code Block | ||
---|---|---|
| ||
<group name="FOO" title="Foo">
<add-after-group>HIVE</add-after-group>
<add-after-group-entry>FOO</add-after-group-entry>
<skippable>true</skippable>
<allow-retry>false</allow-retry>
<service name="FOO2">
<component>BAR2</component>
</service>
</group>
|
The processing section of the upgrade-pack remains the same as what it would be in the stack's upgrade-pack.
Code Block | ||
---|---|---|
| ||
<processing>
<service name="FOO">
<component name="BAR">
<upgrade>
<task xsi:type="restart-task" />
</upgrade>
</component>
<component name="BAR2">
<upgrade>
<task xsi:type="restart-task" />
</upgrade>
</component>
</service>
</processing> |
Define Stack
A stack is a versioned collection of services. Each stack is a folder is defined in ambari-server/src/main/resources/stacks source. Once installed, these stack definitions are available on the ambari-server machine at /var/lib/ambari-server/resources/stacks.
...
- INSTALL
- UNINSTALL
- START
- RESTART
- STOP
- EXECUTE
- ABORT
- UPGRADE
- SERVICE_CHECK
- CUSTOM_COMMAND
- ACTIONEXECUTE
Examples
Role Command Order | Explanation |
---|---|
"HIVE_METASTORE-START": ["MYSQL_SERVER-START", "NAMENODE-START"] | Start MySQL and NameNode components before starting Hive Metastore |
"MAPREDUCE_SERVICE_CHECK-SERVICE_CHECK": ["NODEMANAGER-START", "RESOURCEMANAGER-START"], | MapReduce service check needs ResourceManager and NodeManagers started |
"ZOOKEEPER_SERVER-STOP" : ["HBASE_MASTER-STOP", "HBASE_REGIONSERVER-STOP", "METRICS_COLLECTOR-STOP"], | Before stopping ZooKeeper servers, make sure HBase Masters, HBase RegionServers and AMS Metrics Collector are stopped. |
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
class StackAdvisor(object): """ Abstract class implemented by all stack advisors. Stack advisors advise on stack specific questions. Currently stack advisors provide following abilities: - Recommend where services should be installed in cluster - Recommend configurations based on host hardware - Validate user selection of where services are installed on cluster - Validate user configuration values Each of the above methods is passed in parameters about services and hosts involved as described below. @type services: dictionary @param services: Dictionary containing all information about services selected by the user. Example: { "services": [ { "StackServices": { "service_name" : "HDFS", "service_version" : "2.6.0.2.2", }, "components" : [ { "StackServiceComponents" : { "cardinality" : "1+", "component_category" : "SLAVE", "component_name" : "DATANODE", "display_name" : "DataNode", "service_name" : "HDFS", "hostnames" : [] }, "dependencies" : [] }, { "StackServiceComponents" : { "cardinality" : "1-2", "component_category" : "MASTER", "component_name" : "NAMENODE", "display_name" : "NameNode", "service_name" : "HDFS", "hostnames" : [] }, "dependencies" : [] }, ... ] }, ... ] } @type hosts: dictionary @param hosts: Dictionary containing all information about hosts in this cluster Example: { "items": [ { Hosts: { "host_name": "c6401.ambari.apache.org", "public_host_name" : "c6401.ambari.apache.org", "ip": "192.168.1.101", "cpu_count" : 1, "disk_info" : [ { "available" : "4564632", "used" : "5230344", "percent" : "54%", "size" : "10319160", "type" : "ext4", "mountpoint" : "/" }, { "available" : "1832436", "used" : "0", "percent" : "0%", "size" : "1832436", "type" : "tmpfs", "mountpoint" : "/dev/shm" } ], "host_state" : "HEALTHY", "os_arch" : "x86_64", "os_type" : "centos6", "total_mem" : 3664872 } }, ... ] } Each of the methods can either return recommendations or validations. Recommendations are made in a Ambari Blueprints friendly format. Validations are an array of validation objects. """ def recommendComponentLayout(self, services, hosts): """ Returns recommendation of which hosts various service components should be installed on. This function takes as input all details about services being installed, and hosts they are being installed into, to generate hostname assignments to various components of each service. @type services: dictionary @param services: Dictionary containing all information about services selected by the user. @type hosts: dictionary @param hosts: Dictionary containing all information about hosts in this cluster @rtype: dictionary @return: Layout recommendation of service components on cluster hosts in Ambari Blueprints friendly format. Example: { "resources" : [ { "hosts" : [ "c6402.ambari.apache.org", "c6401.ambari.apache.org" ], "services" : [ "HDFS" ], "recommendations" : { "blueprint" : { "host_groups" : [ { "name" : "host-group-2", "components" : [ { "name" : "JOURNALNODE" }, { "name" : "ZKFC" }, { "name" : "DATANODE" }, { "name" : "SECONDARY_NAMENODE" } ] }, { "name" : "host-group-1", "components" : { "name" : "HDFS_CLIENT" }, { "name" : "NAMENODE" }, { "name" : "JOURNALNODE" }, { "name" : "ZKFC" }, { "name" : "DATANODE" } ] } ] }, "blueprint_cluster_binding" : { "host_groups" : [ { "name" : "host-group-1", "hosts" : [ { "fqdn" : "c6401.ambari.apache.org" } ] }, { "name" : "host-group-2", "hosts" : [ { "fqdn" : "c6402.ambari.apache.org" } ] } ] } } } ] } """ pass def validateComponentLayout(self, services, hosts): """ Returns array of Validation issues with service component layout on hosts This function takes as input all details about services being installed along with hosts the components are being installed on (hostnames property is populated for each component). @type services: dictionary @param services: Dictionary containing information about services and host layout selected by the user. @type hosts: dictionary @param hosts: Dictionary containing all information about hosts in this cluster @rtype: dictionary @return: Dictionary containing array of validation items Example: { "items": [ { "type" : "host-group", "level" : "ERROR", "message" : "NameNode and Secondary NameNode should not be hosted on the same machine", "component-name" : "NAMENODE", "host" : "c6401.ambari.apache.org" }, ... ] } """ pass def recommendConfigurations(self, services, hosts): """ Returns recommendation of service configurations based on host-specific layout of components. This function takes as input all details about services being installed, and hosts they are being installed into, to recommend host-specific configurations. @type services: dictionary @param services: Dictionary containing all information about services and component layout selected by the user. @type hosts: dictionary @param hosts: Dictionary containing all information about hosts in this cluster @rtype: dictionary @return: Layout recommendation of service components on cluster hosts in Ambari Blueprints friendly format. Example: { "services": [ "HIVE", "TEZ", "YARN" ], "recommendations": { "blueprint": { "host_groups": [], "configurations": { "yarn-site": { "properties": { "yarn.scheduler.minimum-allocation-mb": "682", "yarn.scheduler.maximum-allocation-mb": "2048", "yarn.nodemanager.resource.memory-mb": "2048" } }, "tez-site": { "properties": { "tez.am.java.opts": "-server -Xmx546m -Djava.net.preferIPv4Stack=true -XX:+UseNUMA -XX:+UseParallelGC", "tez.am.resource.memory.mb": "682" } }, "hive-site": { "properties": { "hive.tez.container.size": "682", "hive.tez.java.opts": "-server -Xmx546m -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC", "hive.auto.convert.join.noconditionaltask.size": "238026752" } } } }, "blueprint_cluster_binding": { "host_groups": [] } }, "hosts": [ "c6401.ambari.apache.org", "c6402.ambari.apache.org", "c6403.ambari.apache.org" ] } """ pass def validateConfigurations(self, services, hosts): """" Returns array of Validation issues with configurations provided by user This function takes as input all details about services being installed along with configuration values entered by the user. These configurations can be validated against service requirements, or host hardware to generate validation issues. @type services: dictionary @param services: Dictionary containing information about services and user configurations. @type hosts: dictionary @param hosts: Dictionary containing all information about hosts in this cluster @rtype: dictionary @return: Dictionary containing array of validation items Example: { "items": [ { "config-type": "yarn-site", "message": "Value is less than the recommended default of 682", "type": "configuration", "config-name": "yarn.scheduler.minimum-allocation-mb", "level": "WARN" } ] } """ pass |
Examples
- Stack Advisor interface
- Default Stack Advisor implementation - for all stacks
- HDP (2.0.6) Default Stack Advisor implementation
- YARN container size calculated
- Recommended configurations - HDFS, YARN, MapReduce2, HBase (HDP-2.0.6), HBase (HDP-2.3)
- Delete HBase Bucket Cache configs on smaller machines
- Specify maximum value for Tez config
...