THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
For more details on role command order, see the Role Command Order section below.
Define Stack
Service Advisor
From Ambari 2.4, each service can choose to define it own service advisor rather than define the details of its configuration and layout in the stack advisor. This is particularly useful for custom services which are not defined in the stack. Ambari provides the Service Advisor capability where a service can write a Python script named service-advisor.py in their service folder. This folder can be in the stack's services directory where the service is defined or can be inherited from the service definition in common-services or elsewhere. Example: common-services/HAWQ/2.0.0.
Unlike the Stack-advisor scripts, the service-advisor scripts do not automatically extend the parent service's service-advisor scripts. The service-advisor script needs to explicitly extend their parent's service service-advisor script. The following code sample shows how you would refer to a parent's service_advisor.py. In this case it is extending the root service-advisor.py file in the resources/stacks directory.
| Code Block | ||
|---|---|---|
| ||
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
STACKS_DIR = os.path.join(SCRIPT_DIR, '../../../stacks/')
PARENT_FILE = os.path.join(STACKS_DIR, 'service_advisor.py')
try:
with open(PARENT_FILE, 'rb') as fp:
service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, ('.py', 'rb', imp.PY_SOURCE))
except Exception as e:
traceback.print_exc()
print "Failed to load parent"
class HAWQ200ServiceAdvisor(service_advisor.ServiceAdvisor): |
Like the stack advisors, service advisors provide information on 4 important aspects:
- Recommend layout of the service on cluster
- Recommend service configurations
- Validate layout of the service on cluster
- Validate service configurations
By providing the service-advisor.py file, one can control dynamically each of the above for the service.
The main interface for the service-advisor scripts contains documentation on how each of the above are called, and what data is provided.
| Code Block | ||
|---|---|---|
| ||
class ServiceAdvisor(DefaultStackAdvisor):
"""
Abstract class implemented by all service advisors.
"""
"""
If any components of the service should be colocated with other services,
this is where you should set up that layout. Example:
# colocate HAWQSEGMENT with DATANODE, if no hosts have been allocated for HAWQSEGMENT
hawqSegment = [component for component in serviceComponents if component["StackServiceComponents"]["component_name"] == "HAWQSEGMENT"][0]
if not self.isComponentHostsPopulated(hawqSegment):
for hostName in hostsComponentsMap.keys():
hostComponents = hostsComponentsMap[hostName]
if {"name": "DATANODE"} in hostComponents and {"name": "HAWQSEGMENT"} not in hostComponents:
hostsComponentsMap[hostName].append( { "name": "HAWQSEGMENT" } )
if {"name": "DATANODE"} not in hostComponents and {"name": "HAWQSEGMENT"} in hostComponents:
hostComponents.remove({"name": "HAWQSEGMENT"})
"""
def colocateService(self, hostsComponentsMap, serviceComponents):
pass
"""
Any configuration recommendations for the service should be defined in this function.
This should be similar to any of the recommendXXXXConfigurations functions in the stack_advisor.py
such as recommendYARNConfigurations().
"""
def getServiceConfigurationRecommendations(self, configurations, clusterSummary, services, hosts):
pass
"""
Returns an array of Validation objects about issues with the hostnames to which components are assigned.
This should detect validation issues which are different than those the stack_advisor.py detects.
The default validations are in stack_advisor.py getComponentLayoutValidations function.
"""
def getServiceComponentLayoutValidations(self, services, hosts):
return []
"""
Any configuration validations for the service should be defined in this function.
This should be similar to any of the validateXXXXConfigurations functions in the stack_advisor.py
such as validateHDFSConfigurations.
"""
def getServiceConfigurationsValidationItems(self, configurations, recommendedDefaults, services, hosts):
return [] |
Examples
- Service Advisor interface
- HAWQ 2.0.0 Service Advisor implementation
- PXF 3.0.0 Service Advisor implementation
Define Stack
A stack is a versioned collection of services. Each stack is a folder is defined in ambari-server/src/A stack is a versioned collection of services. Each stack is a folder is defined in ambari-server/src/main/resources/stacks source. Once installed, these stack definitions are available on the ambari-server machine at /var/lib/ambari-server/resources/stacks.
...
- INSTALL
- UNINSTALL
- START
- RESTART
- STOP
- EXECUTE
- ABORT
- UPGRADE
- SERVICE_CHECK
- CUSTOM_COMMAND
- ACTIONEXECUTE
Examples
| Role Command Order | Explanation |
|---|---|
| "HIVE_METASTORE-START": ["MYSQL_SERVER-START", "NAMENODE-START"] | Start MySQL and NameNode components before starting Hive Metastore |
| "MAPREDUCE_SERVICE_CHECK-SERVICE_CHECK": ["NODEMANAGER-START", "RESOURCEMANAGER-START"], | MapReduce service check needs ResourceManager and NodeManagers started |
| "ZOOKEEPER_SERVER-STOP" : ["HBASE_MASTER-STOP", "HBASE_REGIONSERVER-STOP", "METRICS_COLLECTOR-STOP"], | Before stopping ZooKeeper servers, make sure HBase Masters, HBase RegionServers and AMS Metrics Collector are stopped. |
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
class StackAdvisor(object):
"""
Abstract class implemented by all stack advisors. Stack advisors advise on stack specific questions.
Currently stack advisors provide following abilities:
- Recommend where services should be installed in cluster
- Recommend configurations based on host hardware
- Validate user selection of where services are installed on cluster
- Validate user configuration values
Each of the above methods is passed in parameters about services and hosts involved as described below.
@type services: dictionary
@param services: Dictionary containing all information about services selected by the user.
Example: {
"services": [
{
"StackServices": {
"service_name" : "HDFS",
"service_version" : "2.6.0.2.2",
},
"components" : [
{
"StackServiceComponents" : {
"cardinality" : "1+",
"component_category" : "SLAVE",
"component_name" : "DATANODE",
"display_name" : "DataNode",
"service_name" : "HDFS",
"hostnames" : []
},
"dependencies" : []
}, {
"StackServiceComponents" : {
"cardinality" : "1-2",
"component_category" : "MASTER",
"component_name" : "NAMENODE",
"display_name" : "NameNode",
"service_name" : "HDFS",
"hostnames" : []
},
"dependencies" : []
},
...
]
},
...
]
}
@type hosts: dictionary
@param hosts: Dictionary containing all information about hosts in this cluster
Example: {
"items": [
{
Hosts: {
"host_name": "c6401.ambari.apache.org",
"public_host_name" : "c6401.ambari.apache.org",
"ip": "192.168.1.101",
"cpu_count" : 1,
"disk_info" : [
{
"available" : "4564632",
"used" : "5230344",
"percent" : "54%",
"size" : "10319160",
"type" : "ext4",
"mountpoint" : "/"
},
{
"available" : "1832436",
"used" : "0",
"percent" : "0%",
"size" : "1832436",
"type" : "tmpfs",
"mountpoint" : "/dev/shm"
}
],
"host_state" : "HEALTHY",
"os_arch" : "x86_64",
"os_type" : "centos6",
"total_mem" : 3664872
}
},
...
]
}
Each of the methods can either return recommendations or validations.
Recommendations are made in a Ambari Blueprints friendly format.
Validations are an array of validation objects.
"""
def recommendComponentLayout(self, services, hosts):
"""
Returns recommendation of which hosts various service components should be installed on.
This function takes as input all details about services being installed, and hosts
they are being installed into, to generate hostname assignments to various components
of each service.
@type services: dictionary
@param services: Dictionary containing all information about services selected by the user.
@type hosts: dictionary
@param hosts: Dictionary containing all information about hosts in this cluster
@rtype: dictionary
@return: Layout recommendation of service components on cluster hosts in Ambari Blueprints friendly format.
Example: {
"resources" : [
{
"hosts" : [
"c6402.ambari.apache.org",
"c6401.ambari.apache.org"
],
"services" : [
"HDFS"
],
"recommendations" : {
"blueprint" : {
"host_groups" : [
{
"name" : "host-group-2",
"components" : [
{ "name" : "JOURNALNODE" },
{ "name" : "ZKFC" },
{ "name" : "DATANODE" },
{ "name" : "SECONDARY_NAMENODE" }
]
},
{
"name" : "host-group-1",
"components" :
{ "name" : "HDFS_CLIENT" },
{ "name" : "NAMENODE" },
{ "name" : "JOURNALNODE" },
{ "name" : "ZKFC" },
{ "name" : "DATANODE" }
]
}
]
},
"blueprint_cluster_binding" : {
"host_groups" : [
{
"name" : "host-group-1",
"hosts" : [ { "fqdn" : "c6401.ambari.apache.org" } ]
},
{
"name" : "host-group-2",
"hosts" : [ { "fqdn" : "c6402.ambari.apache.org" } ]
}
]
}
}
}
]
}
"""
pass
def validateComponentLayout(self, services, hosts):
"""
Returns array of Validation issues with service component layout on hosts
This function takes as input all details about services being installed along with
hosts the components are being installed on (hostnames property is populated for
each component).
@type services: dictionary
@param services: Dictionary containing information about services and host layout selected by the user.
@type hosts: dictionary
@param hosts: Dictionary containing all information about hosts in this cluster
@rtype: dictionary
@return: Dictionary containing array of validation items
Example: {
"items": [
{
"type" : "host-group",
"level" : "ERROR",
"message" : "NameNode and Secondary NameNode should not be hosted on the same machine",
"component-name" : "NAMENODE",
"host" : "c6401.ambari.apache.org"
},
...
]
}
"""
pass
def recommendConfigurations(self, services, hosts):
"""
Returns recommendation of service configurations based on host-specific layout of components.
This function takes as input all details about services being installed, and hosts
they are being installed into, to recommend host-specific configurations.
@type services: dictionary
@param services: Dictionary containing all information about services and component layout selected by the user.
@type hosts: dictionary
@param hosts: Dictionary containing all information about hosts in this cluster
@rtype: dictionary
@return: Layout recommendation of service components on cluster hosts in Ambari Blueprints friendly format.
Example: {
"services": [
"HIVE",
"TEZ",
"YARN"
],
"recommendations": {
"blueprint": {
"host_groups": [],
"configurations": {
"yarn-site": {
"properties": {
"yarn.scheduler.minimum-allocation-mb": "682",
"yarn.scheduler.maximum-allocation-mb": "2048",
"yarn.nodemanager.resource.memory-mb": "2048"
}
},
"tez-site": {
"properties": {
"tez.am.java.opts": "-server -Xmx546m -Djava.net.preferIPv4Stack=true -XX:+UseNUMA -XX:+UseParallelGC",
"tez.am.resource.memory.mb": "682"
}
},
"hive-site": {
"properties": {
"hive.tez.container.size": "682",
"hive.tez.java.opts": "-server -Xmx546m -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseParallelGC",
"hive.auto.convert.join.noconditionaltask.size": "238026752"
}
}
}
},
"blueprint_cluster_binding": {
"host_groups": []
}
},
"hosts": [
"c6401.ambari.apache.org",
"c6402.ambari.apache.org",
"c6403.ambari.apache.org"
]
}
"""
pass
def validateConfigurations(self, services, hosts):
""""
Returns array of Validation issues with configurations provided by user
This function takes as input all details about services being installed along with
configuration values entered by the user. These configurations can be validated against
service requirements, or host hardware to generate validation issues.
@type services: dictionary
@param services: Dictionary containing information about services and user configurations.
@type hosts: dictionary
@param hosts: Dictionary containing all information about hosts in this cluster
@rtype: dictionary
@return: Dictionary containing array of validation items
Example: {
"items": [
{
"config-type": "yarn-site",
"message": "Value is less than the recommended default of 682",
"type": "configuration",
"config-name": "yarn.scheduler.minimum-allocation-mb",
"level": "WARN"
}
]
}
"""
pass |
Examples
- Stack Advisor interface
- Default Stack Advisor implementation - for all stacks
- HDP (2.0.6) Default Stack Advisor implementation
- YARN container size calculated
- Recommended configurations - HDFS, YARN, MapReduce2, HBase (HDP-2.0.6), HBase (HDP-2.3)
- Delete HBase Bucket Cache configs on smaller machines
- Specify maximum value for Tez config
...
You can read about Express Upgrade steps in this documentation.
Examples
Stack Advisor
With each stack containing multiple complex services, it becomes necessary to dynamically determine how the services are laid out on the cluster, and for determining values of certain configurations.
Ambari provides the Stack Advisor capability where stacks can write a Python script named stack-advisor.py in the services/ folder. Example: HDP-2.0.6.
Stack-advisor scripts automatically extend the parent stack-version's stack-advisor scripts. This allows newer stack-versions to change behavior without effecting earlier behavior.
Stack advisors provide information on 4 important aspects:
- Recommend layout of services on cluster
- Recommend service configurations
- Validate layout of services on cluster
- Validate service configurations
By providing the stack-advisor.py file, one can control dynamically each of the above.
The main interface for the stack-advisor scripts contains documentation on how each of the above are called, and what data is provided