THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
*** DRAFT - Being Edited ***
Services managed by Ambari are defined in its stacks folder.
To define your own services and stacks to be managed by Ambari, follow the steps below to define your own services or stacks.
There is also an example you can follow to create your custom stack and service.
A stack is a distribution of a set of services. Multiple versions of a stack can be defined, each with its own set of services. Stacks in Ambari are defined in ambari-server/src/main/resources/stacks folder, which can be found at /var/lib/ambari-server/resources/stacks folder after install.
Services managed by a stack can be defined either in ambari-server/src/main/resources/common-services or ambari-server/src/main/resources/stacks folders. These folders after install can be found at /var/lib/ambari-server/resources/common-services or /var/lib/ambari-server/resources/stacks folders respectively.
Question: When do I define service in common-services vs. stacks folders?
One would define services in the common-services folder if there is possibility of the service being used in multiple stacks. For example, almost all stacks would need the HDFS service - so instead of redefining HDFS in each stack, the one defined in common-services is referenced from multiple stack-versions. Likewise, if a service is going to be contained in only one stack and never going to be shared, it can be defined in the stacks folder. Basically services defined in stacks folder are by used containment, whereas the ones defined in common-services are used by reference.
Define Service
Shown below is how to define a service in common-services folder. The same approach can be taken when defining services in the stacks folder, which will be discussed in the Define Stack section.
Services should provide the main metainfo.xml file which provides important metadata about the service.
Apart from that, other files can be provided to give more information about the service. More details about these files are provided below.
metainfo.xml
In the metainfo.xml file, one can first define the service and its components.
Complete reference can be found in the Writing metainfo.xml page.
A good reference implementation is the HDFS metainfo.xml.
Question: Is it possible to define multiple services in the same metainfo.xml?
Yes. Though it is possible, it is discouraged to define multiple services in the same service folder.
YARN and MapReduce2 are services that are defined together in the YARN folder. Its metainfo.xml defines both services.
Scripts
With the components defined, we need to provide scripts which can handle the various stages of the service and component's lifecycle.
The scripts necessary to manage service and components are specified in the metainfo.xml (HDFS)
These scripts should be provided in the <service-id>/<service-version>/package/scripts folder.
| Folder | Purpose |
|---|---|
| package/scripts | Contains scripts invoked by Ambari. These scripts are loaded into the execution path with the correct environment. Example: HDFS |
| package/files | Contains files used by above scripts. Generally these are other scripts (bash, python, etc.) invoked as a separate process. Example: checkWebUI.py is run in HDFS service-check to determine if Journal Nodes are available |
| package/templates | Template files used by above scripts to generate files on managed hosts. These are generally configuration files required by the service to operate. |
Python
Ambari by default supports Python scripts for management of service and components.
Component scripts should extend resource_management.Script class and provide methods required for that component's lifecycle.
Taken from the page on how to create custom stack, the following methods are needed for MASTER, SLAVE and CLIENT components to go through their lifecycle.
master.py
import sys
from resource_management import *
class Master(Script):
def install(self, env):
print 'Install the Sample Srv Master';
def stop(self, env):
print 'Stop the Sample Srv Master';
def start(self, env):
print 'Start the Sample Srv Master';
def status(self, env):
print 'Status of the Sample Srv Master';
def configure(self, env):
print 'Configure the Sample Srv Master';
if __name__ == "__main__":
Master().execute()
slave.py
import sys
from resource_management import *
class Slave(Script):
def install(self, env):
print 'Install the Sample Srv Slave';
def stop(self, env):
print 'Stop the Sample Srv Slave';
def start(self, env):
print 'Start the Sample Srv Slave';
def status(self, env):
print 'Status of the Sample Srv Slave';
def configure(self, env):
print 'Configure the Sample Srv Slave';
if __name__ == "__main__":
Slave().execute()
client.py
import sys
from resource_management import *
class SampleClient(Script):
def install(self, env):
print 'Install the Sample Srv Client';
def configure(self, env):
print 'Configure the Sample Srv Client';
if __name__ == "__main__":
SampleClient().execute()
Ambari provides helpful Python libraries below which are useful in writing service scripts. For complete reference on these libraries visit the Ambari Python Libraries page.
- resource_management
- ambari_commons
- ambari_simplejson
OS Variant Scripts
If the service is supported on multiple OSes which requires separate scripts, the base resource_management.Script class can be extended with different @OsFamilyImpl() annotations.
This allows for ability to extend only those OS specific methods of the component management.
Example: NameNode default script, NameNode Windows script.
Examples
DataNode Start and Stop.
Custom Actions
Sometimes services need to perform actions unique to that service which go beyond the default actions provided by Ambari (like install, start, stop, configure, etc.).
Services can define such actions and expose them to the user in UI so that they can be easily invoked.
As an example, we show the Rebalance HDFS custom action implemented by HDFS.
Stack Changes
- Define custom command inside the customCommands section of the component in metainfo.xml.
- Implement method with same name as custom command in script referenced from metainfo.xml.
- If custom command does not have OS variants, it can be implemented in the same class that extends resource_management.Script
- If there are OS variants, different methods can be implemented in each class annotated by @OsFamilyImpl(os_family=...). Default rebalancehdfs, Windows rebalancehdfs.
This will provide ability by the backend to run the script on all managed hosts where the service is installed.
UI Changes
No UI changes are necessary to see the custom action on the host page.
The action should show up in the host-component's list of actions. Any master-component actions will automatically show up on the service's action menu.
When the action is clicked in UI, the POST call is made automatically to trigger the script defined above.
Question: How do I provide my own label and icon for the custom action in UI?
In Ambari UI, add your component action to the App.HostComponentActionMap object with custom icon and name. Ex: REBALANCEHDFS.
Question: How do I provide custom UI side logic for the action, to show or enable the action?
Configuration
Configuration files for a service should be placed by default in the configuration folder.
If a different named folder has to be used, the <configuration-dir> element can be used in metainfo.xml to point to that folder.
The important sections of the metainfo.xml with regards to configurations are:
<?xml version="1.0"?>
<metainfo>
<schemaVersion>2.0</schemaVersion>
<services>
<service>
<name>HDFS</name>
<displayName>HDFS</displayName>
<comment>Apache Hadoop Distributed File System</comment>
<version>2.1.0.2.0</version>
<components>
...
<component>
<name>HDFS_CLIENT</name>
...
<configFiles>
<configFile>
<type>xml</type>
<fileName>hdfs-site.xml</fileName>
<dictionaryName>hdfs-site</dictionaryName>
</configFile>
<configFile>
<type>xml</type>
<fileName>core-site.xml</fileName>
<dictionaryName>core-site</dictionaryName>
</configFile>
<configFile>
<type>env</type>
<fileName>log4j.properties</fileName>
<dictionaryName>hdfs-log4j,yarn-log4j</dictionaryName>
</configFile>
<configFile>
<type>env</type>
<fileName>hadoop-env.sh</fileName>
<dictionaryName>hadoop-env</dictionaryName>
</configFile>
</configFiles>
...
<configuration-dependencies>
<config-type>core-site</config-type>
<config-type>hdfs-site</config-type>
</configuration-dependencies>
</component>
...
</components>
<configuration-dir>configuration</configuration-dir>
<configuration-dependencies>
<config-type>core-site</config-type>
<config-type>hdfs-site</config-type>
<config-type>hadoop-env</config-type>
<config-type>hadoop-policy</config-type>
<config-type>hdfs-log4j</config-type>
<config-type>ranger-hdfs-plugin-properties</config-type>
<config-type>ssl-client</config-type>
<config-type>ssl-server</config-type>
<config-type>ranger-hdfs-audit</config-type>
<config-type>ranger-hdfs-policymgr-ssl</config-type>
<config-type>ranger-hdfs-security</config-type>
<config-type>ams-ssl-client</config-type>
</configuration-dependencies>
</service>
</services>
</metainfo>
- config-type - String representing a group of configurations. Example: core-site, hdfs-site, yarn-site, etc. When configurations are saved in Ambari, they are persisted within a version of config-type which is immutable. If you change and save HDFS core-site configs 4 times, you will have 4 versions of config-type core-site. Also, when a service's configs are saved, only the changed config-types are updated.
- configFiles - lists the config-files handled by the enclosing component
- configFile - represents one config-file of a certain type
- type - type of file based on which contents are generated differently
- xml - XML file generated in Hadoop friendly format. Ex: hdfs-site.xml
- env - Generally used for scripts where the content value is used as a template. The template has config-tags whose values are populated at runtime during file generation. Ex: hadoop-env.sh
- properties - Generates property files where entries are in key=value format. Ex: falcon-runtime.properties
- dictionaryName - Name of the config-type as which key/values of this config file will be stored
- type - type of file based on which contents are generated differently
- configuration-dependencies - Lists the config-types on which this component or service depends on. One of the implications of this dependency is that whenever the config-type is updated, Ambari automatically marks the component or service as requiring restart. From the code section above, whenever core-site is updated, both HDFS service as well as HDFS_CLIENT component will be marked as requiring restart.
- configuration-dir - Directory where files listed in configFiles will be. Optional. Default value is configuration.
UI - Categories
Configurations defined above show up in the service's Configs page.
To customize categories and ordering of configurations in UI, the following files have to be updated.
Create Category - Update the ambari-web/app/models/stack_service.js file to add your own service, along with your new categories.
Use Category - To place configs inside a defined category, and specify an order in which configs are placed, add configs to ambari-web/app/data/HDP2/site_properties.js file. In this file one can specify the category to use, and the index where a config should be placed. The stack folders in ambari-web/app/data are hierarchical and inherit from previous versions.