You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 83 Next »


This section describes the following:

Prerequisite Software

Setting up Ambari SCOM assumes the following prerequisite software:

  • Apache Hadoop cluster (HDFS and MapReduce) 1.x 1
  • JDK 1.6
  • Microsoft SQL Server 2012
  • Microsoft JDBC Driver 4.0 for SQL Server 2
  • Microsoft System Center Operations Manager (SCOM) 2012 SP1 or later
  • System Center Monitoring Agent installed on Watcher Node 3

1 Ambari SCOM has been tested with a Hadoop cluster based on Hortonworks Data Platform 1.3 for Windows ("HDP for Windows")
2 Obtain the Microsoft JDBC Driver 4.0 for SQL Server JAR file (sqljdbc4.jar) at http://technet.microsoft.com/en-us/library/ms378749.aspx
3 See Microsoft TechNet topic for Managing Discovery and Agents. Minimum Agent requirements .NET 4 and PowerShell 2.0 + 3.0

Package Contents

├─ ambari-scom-x.y.z
├── README.md
├── LICENSE
├── NOTICE
├── server.zip
├── metrics-sink.zip
├── ambari-scom.msi
└── mp.zip

File

Name

Description

server.zip

Server Package

Contains the required software for configuring the Ambari SCOM Server software.

See Installing and Configuring Ambari SCOM Server

metrics-sink.zip

Metrics Sink Package

Contains the required software for manually configuring SQL Server and the Hadoop Metrics Sink.

See Configuring SQL Server
See Configuring Hadoop Metrics Sink

ambari-scom.msi

MSI Installer

The Ambari SCOM MSI Installer for configuring the Ambari SCOM Server and Hadoop Metrics Sink

See Running the MSI Installer

mp.zip

Management Pack Package

Contains the Ambari SCOM Management Pack software.

See Installing Ambari SCOM Management Pack


Ambari SCOM Server Installation

The Ambari SCOM Management Pack must connect to an Ambari SCOM Server to retrieve cluster metrics. Therefore, you need to have an Ambari SCOM Server running in your cluster. If you have already installed your Hadoop cluster (including the Ganglia Service) with Ambari (minimum Ambari 1.2.5) and have an Ambari Server already running + managing your cluster, you can use that Ambari Server and point the Management Pack that host. You can proceed directly to Installing Ambari SCOM Management Pack and skip these steps to install an Ambari SCOM Server. If you do not have an Ambari Server running + managing your cluster, you must install an Ambari SCOM Server using one of the methods described below.

The following methods are available for installing Ambari SCOM Server:

  • Manual Installation - This installation method requires you to configure the SQL Server database, setup the Ambari SCOM Server and configure the Hadoop Metrics Sink. This provides the most flexible install option based on your environment. See Manual Installation for more information.
  • MSI Installation - This installation method installs the Ambari SCOM Server and configures the Hadoop Metrics Sink on all hosts in the cluster automatically using an MSI Installer. After launching the MSI, you provide information about your SQL Server database and the cluster for the installer to handle configuration. See MSI Installation for more information.

Manual Installation

Configuring SQL Server

  1. Configure an existing SQL Server instance for "mixed mode" authentication.
  2. Confirm SQL Server is installed with TCP/IP active and enabled. (default port: 1433)
  3. Create a user and password. Remember this user and password as this will be the account used by the Hadoop metrics interface for capturing metrics. (default user: sa)
  4. Extract the contents of the metrics-sink.zip package to obtain the DDL script.
  5. Create the Ambari SCOM database schema by running the Hadoop-Metrics-SQLServer-CREATE.ddl script.

The Hadoop Metrics DDL script will create a database called "HadoopMetrics".

Configuring Hadoop Metrics Sink

Preparing the Metrics Sink

  1. Extract the contents of the metrics-sink.zip package to obtain the metrics-sink-version.jar file.
  2. Obtain the Microsoft JDBC Driver 4.0 for SQL Server sqljdbc4.jar file (see Prerequisite Software).
  3. Copy sqljdbc4.jar and metrics-sink-version.jar to each host in the cluster. For example, copy to C:\Ambari\metrics-sink-version.jar and C:\Ambari\sqljdbc4.jar
    on each host.

Setup Hadoop Metrics2 Interface

  1. On each host in the cluster, setup the Hadoop metrics2 interface to use the SQLServerSink.
  2. Edit the hadoop-metrics2.properties file (located in the {C:\hadoop\install\dir}\bin folder of each host in the cluster): 
    *.sink.sql.class=org.apache.hadoop.metrics2.sink.SqlServerSink

namenode.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMetrics;user=[user];password=[password]
datanode.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMetrics;user=[user];password=[password]
jobtracker.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMetrics;user=[user];password=[password]
tasktracker.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMetrics;user=[user];password=[password]
maptask.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMetrics;user=[user];password=[password]
reducetask.sink.sql.databaseUrl=jdbc:sqlserver://[server]:[port];databaseName=HadoopMetrics;user=[user];password=[password]

    In the above SQL statement:

    server = the SQL Server hostname
    port = the SQL Server port (for example, 1433)
    user = the SQL Server user (for example, sa)
    password = the SQL Server password (for example, BigData1)

  3. Update the Java classpath for each Hadoop service to include the metrics-sink.jar and sqljdbc4.jar files.

    1. Example: Updating the Java classpath for HDP for Windows clusters

      The service.xml files will be located in the {C:\hadoop\install\dir}\bin folder of each host in the cluster. The Java classpath is specified for each service in the <arguments> element of the service.xml file.

      For example, to update the Java classpath for the NameNode component, edit the C:\hadoop\bin\namenode.xml file. 

      ...
<arguments>... -classpath ...;C:\Ambari\metrics-sink-1.2.5.0.9.0.0-58.jar;C:\Ambari\sqljdbc4.jar ...</arguments>
...
  4. Restart Hadoop for these changes to take affect.

Verify Metrics Collection

  1. Confirm metrics are being captured in the SQL Server database by querying the MetricRecord table: 
    select * from HadoopMetrics.dbo.MetricRecord

    In the above SQL statement, HadoopMetrics is the database name.

Installing and Configuring Ambari SCOM Server

Running the Server

  1. Designate a machine in the cluster to run the Ambari SCOM Server.
  2. Extract the contents of the server.zip package to obtain the Ambari SCOM Server packages.
    ├── ambari-scom-server-version-conf.zip
    ├── ambari-scom-server-version-lib.zip
    └── ambari-scom-server-version.jar
  3. Extract the contents of the ambari-scom-server-version-lib.zip package to obtain the Ambari SCOM dependencies.
  4. Extract the contents of the ambari-scom-server-version-conf.zip package to obtain the Ambari SCOM configuration files.
  5. From the configuration files, edit the ambari.properties file: 
    scom.sink.db.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
scom.sink.db.url=jdbc:sqlserver://[server]:[port];databaseName=HadoopMetrics;user=[user];password=[password]

    In the above configuration:

    server = the SQL Server hostname
    port = the SQL Server port (for example, 1433)
    user = the SQL Server user (for example, sa)
    password = the SQL Server password (for example, BigData1)

  6. Run the org.apache.ambari.scom.AmbariServer class from the Java command line to start the Ambari SCOM Server.

    Be sure to include the following in the classpath:

    • ambari-scom-server-version.jar file
    • configuration folder containing the Ambari SCOM configuration files
    • lib folder containing the Ambari SCOM dependencies
    • folder containing the clusterproperties.txt file from the Hadoop install. For example, c:\hadoop\install\dir
    • sqljdbc4.jar SQLServer JDBC Driver file (see Prerequisite Software)
    For example: 
    java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Xms512m -Xmx2048m -cp "c:\ambari-scom\server\conf;c:\ambari-scom\server\lib;c:\jdbc\sqljdbc4.jar;c:\hadoop\install\dir;c:\ambari-scom\server\ambari-scom-server-1.2.5.0.9.0.0-53.jar" org.apache.ambari.scom.AmbariServer

    In the above command, be sure to replace the Ambari SCOM version in the ambari-scom-server-version.jar and replace c:\hadoop\install\dir with the folder containing the clusterproperties.txt file.

Verify the Server API

  1. From a browser access the API 
    http://[ambari-scom-server]:8080/api/v1/clusters
  2. Verify that metrics are being reported. 
    http://[ambari-scom-server]:8080/api/v1/clusters/ambari/services/HDFS/components/NAMENODE

MSI Installation

Configuring SQL Server

  1. Configure an existing SQL Server instance for "mixed mode" authentication.
  2. Confirm SQL Server is installed with TCP/IP active and enabled. (default port: 1433)
  3. Create a user and password. (default user: sa)

Running the MSI Installer

  1. Designate a machine in the cluster to run the Ambari SCOM Server.
  2. Extract the contents of the server.zip package to obtain the Ambari SCOM Server packages.
  3. Run the ambari-scom.msi installer. The "Ambari SCOM Setup" dialog appears:
  4. Provide the following information:

    Field

    Description

    Ambari SCOM package directory

    The directory where the installer will place the Ambari SCOM Server packages.
    For example: C:\Ambari

    SQL Server hostname

    The hostname of the SQL Server instance for Ambari SCOM Server to use to store Hadoop metrics.

    SQL Server port

    The port of the SQL Server instance.

    SQL Server login

    The login username.

    SQL Server password

    The login password

    Path to SQL Server JDBC Driver (sqljdbc4.jar)

    The path to the JDBC Driver JAR file.

    Path to the cluster layout file (clusterproperties.txt)

    The path to the cluster layout properties file.

  5. You can optionally select to Start Services
  6. Click Install
  7. After completion, links are created on the desktop to "Start Ambari SCOM Server", "Browse Ambari API" and "Browse Ambari API Metrics". After starting the Ambari SCOM Server, browse the API and Metrics to confirm the server is working properly.

The MSI installer installation log can be found at C:\AmbariInstallFiles\AmbariSetupTools\ambari.winpkg.install.log

Installing Ambari SCOM Management Pack

Before installing the Management pack, be sure to install the Ambari SCOM Server and Configure the Hadoop Metrics Sink using the Ambari SCOM Server Installation instructions above.

Import the Management Pack

Perform the following to import the Ambari SCOM Management Pack into System Center Operations Manager.

  1. Extract the contents of the mp.zip package to obtain the Ambari SCOM management pack (.mpb) files.
  2. Ensure Windows Server 2012 running SCOM with SQL Server (full text search).
  3. Open System Center Operations Manager.
  4. Go to Administration -> Management Packs.
  5. From the Tasks panel, select Import Management Packs...
  6. In the Import Management Packs dialog, select Add -> Add from disk...
  7. You are prompted to search the Online Catalog. Click "No".
  8. Browse for the Ambari SCOM management pack files.
  9. Select the following files: 
    Ambari.SCOM.Monitoring.mpb
Ambari.SCOM.Management.mpb
Ambari.SCOM.Presentation.mpb
  10. Click "Open"
  11. Review the Import list and click "Install".
  12. The Ambari SCOM Management Pack installation will start.

Create Run As Account

Perform the following to configure a account to use when the Ambari SCOM Management Pack talks to the Ambari SCOM Server.

  1. After Importing the Management Pack is complete, go to Administration -> Run As Configuration -> Accounts.
  2. In the Tasks panel, select "Create Run as Account..."
  3. You are presented with the Create Run As Account Wizard.
  4. Go thru the wizard, select Run As account type "Basic Authentication".
  5. Give the account a Display name and click "Next".
  6. Enter the account name and password for the Ambari SCOM Server. This account will be used to connect to the Ambari SCOM Server to access the Ambari REST API. Default is account name is "admin" and password is "admin".
  7. Click "Next"
  8. Select the "Less secure" distribution security option.
  9. Click "Next" and complete the wizard.

Configure the Management Pack

Perform the following to configure the Ambari SCOM Management Pack to talk to the Ambari SCOM Server.

  1. Go to Authoring -> Management Pack Templates -> Ambari SCOM
  2. In the Tasks panel, select "Add Monitoring Wizard".
  3. Select monitoring type "Ambari SCOM"
  4. Provide a name and select the destination management pack (see Best Practice).
  5. Provide the Ambari URI with is the address of the Ambari SCOM Server in the format: 
    http://[ambari-scom-server]:8080/api/

    In the above Ambari URI, ambari-scom-server is the Ambari SCOM Server.

  6. Select the Run As Account that you created in Create Run As Account.
  7. Select "Watcher Node". If node is not listed, click "Add" and browse for the node. Click "Next".
  8. Complete the Add Monitoring Wizard and proceed to the Monitoring Scenarios for information on using the management pack.

Best Practice: Create Management Pack for Customizations

By default, Operations Manager saves all customizations such as overrides to the Default Management Pack. As a best practice, you should instead create a separate management pack for each sealed management pack you want to customize.

When you create a management pack for the purpose of storing customized settings for a sealed management pack, it is helpful to base the name of the new management pack on the name of the management pack that it is customizing, such as Ambari SCOM Customizations.

Creating a new management pack for storing customizations of each sealed management pack makes it easier to export the customizations from a test environment to a production environment. It also makes it easier to delete a management pack, because you must delete any dependencies before you can delete a management pack. If customizations for all management packs are saved in the Default Management Pack and you need to delete a single management pack, you must first delete the Default Management Pack, which also deletes customizations to other management packs.

  • No labels