YARN Integration

Inregration with Yarn supports scheduling and running Apache Ignite nodes in a YARN cluster.
YARN is a resource negotiator which provides a general runtime environment providing all the essentials to deploy, run and manage distributed applications. Its resource manager and isolation helps getting the most out of servers.
For information about YARN refer to http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html

Ignite YARN Application

Deploying Apache Ignite cluster typically involves downloading the Apache Ignite distribution, changing configuration settings and starting the nodes up. Integration with YARN allows to avoid the actions. Apache Ignite Yarn Application allows to greatly simplify cluster deployment. The application consist from the following components: 

• Client downloads ignte distributive, puts necessary resources to HDFS, creates the necessary context for launching the task,  runs the ApplicationMaster process.
• Application master.  Once registration is successful the component will begin requesting of resource from Resource Manager to utilize resources for Apache Ignite nodes. The Application Master will maintain the Ignite cluster at desired total resources level (CPU, memory, etc.).
• Container - the entity that runs Ignite Node on slaves.

Running Ignite YARN application

For running Ignite Application requires YARN and Hadoop cluster are configured and running. For information on how to set up a the cluster please refer to http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html

1.  Download Ignite YARN application.
2.  Run the application.
    

   java -jar ignite-yarn.jar

   or

   java -jar ignite-yarn.jar properties.prop

   where properties.prop is a property file. Example property file:

   # The number of nodes in the cluster.
   IGNITE_NODE_COUNT=2
   
   # The number of CPU Cores for each Apache Ignite node.
   IGNITE_RUN_CPU_PER_NODE=1
   
   # The number of Megabytes of RAM for each Apache Ignite node.
   IGNITE_MEMORY_PER_NODE=2048
   
   # The version ignite which will be run on nodes.
   IGNITE_VERSION=1.0.6

   
   

3. In order to make sure that Application deployed correctly, do the following. Open YARN console at http://<hostname>:8088/cluster. If everything works OK then application with Ignition name.
   
4. Click on application id.
   
5. Retrieve logs from browser. To look through Ignite logs click on Logs
   
   
6. Click on stdout to get stdout logs and on stderr to get stderr logs.
   
   

Configuration