SDSC 2012 Summer Institute Tutorial

Tutorial at the SDSC 2012 Summer Institute - SDSC-SI

Tutorial Title: Developing Science Gateways using Apache Airavata API

Date: Thursday August 8th 2012

Venue: University of California at San Diego, SDSC Buillding

Software Requirements: Java, SSH Shell, Web Browser

Prerequisite Knowledge: General understanding of Cluster and Grid computing, Batch Job Submission, Science Portals/Gateways. Knowledge of Java & Web services is preferable but not required.

Description:

Science Gateways simplify access to XSEDE by providing easy to use familiar interfaces. Gateways abstract the complexity of various grid middleware that in turn absorb the heterogeneity of the underlying batch management systems. Gateway developers are challenge in building and maintaining gateways and are often overwhelmed in choosing middleware. Often these middleware solutions are black boxes to the developers that make them expend valuable man hours in debugging real-time problems. This tutorial will provide by means of number of hands on tutorials practical insights into gateway building aspects. Alongside the tutorial will provide insights into aspects of open community software development practices to not just use open software but how to contribute, get recognized and also have a say in future directions.

The tutorial will be based on Apache Airavata, a software toolkit to build science gateways. Airavata providing features to compose, manage, execute, and monitor small to large-scale applications and workflows on computational resources ranging from local clusters to national grids like XSDE and computing clouds like Amazon Elastic Compute Cloud. Airavata builds on general concepts of service oriented computing, distributed messaging, workflow composition and orchestration. Airavata suite include tools for workflow composition and monitoring. The stand out feature of workflow engine allows to interpret at each step providing dynamic interactive capabilities. The core capabilities provide ability to wrap command line-driven science applications and make them into robust, network- accessible services. Airavata Registry provides persistent data store. The gateway building toolkit also includes publish-subscribe based messaging system with features to incorporate clients behind firewalls and overcomes network glitches.

Target Audience and take home lessons:
The tutorial is targeted for Gateway developers who want to build a new gateway from scratch; for advanced developers who are investigating to improve a particular capability; and Campus Champions who can learn easier mechanisms to move data and execute jobs on XSEDE. The take home lessons include skills to execute compute jobs and workflows on XSEDE resources through grid interfaces; Wrap command line applications turning them into web accessible programmable interfaces and learn about open community development process to contribute to existing software with proper contribution governance models.

Starting up XBaya

Download XBaya - JNLP Link

or

Use the binary distribution (Mac/Linux):

1. Extract the Airavata binary distribution apache-airavata-0.4-incubating-SNAPSHOT-bin.zip to <AIRAVATA_HOME>.
2. Execute <AIRAVATA_HOME>/bin/xbaya-gui.sh

Setup Registry

Select menu,

    Registry -> Setup Airavata Registry...

Provide the following,

URL

    http://demo.airavata.org:8090/jackrabbit/rmi

                   or

    http://gf9.ucs.indiana.edu:8090/jackrabbit-webapp-2.4.0/rmi

Username

    admin

Password

    admin

Click "OK"

Register Applications in Ranger

To register a new application select menu,

    XBaya -> Register Application  menu.

Provide the following Application details

Application name:

    list_files                         (or any other unique name)

Application Parameters:   

Add Deployments

Click on the “New deployment” button. “New Application Deployment” form will appear.

Provide the following details

Application host

    ranger

Executable path

    /bin/ls

Scratch working directory

    /scratch/01437/ogce/test

Click on the “HPC Configuration...” button. The HPC Configuration Options form will appear. Provide the following details,

Job Type

    serial

Project Account Number

    TG-STA110014S

Queue Type

    normal

Click “Update”

Click “Add”

Click “Register”. You will see a Successfully saved confirmation message if the registration is successful.

After registering the Application successfully you will see your application in Left hand side Application Services list. Create a workflow using that service and run the workflow as if you did in previous workflow run.

Compose Workflow

To add a new workflow select menu,

    XBaya -> New Workflow.

A new Workflow editor will open along with the Workflow Properties dialog for that workflow. Enter the following details in the dialog

    Name

        retrieve_files_names

Click “OK”

From the left hand side “Component List” drag & drop the following to the Workflow editor.

2 x System Components -> Input

1 x System Components -> Output

1 x Application Services -> list_files

Connect output of “Input” components to inputs of “list_files_invoke”
Connect output of “list_files_invoke” component to input of “Output”

You may save the workflow by menu item

    XBaya -> Save (in order to save it in the local file system)

      or

    XBaya -> Export -> Workflow to Registry... (in order to save it in the Airavata Registry)

Running & Monitoring a Workflow

If the workflow is not already opened in XBaya, select menu,

    XBaya -> Open Workflow

       or

    XBaya -> Import -> Workflow from Registry

based on whether the workflow was saved in the file system or Airavata Registry.

Once the workflow is loaded on to XBaya select menu,

    Run -> Run on Interpreter Server...

Provide the following as values for the parameters

    path

        /bin

    options

        -al

    Experiment name

        Executables in ranger              (or any name you want to call it)

Click “Run”

The monitoring panel docked at the bottom of XBaya should get focus and show the notifications of the execution of the workflow.

The workflow loaded in XBaya should indicate the progress of the workflow execution.

Viewing Experiment Results

In XBaya select menu,

    View -> Airavata Registry

This will open a panel in XBaya docked at the left hand side. This will show the saved content in the Airavata Registry.

Expand the “Experiments” node to view the details of the experiment.

Next Steps : Executing your own applications

If you reached this step that means you have followed the instructions given here carefully and successfully launched a job in Ranger.

So now you might think that you want to run your own application installed on Ranger. In that case you wont be able to use this same XBaya instance (which was started through JNLP) to invoke your application. That is because, this demo uses a community account associated with the Science Gateway Group to submit the jobs to Ranger. Since you have installed your application through your Ranger account, you should configure backend Airavata server to talk to Ranger properly with your XSEDE credentials.

Please refer the blog post to configure your Airavata Server properly to submit jobs to Ranger.

If you run into any issues please contact airavata-user mailing list and post your question.