Application Catalog Use Cases

General Assumptions

1. Single Application Catalog will exists in Airavata to be used by all the gateways.

2. Each gateway will be creating, updating, deleting, etc... their own application deployments in the supercomputers.

3. Each gateway will have ADMIN users who would do above tasks in the Application Catalog

4. Airavata ADMIN users will be able to view details entered through all the gateways using Airavata instance. They can also search and view application deployments;

   E.g.:

• • Search for all applications which are in turned-on state in all supercomputers

  • Search for application deployments of  specific gateway

If necessary Airavata ADMIN users will also be able to maintain application deployments across gateways. Applications and workflows will always be deployed, created (workflows will be created using XBAYA), etc... prior to entering in the Application Catalog.

Existing Descriptors

1. In Airavata there are two ways of executing experiments in supercomputers/ computational resources (Trestles, Stampede, Amazon EC2, Big Red II, etc…).

• • Execute applications - Single submission experiments

  • Execute workflows - Multiple node OR single node experiment

1. To execute experiment jobs of either above types; we rely on ‘Descriptors’ in Airavata to provide information required on application, application hosted supercomputers and input output parameters of the application.

2. There are three types of descriptors;

• • Host Descriptors

 Specifies the application residing/hosting resource information. GRAM, GridFTP, Amazon EC2, etc….

• • Service Descriptors

  Input and output parameter information for an application

• • Application Descriptors

Specifies executable location information, deployment location information, scratch locations of output files, meta data specific for the application, meta data specific for the deployment (location where the application is deployed in the resource), etc…

Introduction - Application Catalog

1. So what is more with Application Catalog? With application catalog user will be able to;

• • Store same set of information as with descriptors

  • Have number of new API methods to access and manage the applications as required

    • Create Application

    • Publish Application

    • Search/List Application

    • Update Application

    • Clone Application

    • Import Application

    • Delete Application

    • Turn On/Turn Off Application

  • Read from Application Catalog  - Other components will read from the Catalog in order to get relevant information required for experiments.

1. In descriptors we have same set of parameters captured for clouds, supercomputers irrespective of the necessity. With the Catalog ‘the not required parameters’ will be ignored when creating, updating application information.

2. Airavata components; Orchestrator, Workflow Interpreter & GFac will connect to Application Catalog to obtain information required for executing jobs at resources.

3. Back-end services of Application Catalog. These will be automated services which would run periodically as specified by the Airavata ADMIN users.

• • Verifying application location in the resource(s) and actual existence

  • Verifying application version in the resource

  • Verify availability of space on scratch location folders.

Status of Applications & Workflows

API Methods

Create Application/Workflow

1. Create application includes many sub tasks in the Application Catalog. Creating an application instance extends to;

• • Entering application specific information such as;

    • Application version details

    • Deployment visibility to gateway users

    • Defining application specific parameters

    • Defining resource specific parameters required for the application execution

    • Define input and output data file sizes, file locations and scratch folder locations and also their folder sizes, etc for the application

    • Error file locations, log file locations ,etc…. for an application in residing resource

    • Application user group (Application is available only for a selected user group(s)

    • Maximum wall-time, number of processors, nodes per processor, etc….

  • Entering resource specific information such as;

  • • Deployed resource/supercomputer information

    • Defining resource specific parameters such as application residing location

    • Queue information

    • Deployment information such as using MPI, GPU, Serial, etc….

1. Creating workflow consists;

• • Enter workflow information & resources where the workflow will be executed

  • Define workflow nodes and applications to be used at each nodes in the workflow

  • Defining subsequent input and output files

1. Application & workflow creation in the catalog will be a gateway ADMIN task.

2. Applications and workflows can be created by cloning & importing existing applications and workflows.

3. Creating applications and workflows will not make them available for users to execute their experiments.

Publish Application/Workflow

1. Application deployments and workflows which are CREATED will be published in order to be used by gateway users.

2. Publishing applications and workflows is a gateway ADMIN user level task.

3. At the time of publishing applications the system would validate;

• • Existence of at least 1 deployment of the application

  • Existence of at least 1 application and resource level parameters

  • …

1. At the time of publishing workflows system should validate;

• • Existence of at least one application in the workflow

  • Existence of at least one node in the workflow

  • ...

1. User can search within created applications in order to publish by providing;

• • Application ID/Workflow ID (Generate ID within Application Catalog)

  • Application Name

  • Resource

  • Application User Group

  • ...