THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Page History
| Table of Contents |
|---|
General Assumptions
- Single Application Catalog will exists in Airavata to be used by all the gateways.
- Each gateway will be creating, updating, deleting, etc... their own application deployments in the supercomputers.
- Each gateway will have ADMIN users who would do above tasks in the Application Catalog
- Airavata ADMIN users will be able to view details entered through all the gateways using Airavata instance. They can also search and view application deployments;
E.g.:- Search for all applications which are in turned-on state in all supercomputers
- Search for application deployments of specific gateway
- If necessary Airavata ADMIN users will also be able to maintain application deployments across gateways.
- Applications and workflows will always be deployed, created (workflows will be created using XBAYA), etc... prior to entering in the Application Catalog.
Existing Descriptors
...
- In Airavata there are two ways of executing experiments in supercomputers/ computational resources (Trestles, Stampede, Amazon EC2, Big Red II, etc…).
- Execute applications - Single submission experiments
- Execute workflows - Multiple node OR single node experiment
- To execute experiment jobs of either above types; we rely on ‘Descriptors’ in Airavata to provide information required on application, application hosted supercomputers and input output parameters of the application.
- There are three types of descriptors;
- Host Descriptors
Specifies the application residing/hosting resource information. GRAM, GridFTP, Amazon EC2, etc…. - Service Descriptors
Input and output parameter information for an application - Application Descriptors
Specifies executable location information, deployment location information, scratch locations of output files, meta data specific for the application, meta data specific for the deployment (location where the application is deployed in the resource), etc…
- Host Descriptors
Introduction - Application Catalog
...
Create Application/Workflow
Create application includes many sub tasks in the Application Catalog. Creating an application instance extends to;
Entering application specific information such as;
Application version details
Deployment visibility to gateway users
Defining application specific parameters
Defining resource specific parameters required for the application execution
Define input and output data file sizes, file locations and scratch folder locations and also their folder sizes, etc for the application
Error file locations, log file locations ,etc…. for an application in residing resource
Application user group (Application is available only for a selected user group(s)
Maximum wall-time, number of processors, nodes per processor, etc….
Entering resource specific information such as;
Deployed resource/supercomputer information
Defining resource specific parameters such as application residing location
Queue information
Deployment information such as using MPI, GPU, Serial, etc…
Creating workflow consists;
Enter workflow information & resources where the workflow will be executed
Define workflow nodes and applications to be used at each nodes in the workflow
Defining subsequent input and output files
Application & workflow creation in the catalog will be a gateway ADMIN task.
Applications and workflows can be created by cloning & importing existing applications and workflows.
Creating applications and workflows will not make them available for users to execute their experiments.
Publish Application/Workflow
Application deployments and workflows which are CREATED will be published in order to be used by gateway users.
Publishing applications and workflows is a gateway ADMIN user level task.
At the time of publishing applications the system would validate
;Existence of at least 1 deployment of the application
Existence of at least 1 application and resource level
...
parameter
...
- ...
- ...
At the time of publishing workflows system should validate;
- Existence of at least one application in the workflow
Existence of at least one node in the workflow
- ...
- Existence of at least one application in the workflow
User can search within created applications in order to publish by providing;
Application ID/Workflow ID (Generate ID within Application Catalog)
Application Name
Resource
Application User Group
...
Search/List Application/Workflow
Gateway users (ADMIN, normal) and Airavata ADMIN users will search/list applications and workflows for different purposes. when searching/listing for applications and workflows different keys can be used;
Some examples;
...
Application ID/Workflow ID
Application name/Workflow Name
Resource
By ON/OFF flag
Status (CREATED & PUBLISHED)
By the special application user groups
...
- Gateway users or Airavata users may want to use above listing for auditing purposes.
...
Application/Workflow can be turned ON and OFF whenever required by the gateway or Airavata ADMIN users.
This functionality will be used when the application/workflow need to be temporarily unavailable for the users.
Applications/workflows which are published can be turned ON and OFF.
Turning ON and Off can be done at different levels. Such as;
Resource level
Application user group level
Application level
When an application or a workflow is turned OFF from a certain level it needs to be turned ON from the same level as well.
E.g.: When applications are turned OFF at resource level they cannot be turned ON individually; turning ON also will happen at resource level.
Read from Application Catalog
Airavata internal component can also query information from the Application Catalog. Orchestrator, Workflow Interpreter, GFac can query information required to formulate the tasks to be executed in the supercomputers.
Orchestrator will query Application Catalog to retrieve information required to submit single submission jobs into GFac. It will retrieve;
Information on application residing resource
Information on application specific parameters
If the experiment didn’t specify resource Orchestrator will select a resource from Application Catalog
…
Workflow interpreter will retrieve information required to submit jobs of the workflows.
Retrieve information on applications to execute for each node
Information on resources where applications are running
Application version information
...
...
GFac will also use information stored in the Application Catalog
Information on locations for input data files, output data files, error files and log files for each application
Information on application versions and other information required for application execution
Information to select the correct providers and handlers
...
Use Cases - Setting-up & maintenance of Application Catalog
...
Enter details of application deployment (WRF) in supercomputer (Big Red II) into the Application Catalog.
Create a new application deployment instance in the catalog
Add supercomputer detail which has it deployed
Define application specific, application deployment specific information and parameters required to use the application
Application specific information such as;
Application version
Input & output file sizes (maximum file size)
Input and output file locations (scratch locations)
duration of scratch location holding the output files
Error file and log file locations
Application deployment specific information;
Deployed application version
Deployed Application description
Deployed location
Application accessible providers (GSI-SSH)
- E.g.: Define echo application in Big Red II (or some other application)
- Step 1 : Define Big Red II details
- Host address : bigred2.uits.iu.edu
- Remote Access Protocol : GSI-SSH
- Port : 22
- Installed Path : /opt/torque/bin/
- Step 1 : Define Big Red II details
...
- Authentication : ???
- Step 2 : Define application details
- Input : [name: echo_input, type:String]
- Output : [name: echo_output, type:String]
- Executable Location : /bin/echo
- App version : 1.0
- Step 3 : Define Job details
- Scratch Location : /home/ogce/scratch
- Project Account : sds128
- Queue Type : normal
- CPU Count : 1
- Job Type : Serial
- Node Count : 1
- Processors Per Node : 1
- Max Wall Time : 10
Use Case II
Enter details of the application (Amber) which has the same version (V 12) deployed in multiple supercomputers (BigRed II & Stampede)
Create two instances of the application deployments in the catalog for each supercomputer deployment with application version information
Define general set of application parameters for the deployment in first supercomputer
Define resource specific set of parameters for the second supercomputer deployment
Use Case III
Enter details of the application (Amber) which has the same application version (V 12) deployed in BigRed II but with different deployment variations; mpi & gpu
Create two instances of the same application version in the catalog for Big Red II
Define one instance as MPI deployment and other as GPU deployment
Define the mandatory primary parameters and additional advance parameters for both instances
Use Case IV
Enter details of WRF application which has two different versions (cray/3.5 & 3.4.1) deployed in Big Red II & Quarry supercomputers
Create two instances of the application deployments in the catalog for each supercomputer deployment for each version
Use Case V
Create two instances of application Scorep with two different versions (gnu/1.2beta & gnu/1.2.3) which are deployed in the Big Red II
Create two instances of the application deployment in Big Red II with different versions
One version of Deployment will have general set of application parameters
The second version will have supercomputer specific set of parameters
Use Case VI
Create an application deployment instance which will not be visible to the gateway user to select at the time of creating experiment.
Create the application deployment with a flag to say ‘ NOT visible’
Use Case VII
Create an application deployment instance which can only be used in workflows; not available for single submissions.
Create the application deployment with a flag to say ‘ ONLY for workflows’
Use Case VIII
Create an application deployment which has its deployments in supercomputers hidden from the gateway users
Create the application deployment with a flag to say ‘ Hidden deployments’
NOTE: Gateway users can only select the application but not the supercomputer; user will not know where the application is running
...
Enter details of an workflow and its nodes which executes using multiple applications deployed in Big Red II
Create the workflow in the catalog and define the supercomputer as supercomputer-I
Define the applications which will be used in the workflow
Use Case X
Enter details of the workflow which executes using multiple applications from two different supercomputers (Stampede & Lonestar)
Create the workflow and its nodes in the catalog and define the multiple supercomputers which runs the applications
Define the applications which will be used in the workflow from each supercomputer
Publish Application/Workflow
...
Assumption: Each gateway will enter details of their application deployments in to the application catalog.
E.g. if two gateways are using the same application version deployed in the same supercomputer there will be two records in the catalog; one record for each gateway.
...
Update details of a created workflow which executes using multiple applications deployed in the supercomputsupercomputer
Search for the workflow in the supercomputer
Add a another application in to the newly defined workflow node
...
Update details of a published workflow which executes using multiple applications deployed in the supercomputsupercomputer
Search for the workflow in the supercomputer
Add a another application in to the newly defined workflow node
...
Search for the workflow in the supercomputer-I & supercomputer-II
Add a new application to the new workflow node which is in the supercomputer-I
Remove existing application from a existing node which is in the supercomputer-II
Clone Application/Workflow
Use Case I
Clone an existing application deployment which is in CREATED state and its owned by the same gateway
Search for the Application using application ID/Resource/Status/etc….
Clone the application and create the new application
The new application will get saved in CREATED state
Use Case II
Clone an existing application which is in PUBLISHED state and its owned by the same gateway
Search for the Application using application ID/Resource/Status/etc….
Clone the application and create the new application
The new application will get saved in CREATED state
Use Case III
Clone an existing application which is in CREATED state and its owned by the a different gateway
Search for the Application using application ID/Resource/Status/etc….
Clone the application to create the new application
Cloning between gateways will be restricted
Use Case IV
Clone an existing application which is in CREATED state but flagged as non-cloneable and owned by the same gateway
Search for the Application using application ID/Resource/Status/etc….
Clone the application to create the new application
Cloning will be restricted as the primary application is flagged as non-cloneable
Use Case V
Clone an existing workflow which is in CREATED state and its owned by the same gateway
Search for the workflow using workflow ID/Status/etc….
Clone the workflow to create the new workflow
The new workflow will get saved in CREATED state
Use Case VI
Clone an existing workflow which is in CREATED state and its owned by a different gateway
Search for the workflow using workflow ID/Status/etc….
Clone the workflow to create the new workflow
Cloning between gateways will be restricted
Use Case VII
Clone an existing workflow which is in PUBLISHED state but flagged as non-cloneable and owned by the same gateway
Search for the workflow using workflow ID/Status/etc….
Clone the workflow to create the new workflow
Cloning will be restricted as the primary workflow is flagged as non-cloneable
Import Application
NOTE: Importing applications & workflows are done ONLY between gateways.
Use Case I
Import an application in CREATED state from a different gateway and create a new application
Search for an application from other gateways using gateway ID/Application ID/Application Name, /Resource/etc…
Select the application and import
A new instance will get imported leaving the primary application in the specific gateway
The new application will get saved as a CREATED application
Use Case II
Import an application in PUBLISHED state from a different gateway and create a new application
Search for an application from other gateways using gateway ID/Application ID/Application Name, /Resource/etc…
Select the application and import
A new instance will get imported leaving the primary application in the specific gateway
The new application will get saved as a CREATED application
Use Case III
Import an application in PUBLISHED state from a different gateway which is flagged as non-importable
Search for an application from other gateways using gateway ID/Application ID/Application Name, /Resource/etc…
Select the application and import
Importing the application details will fail as the primary application is tagged as non-importable
Use Case IV
Import a workflow in CREATED state from a different gateway and create a new workflow
Search for the workflow from other gateways using gateway ID/workflow ID/workflow Name, /etc…
Select the workflow and import
A new instance will get imported leaving the primary workflow in the specific gateway
The new workflow will get saved as a CREATED workflow
Use Case V
Import a workflow in CREATED state from a different gateway which is tagged as non-importable
Search for the workflow from other gateways using gateway ID/workflow ID/workflow Name, /etc…
Select the workflow and import
Importing the workflow details will fail as the primary workflow is tagged as non-importable
Delete Application/Workflow
Use Case I
Delete a selected application deployment from the resource. The particular application has only a single deployment
Search for the application by giving the application ID
Delete the selected application deployment
The application is no longer available for the gateway users.
NOTE: Existing experiments which used or currently using the application will not have any impact
Use Case II
Delete a selected application deployment from the resource. The particular application has multiple deployments in multiple resources
Search for the application by giving the application ID
Delete the selected application deployment
The application is no longer available for the gateway users.
NOTE: Existing experiments which used or currently using the application will not have any impact
Use Case III
Delete a selected application deployment from the resource. The particular application has two different application versions deployed in the same resource. Only one of those is required to be deleted.
Search for the application by giving the resource ID
Delete the selected application deployment
The application is no longer available for the gateway users.
NOTE: Existing experiments which used or currently using the application will not have any impact. New users can only use the remaining version in the same resource.
Use Case IV
Delete all deployments of a selected application. Application has multiple versions and multiple deployments in multiple resources.
Search for all the application deployments (all versions)
Delete them all
The application is no longer available for all the gateway users using Airavata.
NOTE: This is possibly a Airavata ADMIN task.
Use Case V
Delete all deployments of a selected application. Application has multiple versions and multiple deployments in multiple resources.
Search for all the application deployments (all versions)
Delete them all
The application is no longer available for the gateway.
NOTE: This is possibly a Gateway ADMIN task. Only the application deployments of that particular gateway will get affected. Other gateways would have their deployments of the same application remaining.
Use Cases - Application Catalog in Use
Use Case I
Turn off a specific application from all the supercomputers it is been deployed in
Search for all the application deployments (different versions as well)
Temporarily turn-off all of them
Gateway users will not be able to create or launch experiments using the particular application
NOTE: Any ongoing experiments will not be interrupted. This is a gateway ADMIN level task.
Use Case II
Turn off a specific application deployment from a supercomputer it is been deployed in
Search for the application by providing the resource
Temporarily turn-off the particular application deployment
Users will not be able to create or launch experiments using the particular application deployment in the resource
NOTE: Any ongoing experiments will not be interrupted. This is a gateway ADMIN level task.
Use Case III
Turn off all the applications in Stampede for 48 hours as it is not available to launch experiments
Search for all the applications by providing the resource
Turn-off all the applications in the resource by giving start and end date/time
Users will not be able to create ,launch, list experiments in the particular resource for 48 hours
After 48 hours applications will be auto turned ON to use
Use Case IV
Turn on the specific application from all the supercomputers it is been deployed in
Search for the application by providing the application ID
Turn-on through method call in all resources
Users will be able to create,launch, list experiments using the particular application
Use Case V
Turn on the specific application version which was turned off from only one supercomputer
Search for the application version by providing the application ID
Turn-on through method call in all resources
Users will be able to create,launch, list experiments using the particular application
Use Case VI
Turn off a specific workflow by searching for the using the workflow ID
Temporarily turn-off the workflow
Gateway users will not be able to create or launch experiments using the particular workflow
NOTE: Any ongoing experiments will not be interrupted. This is a gateway ADMIN level task.
Use Case VII
Turn off all the workflows which runs one or many applications residing in Big Red II
Search for the applications in Big Red II which are used in the workflow
Temporarily turn-off all the workflows using applications from the supercomputer
Users will not be able to create or launch experiments using the particular application deployment in the resource
NOTE: Any ongoing experiments will not be interrupted. This is a gateway ADMIN level task.
Use Case VIII
Turn on the specific workflow which was turned OFF at workflow level
Search for the workflow by providing the workflow ID
Turn-on the workflow
Users will be able to create,launch, list experiments using the particular application
Use Case IX
Select a workflow which is turned of at resource level and turn it on
Search for all the workflow by providing the workflow ID/workflow name
Turn-on all the workflow
workflow cannot be turned on as it was turned OFF at resource level. use has to turn it ON at resource level
Use Case X
List all the applications/workflows in the Big Red II which are turned-on
Search for all the application deployments by giving ‘turn-on’ as current status
Use Case XI
List all deployments of the ‘application WRF’ which are in turned-on state
Search for all the application deployments of ‘application A’ & has state as ‘turn-on’
Use Case XII
List all workflows which runs application WRF and in turned-on state
Search for all the workflows which runs WRF & has state as ‘turn-on’
Use Case XIII
List all the applications/workflows in the supercomputer-I which are turned-off
Search for all the application deployments by giving ‘turn-off’ as current status
Use Case XIV
List all deployments of the ‘application A’ which are in turned-off state
Search for all the application deployments of ‘application A’ & has state as ‘turn-off’
Use Case XV
List all workflows which are in turned-off state
Search for all the workflows with state as ‘turn-off’
Use Case XVI
List all deployments of the ‘application A’ irrespective of the application active state
Search for all the application deployments of ‘application A’
Use Case XVII
List all deployments of the particular gateway which are in ‘turn-on’ state
Search for all active application deployments
Use Case XVIII
List all deployments of the particular gateway which are in ‘turn-off’ state
Search for all inactive application deployments
Overview
Content Tools
Apps