Beam library developers and Beam users would appreciate this : )

This project involves prototyping a few different solutions, so it will be large.

Our Commons components use Commons Skin, a skin, or theme, for Apache Maven Site.

Our skin uses Bootstrap 2.x, but Bootstrap is already at 5.x release, and we are missing several improvements (UIUX, accessibility, browser compatibility) and JS/CSS bugs fixed over the years.

Work happening on Apache Maven Skins. Maybe we could adapt/use that one?

https://issues.apache.org/jira/browse/MSKINS-97

The Airavata Django Portal [1] allows users to create, execute and monitor computational experiments. However, when a user wants to then post-process or visualize the output of that computational experiment they must then download the output files and run tools that they may have on their computer or other systems. By integrating with JupyterHub the Django Portal can give users an environment in which they can explore the experiment's output data and gain insights.

The main requirements are:

• from the Django Portal a user can click a button and navigate to a JupyterHub instance that the user is immediately logged into using single sign on
• the user can save the Jupyter notebook and later retrieve it
• the user's files are available within the context of the running Jupyter instance
  • ideally users can also generate new outputs in the Jupyter instance and have them saved back in their portal data storage
• users can share their notebooks with other portal users
• (bonus) portal admins can suggest notebooks to use with specific applications so that with one click a user can open an experiment in a provided notebook
• users can manage their notebooks and can, for example, clone a notebook

[1] https://github.com/apache/airavata-django-portal

Integrate Apache Superset (https://superset.apache.org/) to visualize Airavata Catalogs (https://github.com/apache/airavata/tree/master/modules/registry) 

• Examples like this and stack overflow threads seem to indicate it is possible - https://medium.com/@s.akashb/apache-superset-integration-with-keycloak-a302840c290c 

• Integrate with Custos 
• We can start out by directly interfacing with MariaDB, but need to explore if we can write a superset DB Driver following the Hive example - https://github.com/apache/superset/blob/0409b12a55e893d88f6e992a7df247841a2da8f0/superset/db_engines/hive.py

1. UI Framework 
   1. To host the jupyter environment we will need to envolop the notebooks in a user interface and connect it with Apache Airavata services 
   2. Leverage Airavata communications from within the Django Portal - https://github.com/apache/airavata-django-portal 
   3. Explore if the platform is better to be developed as VSCode extensions leveraging jupyter extensions like - https://github.com/Microsoft/vscode-jupyter
   4. Alternatively, explore developing a standalone native application using ElectronJS
2. Draft up a platform architecture - Airavata based infrastructure with functionality similar to collab. 
3. Authenticate with Airavata Custos Framework - https://github.com/apache/airavata-custos 
4. Extend Notebook filesystem using the virtual file system approaching integration with Airavata based storage and catalog
5. Make the notebooks registered with Airavata app catalog and experiment catalog. 

Advanced Possibilities:

Explore Multi-tenanted JupyterHub 

• Can K8 namespace isolation accomplish?
• Make deployment of Jupyter support as part of the default core
• Data and the user-level tenancy can be assumed, how to make sure infrastructure can isolate them, like not one gateway crashing a hosting environment.

1. How to leverage computational resources jupypter hub

Gateway admins need period reports for various reporting and planning. 

Features Include:

• Compute resources across that had at least one job submitted during the period <start date - End date>
• User groups created within a given period and how many users are in those and with permission levels and also number of jobs each user have submitted.
• List applications and number of jobs for each applications for a given period and group them by job status.
• Number of users that at least submitted a single job for the period <start date - End date>
• Total number of Unique Users
• User Registration Trends
• Number of experiments for a given period <Start date - End date> grouped by the experiment status
• The total cpu-hours used by a users, sorted, quarterly, plotted over a period of time
• The total cpu-hours consumed by application, sorted, quarterly, plotted over a period of time

As discussed on the architecture mailing list [1] and summarized at [2], Airavata will need to develop a metascheduler. In the short term, a user request (demeler, gobert) is to have airavata throttle jobs to resources. In the future more informed scheduling strategies needs to be integrated. Hopefully, the actual scheduling algorithms can be borrowed from third party implementations.

[1] - http://markmail.org/message/tdae5y3togyq4duv
[2] - https://cwiki.apache.org/confluence/display/AIRAVATA/Airavata+Metascheduler

Complete all transports in MFT

• Currently SCP, S3 is known to work
• Others need effort to optimize, test, and declare readiness
• Develop a complete a fully functional MFT Command-line interface
• Have a feature-complete Python SDK
• A minimum implementation will be prvoided, students need to complete it and test it. 

Custos does not have the capabilities to efficiently backup and restore a live instance. This is essential for high available services. 

Using SEAGrid Rich Client as an example, develop a native application based on electronJS to mimic Airavata Django Portal.

Reference example - https://github.com/SciGaP/seagrid-rich-client