Currently, all Airavata services are packaged and deployed as Java bundles. The goal is to containerize each service by wrapping it within a Dockerfile, allowing seamless deployment on container-enabled resources while also enabling local execution for development purposes.

This enhancement has potential to improve deployment consistency, simplify dependency management, and provide greater flexibility in running Airavata services across different environments, for both testing and production use cases.

Objective

Enhance the current development workflow by incorporating a simulated High-Performance Computing (HPC) environment into Apache Airavata's existing Integrated Development Environment (IDE) integration. This will enable developers to test and validate features locally without relying on physical HPC resources.

Requirements

• Simulated HPC Environment Integration
  • Dockerized Slurm Simulation: Develop a Docker container that emulates an HPC environment using Slurm, facilitating the testing of job scheduling and execution.​
  • Seamless IDE Integration: Ensure that this simulated environment integrates smoothly with the existing IDE setup, allowing developers to initiate and monitor jobs as they would in a real HPC setting.

• Development of Comprehensive Test Scenarios
  • Job Submission Tests: Create scripts to test various job submission scenarios, including successful executions, intentional failures, and long-running processes.​
  • Feature Validation: Ensure that all features exposed by Apache Airavata can be tested within this simulated environment.

• User-Friendly Setup
  • Simplified Configuration: Design the setup process to require minimal configuration, enabling developers to initiate the environment and execute tests with just a few commands

Develop a devops dashboard to monitor Apache Airavata services, enabling real-time tracking of service health, uptime, and logs independent of the science gateway(s).
This centralized tool will help administrators efficiently monitor service performance and troubleshoot issues. The dashboard will feature a user-friendly monitoring UI that displays real-time status updates and logs for each service.

Proposed Solution:

• A logging subproces alongside each service, pushing logs to an external service.
• A devops dashboard that aggregates the logs and provides a unified view into the system.
• API calls from devops dashboard to each service, for proactive health-checking.
• Ability to monitor multiple gateways from the same dashboard.

Objective

Enhance the current development workflow by incorporating a simulated High-Performance Computing (HPC) environment into Apache Airavata's existing Integrated Development Environment (IDE) integration. This will enable developers to test and validate features locally without relying on physical HPC resources.

Requirements

• Simulated HPC Environment Integration
  • Dockerized Slurm Simulation: Develop a Docker container that emulates an HPC environment using Slurm, facilitating the testing of job scheduling and execution.​
  • Seamless IDE Integration: Ensure that this simulated environment integrates smoothly with the existing IDE setup, allowing developers to initiate and monitor jobs as they would in a real HPC setting.

• Development of Comprehensive Test Scenarios
  • Job Submission Tests: Create scripts to test various job submission scenarios, including successful executions, intentional failures, and long-running processes.​
  • Feature Validation: Ensure that all features exposed by Apache Airavata can be tested within this simulated environment.

• User-Friendly SetupSimplified Configuration: Design the setup process to require minimal configuration, enabling developers to initiate the environment and execute tests with just a few commands

Embed a researcher-focused dashboard to group and preview experiments in the Django portal. The goal is to improve how past experiment runs can be tracked, grouped, and arranged for faster lookup and insights.

Proposed Improvements:

• Group submitted experiments by project, application, allocation, etc.
• Clean, customizable dashboard elements (e.g., charts) to preview past experiments

Develop a devops dashboard to monitor Apache Airavata services, enabling real-time tracking of service health, uptime, and logs independent of the science gateway(s).
This centralized tool will help administrators efficiently monitor service performance and troubleshoot issues. The dashboard will feature a user-friendly monitoring UI that displays real-time status updates and logs for each service.

Proposed Solution:

• A logging subproces alongside each service, pushing logs to an external service.
• A devops dashboard that aggregates the logs and provides a unified view into the system.
• API calls from devops dashboard to each service, for proactive health-checking.
• Ability to monitor multiple gateways from the same dashboard.

Currently, all Airavata services are packaged and deployed as Java bundlesEmbed a researcher-focused dashboard to group and preview experiments in the Django portal. The goal is to improve how past experiment runs can be tracked, grouped, and arranged for faster lookup and insights.

Proposed Improvements:

• Group submitted experiments by project, application, allocation, etc.
• Clean, customizable dashboard elements (e.g., charts) to preview past experiments.

containerize each service by wrapping it within a Dockerfile, allowing seamless deployment on container-enabled resources while also enabling local execution for development purposes.

This enhancement has potential to improve deployment consistency, simplify dependency management, and provide greater flexibility in running Airavata services across different environments, for both testing and production use cases.

The Apache Beam JupyterLab Sidepanel provides a valuable tool for interactive development and visualization of Apache Beam pipelines within the JupyterLab environment. This project aims to significantly enhance the sidepanel by achieving full compatibility with the latest JupyterLab 4.x release and implementing substantial UI/UX improvements. This will ensure seamless integration with modern JupyterLab workflows and provide a more intuitive and user-friendly experience for Apache Beam developers.

Apache Beam 's YAML DSL provides a powerful and declarative way to define data processing pipelines. However, its adoption for complex use cases like Machine Learning (ML) and Managed IO (specifically Apache Iceberg and Kafka) is hindered by a lack of comprehensive documentation and practical examples. This project aims to significantly improve the Beam YAML documentation and create illustrative examples focused on ML workflows and Iceberg/Kafka integration, making these advanced features more accessible to users.it is increasingly important for users to be able to track how their data is moving through Beam so that they can make informed choices on how they manage their data at the source or sink of their pipeline. To solve for this, we have recently introduced data lineage in Beam - https://en.wikipedia.org/wiki/Data_lineage - to this point support is still relatively limited though.

For this project, the focus would be on adding broader lineage support to Beam. This could include:

• adding column level lineage to more transforms
• adding direct runner support for lineage graphs (https://github.com/apache/beam/issues/33980)
• Integrating Beam with Open Lineage (https://github.com/apache/beam/issues/33981)

Apache Beam's Python SDK provides a powerful way to define data processing pipelines. In particular, many users want to use Beam for machine learning use cases like feature generation, embedding generation, and retrieval augmented generation (RAG). Today, however, Beam integrates with a relatively limited set of feature stores and vector DBs for these use cases. This project aims to build out a rich ecosystem of connectors to systems like Pinecone and Tecton to enable these ML use cases.

Apache Beam provides a powerful way to define data processing pipelines. However, it is increasingly important for users to be able to track how their data is moving through Beam so that they can make informed choices on how they manage their data at the source or sink of their pipeline. To solve for this, we have recently introduced data lineage in Beam - https://en.wikipedia.org/wiki/Data_lineage - to this point support is still relatively limited though.

For this project, the focus would be on adding broader lineage support to Beam. This could include: