The Airavata Django Portal currently runs on Python 3.6, which reached its end-of-life (EOL) in 2022. Continuing to use an unsupported Python version poses security risks and limits access to new features and package updates. Upgrading to a supported version (Python 3.12 or later) will ensure long-term maintainability, security, and compatibility with modern dependencies.

Status of Python versions

Impact:
• Improved security and stability
• Access to the latest language features and performance improvements
• Compatibility with actively maintained third-party packages

Proposed Solution: Update the codebase and dependencies for compatibility with Python 3.12+, and ensure everything works as expected post-upgrade.

Objective

Replace existing Ansible deployment scripts with OpenTofu configurations to improve deployment efficiency and maintainability for bare-metal environments.

Requirements

• Assessment of Current Ansible Scripts
  • Review Existing Playbooks: Analyze the current Ansible playbooks located in the Airavata GitHub repository to understand the deployment processes and dependencies.
  • Identify Core Components: Determine the essential services and configurations managed by Ansible, such as Kafka, RabbitMQ, Zookeeper, MariaDB, etc.

• Development of OpenTofu Configurations
  • Define Infrastructure as Code (IaC): Utilize OpenTofu's declarative language to codify the infrastructure components identified in the assessment phase.
  • Module Creation: Develop reusable modules for each service (e.g., Kafka, RabbitMQ, Zookeeper) to promote consistency and ease of management.

• Testing and Validation
  • Simulate Deployments: Use OpenTofu's planning capabilities to simulate deployments, ensuring configurations align with the desired infrastructure state.
  • Iterative Refinement: Address any discrepancies or issues identified during testing to refine the OpenTofu configurations.

• Documentation
  • Update Deployment Guides: Revise existing documentation to reflect the new OpenTofu-based deployment process, providing clear instructions for users.

Objective

Replace existing Ansible deployment scripts with OpenTofu configurations to improve deployment efficiency and maintainability for bare-metal environments.

Requirements

• Assessment of Current Ansible Scripts
  • Review Existing Playbooks: Analyze the current Ansible playbooks located in the Airavata GitHub repository to understand the deployment processes and dependencies.
  • Identify Core Components: Determine the essential services and configurations managed by Ansible, such as Kafka, RabbitMQ, Zookeeper, MariaDB, etc.

• Development of OpenTofu Configurations
  • Define Infrastructure as Code (IaC): Utilize OpenTofu's declarative language to codify the infrastructure components identified in the assessment phase.
  • Module Creation: Develop reusable modules for each service (e.g., Kafka, RabbitMQ, Zookeeper) to promote consistency and ease of management.

• Testing and Validation
  • Simulate Deployments: Use OpenTofu's planning capabilities to simulate deployments, ensuring configurations align with the desired infrastructure state.
  • Iterative Refinement: Address any discrepancies or issues identified during testing to refine the OpenTofu configurations.

• Documentation
  • Update Deployment Guides: Revise existing documentation to reflect the new OpenTofu-based deployment process, providing clear instructions for users.

Objective

Enhance the current development workflow by incorporating a simulated High-Performance Computing (HPC) environment into Apache Airavata's existing Integrated Development Environment (IDE) integration. This will enable developers to test and validate features locally without relying on physical HPC resources.

Requirements

• Simulated HPC Environment Integration
  • Dockerized Slurm Simulation: Develop a Docker container that emulates an HPC environment using Slurm, facilitating the testing of job scheduling and execution.​
  • Seamless IDE Integration: Ensure that this simulated environment integrates smoothly with the existing IDE setup, allowing developers to initiate and monitor jobs as they would in a real HPC setting.

• Development of Comprehensive Test Scenarios
  • Job Submission Tests: Create scripts to test various job submission scenarios, including successful executions, intentional failures, and long-running processes.​
  • Feature Validation: Ensure that all features exposed by Apache Airavata can be tested within this simulated environment.

• User-Friendly Setup
  • Simplified Configuration: Design the setup process to require minimal configuration, enabling developers to initiate the environment and execute tests with just a few commands

Develop a devops dashboard to monitor Apache Airavata services, enabling real-time tracking of service health, uptime, and logs independent of the science gateway(s).
This centralized tool will help administrators efficiently monitor service performance and troubleshoot issues. The dashboard will feature a user-friendly monitoring UI that displays real-time status updates and logs for each service.

Proposed Solution:

• A logging subproces alongside each service, pushing logs to an external service.
• A devops dashboard that aggregates the logs and provides a unified view into the system.
• API calls from devops dashboard to each service, for proactive health-checking.
• Ability to monitor multiple gateways from the same dashboard.

Currently, all Airavata services are packaged and deployed as Java bundles. The goal is to containerize each service by wrapping it within a Dockerfile, allowing seamless deployment on container-enabled resources while also enabling local execution for development purposes.

This enhancement has potential to improve deployment consistency, simplify dependency management, and provide greater flexibility in running Airavata services across different environments, for both testing and production use cases.

Currently, all Airavata services are packaged and deployed as Java bundles. The goal is to containerize each service by wrapping it within a Dockerfile, allowing seamless deployment on container-enabled resources while also enabling local execution for development purposes.

This enhancement has potential to improve deployment consistency, simplify dependency management, and provide greater flexibility in running Airavata services across different environments, for both testing and production use cases.

Develop a devops dashboard to monitor Apache Airavata services, enabling real-time tracking of service health, uptime, and logs independent of the science gateway(s).
This centralized tool will help administrators efficiently monitor service performance and troubleshoot issues. The dashboard will feature a user-friendly monitoring UI that displays real-time status updates and logs for each service.

Proposed Solution:

• A logging subproces alongside each service, pushing logs to an external service.
• A devops dashboard that aggregates the logs and provides a unified view into the system.
• API calls from devops dashboard to each service, for proactive health-checking.
• Ability to monitor multiple gateways from the same dashboard.

Background and Goal

Service Discovery

1. Well organized logs
2. Actuator endpoints
3. Tools

Relevant Skills

1. Familiar with Java
2. Familiar with Microservice architecture

Potential Mentors

Jun Liu, Apache Dubbo PMC Chair, junliu@apache.org^{Image Removed}

The Airavata Django Portal currently runs on Python 3.6, which reached its end-of-life (EOL) in 2022. Continuing to use an unsupported Python version poses security risks and limits access to new features and package updates. Upgrading to a supported version (Python 3.12 or later) will ensure long-term maintainability, security, and compatibility with modern dependencies.

Status of Python versions

Impact:
• Improved security and stability
• Access to the latest language features and performance improvements
• Compatibility with actively maintained third-party packages

Proposed Solution: Update the codebase and dependencies for compatibility with Python 3.12+, and ensure everything works as expected post-upgrade.

Background and Goal

Service Discovery

1. Well organized logs
2. Actuator endpoints
3. Tools

Relevant Skills

1. Familiar with Java
2. Familiar with Microservice architecture

Potential Mentors

1. Jun

Background and Goal

The concept of[ Proxyless Mesh|https://istio.io/v1.15/blog/2021/proxyless-grpc/] was first introduced in this blog. Please read it to learn more concept details.

We have started the development of Dubbo Proxyss Mesh for a while, so that means you don't have to start the project from scratch, anyone who gets involved can start with a specific task at hand. 

In this specific GSoC project, we need developers to mainly focus on implementing more traffic management features of Istio for Dubbo.

Relevant Skills

1. Familiar with Java
2. Familiar with Service Mesh, istio and Microservice architectures
3. Familiar with Kubernetes

Potential Mentors

1. Jun Liu, Apache Dubbo PMC Chair, junliu@apache.org
2. dev@dubbo.apache.org

    

Background and Goal

Dubbo is an easy-to-use, high-performance microservice framework that provides both RPC and rich enterprise-level traffic management features.

The community has been working on the improvement of Dubbo's traffic management abilities, to make it support rich features like traffic spliting, canary release, a/b testing, circuit breaker, mocking, etc. The complete traffic management architecture in Dubbo consists of two major parts, Control Plane and Data Plane. In Dubbo, Control Plane refers to Dubbo Admin, with source code in apache/dubbo-kubernetes. Dubbo Data Plane is implemented by Dubbo sdk (Java, Go, etc)

The concept of[ Proxyless Mesh|https://istio.io/v1.15/blog/2021/proxyless-grpc/] was first introduced in this blog. Please read it to learn more concept details.

We have started the development of Dubbo Proxyss Mesh for a while, so that means you don't have to start the project from scratch, anyone who gets involved can start with a specific task at hand. 

In this specific GSoC The traffic management rules Dubbo ueses now is compatible with the rules in Istio. That means the rules generated by Dubbo Admin and sent to SDK is Istio compatible rules. In this project, we need developers to work mainly focus on Dubbo Admin to make sure it generates and sends those rules correctly.

implementing more traffic management features of Istio for Dubbo.

Relevant Skills

1. Familiar with GolangJava
2. Familiar with Service Mesh, istio and Microservice architectures
3. Familiar with Kubernetes

Potential Mentors

1. Jun Liu, Apache Dubbo PMC Chair, junliu@apache.org
2. dev@dubbo.apache.org

Background and Goal

In the API-First design paradigm, IDL (Interface Definition Language) and its corresponding generation tools have become essential. IDL files are the specifications for defining service interfaces, and generation tools can convert IDL files into executable code, thereby simplifying the development process and improving efficiency.
Currently, Apache Dubbo only provides a Maven IDL generation plugin, lacking a Gradle plugin. This brings inconvenience to developers using Gradle to build projects.

Necessity

1. Unify Build Tools: Gradle is the preferred build tool for Android projects and many Java projects. Providing a Dubbo Gradle IDL plugin can maintain the consistency of build tools and reduce the cost for developers to switch between different build tools.
2. Simplify Configuration: Gradle plugins can simplify the configuration and generation process of IDL files. Developers only need to add plugin dependencies and simple configurations in the `build.gradle` file to complete the generation of IDL files without manually executing complex commands.
3. Integrate Development Process: Gradle plugins can be better integrated with IDEs (Integrated Development Environments). Developers can directly execute Gradle tasks in the IDE, thereby realizing the automatic generation of IDL files and improving development efficiency.

Implementation Plan

1. Plugin Development: Develop a Gradle plugin that encapsulates the Dubbo IDL generation tool and provides a concise configuration interface.
2. Configuration: In the `build.gradle` file, developers can configure parameters such as the path of the IDL file and the directory of the generated code.
3. Task: The plugin provides a Gradle task for executing the generation of IDL files. Developers can execute the task through the command line or the IDE.
4. Dependency Management: The plugin can automatically manage the dependencies of the Dubbo IDL generation tool, ensuring that developers do not need to manually download and configure it.

Expected Results

• Developers can use Gradle to build Dubbo projects and easily generate the code corresponding to the IDL.
• Simplify the configuration and generation process of IDL files, and improve development efficiency.
• Better integration with IDEs to achieve automatic generation of IDL files.

Potential Mentors

1. Albumen Kevin, Apache Dubbo PMC, albumenj@apache.org
2. dev@dubbo.apache.org

    

    

Background and Goal

Currently, Dubbo Python exposes a serialization function interface that requires users to implement their own serialization methods. For commonly used serialization formats such as JSON and Protobuf, users must manually configure them each time. To streamline this process, we aim to build a built-in serialization layer that provides support for these common serialization formats by default.

Goal

We recommend using Pydantic to achieve this. Therefore, we expect the implementation to:

1. an internal serialization layer based on Pydantic, with support for at least JSON and Protobuf.

2. Leverage Pydantic's additional features, including data validation and other useful functionalities.

Relevant Skills

1. Familiar with Python

2. Familiar with RPC

Potential Mentors

1. Albumen Kevin, Apache Dubbo PMC, albumenj@apache.org
2. dev@dubbo.apache.org

    

    

   
   

NuttX has gained support for NXboot firmware loader and updater recently. It allows fail-safe upgrades and recovery from upgrade failures caused by resets during upgrades, temporal MTD devices failures or upgrade to non functional firmware version.

The silicon-heaven is open source communication protocol which allows to build distributed applications mounted into tree like hierarchical system of brokers. The update service implemented over this protocol has advantage that even nodes behind NATs firewalls can be reached, commanded and updated.

pysmCoder is block diagram editor and real time code generator implemented in Python which can be used for rapid control applications prototyping for NuttX (see related documentation). The silicon-heaven protocol allows to connect and inspect running application internal states at tune blocks parameters at runtime.

The goal of this project is to collect and finalize silicon-heaven libraries allowing to build updater application which connect into complete silicon-heaven tree yet keep it small enough to fit into memory constrained targets. The implemented libraries should be integrated into mainline NuttX applications configuration system and simple application should be available as a new example in the NuttX system. The standalone (probably Python/pyshv) host system application with graphic and plain command line mode should be implemented. Then whole solution integration into pysimCoder should be demonstrated. The firmware update file node and control properties should be compiled into control model SHV properties tree when runtime upgrade is enabled in the model editor.

The 10 MBit/s multidrop Ethernet is promising standard for automotive peripherals and other industrial systems  communication where minimal wires count and no need for a central communication element (hub/switch) is a priority over communication speed. The only single pair of wires is enough to connect multiple devices (the basic standard guarantees up to  8 nodes with link length at least up to 25 meters.) implementing communication according IEEE 802.3cg 10BASE−T1S to standard.

The goal of this Apache NuttX GSoC project is to implement driver for the T1S MACPHY combined interfaces. The core of the driver should be implemented as independent of specific platforms and targets architectures with minimized amount of bring-up code need for mapping onto specific board. The general usability with whole range of the MACPHY interfaces from multiple vendors compliant to Open Alliance MACPHY SPI protocol should be kept in the mind. In the frame of the long GSoC project the support for beacon synchronized mode should be implemented as well as its configuration and test application.

The ESP32C6 is proposed as initial widely available and cheap platform for driver development. Then the portability should be demonstrated on ATSAMV71 or other ARM based industrial environment focused targets.

The possible mentors:Karel Koci, Pavel Pisa and Michal Lenc

Background

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform designed to orchestrate complex data processing tasks. It provides a user-friendly interface for defining, scheduling, and monitoring workflows, making it easier to manage and automate data pipelines. DolphinScheduler supports various types of tasks, including shell scripts, SQL queries, and custom scripts, and integrates seamlessly with popular big data ecosystems.

Currently, the Apache DolphinScheduler system supports user login via Password, LDAP, Casdoor SSO, and OAuth. However, as a data platform, it

Background

Apache DolphinScheduler is a distributed and extensible workflow scheduler platform designed to orchestrate complex data processing tasks. It provides a user-friendly interface for defining, scheduling, and monitoring workflows, making it easier to manage and automate data pipelines. DolphinScheduler supports various types of tasks, including shell scripts, SQL queries, and custom scripts, and integrates seamlessly with popular big data ecosystems.

Currently, the Apache DolphinScheduler system supports user login via Password, LDAP, Casdoor SSO, and OAuth. However, as a data platform, it frequently needs to integrate with enterprise - internal user accounts to achieve unified identity authentication, which is crucial for ensuring system security and unified user account management. The existing implementation of Casdoor has a high degree of dependence on the Casdoor project, and the OAuth implementation lacks universality and flexibility.

Our objective is to implement a more generalized OIDC (OpenID Connect) login authentication mechanism. This will enable users to make better use of unified login authentication. Moreover, popular open source login authentication projects like Dexidp, Keycloak, and OAuthProxy all support OIDC. By supporting OIDC, users can integrate with both internal and third-party login authentication methods, such as Feishu Login and WeChat Work Login.

Relevant Skills

Strong proficiency in Java development.
Experience in modern frontend technologies and frameworks.
Highlevel expertise in Spring Boot development.
Thorough familiarity with OIDC and OAuth2 protocols.
Keen interest in opensource projects and eagerness to learn and adapt.

Tasks

Initiate and conduct experiments with Apache DolphinScheduler to comprehensively understand its current functionalities.
Implement and support a more generalized OIDC (OpenID Connect) login authentication mechanism.
Compose corresponding E2E test cases.
Create corresponding documentation for third-party login integrations, covering Keycloak, Dexidp, OAuthProxy, as well as Feishu Login and WeChat Work Login.
Optimize the UI of the Apache DolphinScheduler login page.
Ensure compatibility with the existing functionalities of Apache DolphinScheduler during the process of focusing on enhancements.

Learning Material

 
Apache DolphinScheduler HomePage: https://dolphinscheduler.apache.org
Apache DolphinScheduler GitHub Repository: https://github.com/apache/dolphinscheduler
Sprint OAuth 2.0 Client: https://docs.spring.io/spring-security/reference/reactive/oauth2/client/index.html

pac4j OIDC: https://www.pac4j.org/docs/clients/openid-connect.html
OIDC (OpenID Connect): https://openid.net/developers/how-connect-works/

Mentor

Gallardot, Apache DolphinScheduler committer, gallardot@apache.org

SbloodyS, Apache DolphinScheduler PMC, zihaoxiang@apache.org

Difficulty: Medium
Project Size: ~150 hours (medium)

Apache CloudStack currently relies on traditional monitoring tools, which may lack deep visibility into kernel-level events and networking performance. This project aims to integrate eBPF-based monitoring into CloudStack to provide lightweight, real-time performance analysis and security auditing.

Benefits to CloudStack

• Improved observability: Gain fine-grained insights into VM performance metrics.
• Lower overhead: eBPF runs in the kernel and avoids the performance penalties of user-space monitoring tools.
• Enhanced security auditing: Detect and log anomalies in system behavior.

Deliverables

• Implement eBPF programs to track:
  • VM CPU usage
  • Memory consumption
  • Disk I/O metrics
  • Network traffic analysis
• Develop a CloudStack-compatible API or CLI for retrieving eBPF-generated insights.
• Provide visualization support using Prometheus/Grafana.
• Write documentation for setup and usage.

Expected Outcome

A robust eBPF-based monitoring solution integrated into CloudStack, offering real-time performance insights with minimal overhead.

ref. cloudstack issue: https://github.com/apache/cloudstack/issues/10415

This project is marked as part-time, but the scope can be extended to full-time. This depends largely on whether the full amount of metrics to track is implemented or only one, as a proof of concept.

Expected Outcome

A robust eBPF-based monitoring solution integrated into CloudStack, offering real-time performance insights with minimal overhead.

ref. cloudstack issue: https://github.com/apache/cloudstack/issues/10415

This project is marked as part-time, but the scope can be extended to full-time. This depends largely on whether the full amount of metrics to track is implemented or only one, as a proof of concept.

Background

 
Apache StreamPipes is a self-service Industrial IoT toolbox which helps users to connect, analyze and exploit industrial data streams. StreamPipes offers a variety of tools which help users to interact with data from industrial sources such as PLCs. An adapter library allows to get real-time data from industrial controllers or other systems, a pipeline editor allows to build stream processing pipelines using either graphical or code-based flow modeling, and a data explorer allows to quickly create visualizations based on connected adapters.

 Current Challenges

The StreamPipes data explorer consists of a chart view, where users can create charts based on live data, and a dashboard view, where users can create live dashboards based on charts.

The data explorer provides a set of charts, which are mainly based on Apache ECharts. The currently available chart library includes time-series line/bar charts, heatmaps, scatter plots, density charts and others. To improve the user experience and add additional capabilities, we plan to extend this chart library with additional charts that are useful for industrial data analytics. 

 Objectives

The primary objectives of this project are as follows:

• Explore the Apache ECharts library and identify useful additional charts for industrial data analytics
• Improve the StreamPipes data explorer by adding new chart types using Apache ECharts
• Add a more advanced table visualization
• Extend existing charts with additional configurations (e.g., axis configurations, labels, data transformations)
• Add a data preview for all charts, which is shown below the actual chart in the chart view
• Design and implement end-to-end-tests using Cypress

 Recommended Skills

1. Proficiency in TypeScript programming + testing 
2. Proficiency in Angular
3. Excellent logical thinking and problem-solving skills.
4. Good sense for beautifully looking user interfaces 

Mentor

Dominik Riemer, Apache StreamPipes PMC, riemer@apache.org

Difficulty: Major
Project Size: ~350 hours (large)

Background

Apache Kvrocks is a distributed key-value NoSQL database that uses RocksDB as its storage engine and is compatible with Redis protocol.
 
In the past, basic Web UI capabilities have been provided for Apache Kvrocks Controller, including features such as cluster creation and migration. In the future, we aim to offer a better and more modern UI experience, also enhancing centralized visualization capabilities.

Objectives

The key objectives of the project include the following:

• Refactor the existing UI pages

• Enhance the visualization capabilities for cluster migration

• Provide a cluster Overview dashboard

Recommend Skills

1. Familiar with next.js & tailwind
2. Have a basic understanding of RESTFul
3. Have an experience of Apache Kvrocks

Mentor: Hulk Lin, Apache Apache Kvrocks PMC,  hulk@apache.org
Mailing List: dev@kvrocks.apache.org
Please leave comments if you want to be a mentor

Background

Apache Kvrocks is a distributed key-value NoSQL database that uses RocksDB as its storage engine and is compatible with Redis protocol.
 
In the past, basic Web UI capabilities have been provided for Apache Kvrocks Controller, including features such as cluster creation and migration. In the future, we aim to offer a better and more modern UI experience, also enhancing centralized visualization capabilities.

Objectives

The key objectives of the project include the following:

• Refactor the existing UI pages

• Enhance the visualization capabilities for cluster migration

• Provide a cluster Overview dashboard

Recommend Skills

1. Familiar with next.js & tailwind
2. Have a basic understanding of RESTFul
3. Have an experience of Apache Kvrocks

Mentor: Hulk Lin, Apache Apache Kvrocks PMC,  hulk@apache

Backgroud:

Kvrocks is a key-value database that provides a Redis-compatible API on top of RocksDB. Currently, Kvrocks lacks a built-in mechanism for database backup to cloud storage, which is crucial for data durability, disaster recovery, and scalability in cloud environments.

This project aims to implement a robust backup system that allows users to store Kvrocks backups directly in cloud storage services such as Amazon S3, Google Cloud Storage, and/or Azure Blob Storage. The solution will integrate with the existing Kvrocks backup and restore mechanisms while ensuring efficient and secure data transfer.

Deliverables:

1. Cloud Storage Integration: Implement backup storage support for Amazon S3, Google Cloud Storage, and Azure Blob Storage using SDKs, REST APIs or libraries (e.g. Apache OpenDAL).

1. Backup & Restore Commands: Extend Kvrocks’ backup functionality to allow exporting and importing database snapshots from cloud storage.

1. Configuration & Authentication: Provide user-configurable options to specify storage credentials and backup parameters.

1. Incremental Backup Support (Stretch Goal): Optimize storage usage by implementing differential or incremental backup capabilities.

1. Documentation & Tests: Comprehensive documentation and test coverage to ensure reliability and ease of use.

Recommended Skills:

1. Good at coding in C++;
2. Knowledge about database internals and cloud storage;
3. Knowledge about Kvrocks or Redis.

Mentor: Mingyang Liu, Apache Kvrocks PMC member,  twice@apache.org
Mailing List: dev@kvrocks.apache.org
Please leave comments if you want to be a mentor

Backgroud:

Kvrocks is a key-value database that provides a Redis-compatible API on top of RocksDB. Currently, Kvrocks lacks a built-in mechanism for database backup to cloud storage, which is crucial for data durability, disaster recovery, and scalability in cloud environments.

This project aims to implement a robust backup system that allows users to store Kvrocks backups directly in cloud storage services such as Amazon S3, Google Cloud Storage, and/or Azure Blob Storage. The solution will integrate with the existing Kvrocks backup and restore mechanisms while ensuring efficient and secure data transfer.

Deliverables:

1. Cloud Storage Integration: Implement backup storage support for Amazon S3, Google Cloud Storage, and Azure Blob Storage using SDKs, REST APIs or libraries (e.g. Apache OpenDAL).

1. Backup & Restore Commands: Extend Kvrocks’ backup functionality to allow exporting and importing database snapshots from cloud storage.

1. Configuration & Authentication: Provide user-configurable options to specify storage credentials and backup parameters.

1. Incremental Backup Support (Stretch Goal): Optimize storage usage by implementing differential or incremental backup capabilities.

1. Documentation & Tests: Comprehensive documentation and test coverage to ensure reliability and ease of use.

Recommended Skills:

1. Good at coding in C++;
2. Knowledge about database internals and cloud storage;
3. Knowledge about Kvrocks or Redis.

Mentor: Mingyang Liu, Apache Kvrocks PMC member,  twice@apache.org
Mailing List: dev@kvrocks.apache.org

This project consists in a series of tasks that build a sort of 'infra platform' for Beam. Some tasks include:

• Automated cleaning of infrastructure: [Task]: Build a cleaner for assets in the GCP test environment #33644
• Implement Infra-as-code for Beam infrastructure
  • Implement access permissions using IaC: [Task]: Build a cleaner for assets in the GCP test environment #33644
  • Implement drift detection for IaC resources for Beam
  • Implement 'best-practice' key management for Beam (i.e. force key rotation for service account keys, and store in secret manager secrets)

A quality proposal will include a series of features beyond the ones listed above. Some ideas:

This project consists in a series of tasks that build a sort of 'infra platform' for Beam. Some tasks include:

• Automated cleaning of infrastructure: [Task]: Build a cleaner for assets in the GCP test environment #33644
• Implement Infra-as-code for Beam infrastructure
  • Implement access permissions using IaC: [Task]: Build a cleaner for assets in the GCP test environment #33644
  • Implement drift detection for IaC resources for Beam
  • Implement 'best-practice' key management for Beam (i.e. force key rotation for service account keys, and store in secret manager secrets)

A quality proposal will include a series of features beyond the ones listed above. Some ideas:

• Detection of policy breakages, and nagging to fix
• Security detections based on cloud logging
• others?

The Apache Beam JupyterLab Sidepanel provides a valuable tool for interactive development and visualization of Apache Beam pipelines within the JupyterLab environment. This project aims to significantly enhance the sidepanel by achieving full compatibility with the latest JupyterLab 4.x release and implementing substantial UI/UX improvements. This will ensure seamless integration with modern JupyterLab workflows and provide a more intuitive and user-friendly experience for Apache Beam developers.

Apache Beam provides a powerful way to define data processing pipelines. However, it is increasingly important for users to be able to track how their data is moving through Beam so that they can make informed choices on how they manage their data at the source or sink of their pipeline. To solve for this, we have recently introduced data lineage in Beam - https://en.wikipedia.org/wiki/Data_lineage - to this point support is still relatively limited though.

For this project, the focus would be on adding broader lineage support to Beam. This could include:

Apache Beam's Python SDK YAML DSL provides a powerful and declarative way to define data processing pipelines. In particular, many users want to use Beam for machine learning use cases like feature generation, embedding generation, and retrieval augmented generation (RAG). Today, however, Beam integrates with a relatively limited set of feature stores and vector DBs for these use cases. This project aims to build out a rich ecosystem of connectors to systems like Pinecone and Tecton to enable these ML use casesHowever, its adoption for complex use cases like Machine Learning (ML) and Managed IO (specifically Apache Iceberg and Kafka) is hindered by a lack of comprehensive documentation and practical examples. This project aims to significantly improve the Beam YAML documentation and create illustrative examples focused on ML workflows and Iceberg/Kafka integration, making these advanced features more accessible to users.