Apache Beam provides a powerful way to define data processing pipelines. However, it is increasingly important for users to be able to track how their data is moving through Beam so that they can make informed choices on how they manage their data at the source or sink of their pipeline. To solve for this, we have recently introduced data lineage in Beam - https://en.wikipedia.org/wiki/Data_lineage - to this point support is still relatively limited though.

For this project, the focus would be on adding broader lineage support to Beam. This could include:

• adding column level lineage to more transforms
• adding direct runner support for lineage graphs (https://github.com/apache/beam/issues/33980)
• Integrating Beam with Open Lineage (https://github.com/apache/beam/issues/33981)

Background

 
Apache RocketMQ is a distributed messaging and streaming platform that supports various messaging protocols. One of the key features of RocketMQ is its orderly message consumption capability, which guarantees that messages are processed in the order they are sent. However, there are existing issues with the POP Orderly consumption process that need to be addressed to enhance its reliability and performance.

 Current Challenges

Currently, the POP Orderly feature faces several shortcomings, particularly in scenarios where network instability leads to the loss of the attemptId carried by the consumer from the previous round. This issue can result in message consumption getting stuck until the acknowledgment response (ack) for the previous message pull times out. Such situations hinder the efficient processing of messages and reduce the overall effectiveness of the messaging system.

 Objectives

The primary objectives of this project are as follows:
●Refactor the POP Orderly Code: Analyze and redesign the existing codebase to improve its structure, maintainability, and performance.
●Optimize Performance: Implement performance enhancements that allow the POP Orderly feature to cope with network fluctuations and reduce the likelihood of consumption halting.
●Elegant Process Resolution: Develop a more graceful approach to handling the issue of consumption stalling, ensuring that the system can recover more smoothly from failures.

 Recommended Skills

1. Proficiency in Java programming.
2. Strong understanding of concurrent programming.
3. Excellent logical thinking and problem-solving skills.
4. Familiarity with message queue systems, particularly Apache RocketMQ.

Mentor

Rongtong Jin, Apache RocketMQ PMC, jinrongtong@apache.org

Potential Mentor

Juntao Ji, 3160102420@zju.edu.cn

Difficulty: Major
Project Size: ~350 hours (large)

Background

Apache RocketMQ is renowned as a cloud-native messaging and streaming platform, enabling the creation of event-driven applications with simplicity and flexibility. The RocketMQ Dashboard is a crucial component that provides users with insight into system performance and client interactions through intuitive graphs and statistical data. Despite its fundamental role, the current user interface (UI) of the RocketMQ Dashboard is

Background

Apache RocketMQ is renowned as a cloud-native messaging and streaming platform, enabling the creation of event-driven applications with simplicity and flexibility. The RocketMQ Dashboard is a crucial component that provides users with insight into system performance and client interactions through intuitive graphs and statistical data. Despite its fundamental role, the current user interface (UI) of the RocketMQ Dashboard is outdated, affecting user experience and interaction efficiency. Additionally, while the Dashboard offers valuable functionalities, there is a pressing need to enhance its usability and ensure robust security. This project aims to refactor the RocketMQ Dashboard by redesigning its UI with a more contemporary and user-friendly approach, improving overall usability, and introducing effective security measures to safeguard data and user interactions.

Relevant Skills

• Strong Java development skills.
• Experience with modern front-end technologies and frameworks
• Proficiency in Spring Boot development.
• Understanding of UX/UI design principles. - Knowledge of security best practices in web applications.
• A keen interest in open-source projects and a willingness to learn and adapt.

Tasks

• Launch and experiment with the RocketMQ Dashboard to understand current functionalities.
• Refactor the UI of the RocketMQ Dashboard to align with modern user interface standards, ensuring it is intuitive and visually appealing.  
• Improve usability by streamlining workflows, enhancing navigation, and incorporating responsive design. 
• Integrate security features to protect user data, prevent unauthorized access, and mitigate potential vulnerabilities.
• Maintain compatibility with existing RocketMQ functionalities while focusing on enhancements. 

Learning Material

• RocketMQ HomePage: https://rocketmq.apache.org([https://rocketmq.apache.org|https://rocketmq.apache.org/])
• RocketMQ GitHub Repository: https://github.com/apache/rocketmq([https://github.com/apache/rocketmq]) 
• RocketMQ Dashboard GitHub Repository: https://github.com/apache/rocketmq-dashboard([https://github.com/apache/rocketmq-dashboard]) 

Mentor

Rongtong Jin, Apache RocketMQ PMC, jinrongtong@apache.org

Potential Mentor

Juntao Ji, 3160102420@zju.edu.cn

Apache CloudStack currently relies on traditional monitoring tools, which may lack deep visibility into kernel-level events and networking performance. This project aims to integrate eBPF-based monitoring into CloudStack to provide lightweight, real-time performance analysis and security auditing.

Benefits to CloudStack

• Improved observability: Gain fine-grained insights into VM performance metrics.
• Lower overhead: eBPF runs in the kernel and avoids the performance penalties of user-space monitoring tools.
• Enhanced security auditing: Detect and log anomalies in system behavior.

Deliverables

• Implement eBPF programs to track:
  • VM CPU usage
  • Memory consumption
  • Disk I/O metrics
  • Network traffic analysis
• Develop a CloudStack-compatible API or CLI for retrieving eBPF-generated insights.
• Provide visualization support using Prometheus/Grafana.
• Write documentation for setup and usage.

Expected Outcome

A robust eBPF-based monitoring solution integrated into CloudStack, offering real-time performance insights with minimal overhead.real-time performance insights with minimal overhead.

ref. cloudstack issue: https://github.com/apache/cloudstack/issues/10415

This project is marked as part-time, but the scope can be extended to full-time. This depends largely on whether the full amount of metrics to track is implemented or only one, as a proof of concept.

Summary

Currently a lot of API parameters do not get auto-completed as cloudmonkey isn't able to deduce the probable values for those parameters based on the list APIs heuristics. A lot of these parameters are enums on CloudStack end and by finding a way to expose these and consume them on cloudmonkey side, we could improve the usability of the CLI greatly.

Benefits to CloudStack

• Improved end user experience when using CLI
• Reduce incorrect inputs

Deliverables

• Expose enums and all other relevant information that can be used to enhance auto-completion of parameters on CloudStack end -
  • May require framework level changes and changes to APIs
• Consume these exposed details on Cloudmonkey end

Dependent projects

https://github.com/apache/cloudstack-cloudmonkey/

Ref CloudStack Issue: https://github.com/apache/cloudstack/issues/10442

Currently, Cloudstack only has ACLs (in Advanced Networks) that as a layer of securing access to the networks (VPCs). However, these only operate in the Layer 3 and 4 of OSI Layer.

In todays day and age, where Cybersecurity threats become more advanced, complex and operate in Layer 7 OSI layer, there needs to be a way for Cloudstack to allow its own tenants to implement its own form of mature cybersecurity solution.

The problem all this while is that if a user is using a VPC or L2 Networks, 3rd party firewalls such as PFsense, FortinetVM Firwall etc cant be implemented effectively due to a lack of being able to set static routes that stays with the VR after it is recreated.

There needs to be a better option for users of cloudstack to implement a deeper form of cybersecurity to protect their workloads.

ref. github issue:  ref. cloudstack issue: https://github.com/apache/cloudstack/issues/10415

This project is marked as part-time, but the scope can be extended to full-time. This depends largely on whether the full amount of metrics to track is implemented or only one, as a proof of concept.

10445