DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Contents
...
Dynamic Memory Management
AsterixDB currently uses a static approach for memory allocation in memory-intensive operators, where each operator is assigned a fixed memory budget, either user-provided or derived from defaults. Static budgeting can lead to several issues. Long-running queries may hold large memory allocations for extended periods, reducing concurrency and blocking other queries. In addition, memory estimation errors can result in over-allocation that wastes resources or under-allocation that causes spills and performance degradation.
This project will make key memory-intensive operators dynamically adaptive to memory reallocation requests from a resource broker. The broker will adjust operator memory budgets at runtime based on system conditions and workload objectives, such as improving fairness across concurrent queries, increasing overall throughput, and maintaining predictable performance under contention. The expected outcome is a coordinated memory management loop where operators expose safe resizing hooks and the broker uses feedback signals to rebalance memory across running queries.
...
Dropbear port (or other SSH Server/Client) to NuttX
NuttX doesn't have a SSH Client/Server support yet.
Supporting a SSH server will open doors to let NuttX boards in the fields to be access remotely for maintenance
Adding support to SSH client will let low cost boards powered by NuttX and LVGL to become a remote console control for more advanced Linux server.
...
Apache Fory Swift & Dart gRPC integration
Project size missing! Please add appropriate label (small/medium/large)
Description
Apache Fory does not yet generate Swift and Dart gRPC service bindings.
This project will add Swift and Dart gRPC code generation to the Fory compiler. For each service definition, the compiler should generate language-native service interfaces and gRPC transport bindings that follow existing Swift and Dart generator layouts and use a Fory codec instead of protobuf runtime payload types.
The implementation must keep the Fory runtimes free of gRPC dependencies. Any required gRPC glue should be emitted as generated helper code. Runtime behavior should remain low-overhead and allocation-conscious.
Potential Outcomes
- Generate Swift and Dart service interface and gRPC binding outputs from service definitions, aligned with current language generator conventions.
- Generate Swift and Dart gRPC server and client stubs for unary and streaming RPCs using each ecosystem's gRPC APIs.
- Wire request/response handling through generated Fory serializer and deserializer functions in both languages.
- Implement zero-copy deserialization buffer support for inbound gRPC payloads, with a safe fallback path when zero-copy cannot be applied.
- Coordinate with Swift and Dart type generation so emitted message, enum, and union types are directly usable by generated gRPC stubs.
- Add golden codegen tests for generated file names and key signatures for both Swift and Dart outputs.
- Provide runnable Swift and Dart server/client examples using generated bindings and the Fory codec.
- Update compiler documentation for Swift and Dart gRPC code generation usage and constraints.
Skills
Swift, Dart, gRPC (`grpc-swift`, `grpc`), compiler/code generation, serialization internals, async programming, testing, performance optimization.
Difficulty
Hard
Project Size
350 hours
Potential Mentors
Chaokun Yang, Weipeng Wang
Source Links
- https://github.com/apache/fory/issues/3370
- https://github.com/apache/fory/issues/3279
- https://github.com/apache/fory/issues/3281
- https://fory.apache.org/docs/next/compiler/compiler_guide
- https://github.com/apache/fory/tree/main/compiler
- https://github.com/apache/fory/tree/main/dart
- https://github.com/apache/fory/blob/main/dart/README.md
- https://github.com/apache/fory/tree/main/dart/packages/fory
- https://github.com/apache/fory/tree/main/swift
- https://github.com/apache/fory/blob/main/swift/README.md
Airflow
Apache HTTP Server
httpd server Improve a prototype of mod_h3 using openssl and nghttp3
Apache Airflow Contribution & Verification Agent SkillsProject size missing! Please add appropriate label (small/medium/large)
Background
Apache Airflow’s Breeze environment is the de facto way to reproduce CI, run tests, and verify changes locally. It encapsulates complex tooling (Docker, integrations, static checks, tests, system verification) behind a single, consistent developer interface.
However, modern AI coding tools (e.g. Claude Code, Gemini CLI, GitHub Copilot–style agents) currently treat Airflow’s repo like any generic Python project. They rarely:
- Understand whether they are running inside or outside Breeze.
- Choose the correct commands for host vs. container.
- Follow the same workflows that Airflow contributors actually use (e.g. prek, breeze shell, breeze start-airflow).
We already expose some information through docs (e.g. AGENTS.md), but this mostly inflates the context window rather than giving agents a structured, machine-usable interface to Breeze.
This project aims to bridge that gap by creating an “Airflow Breeze Contribution / Contribution Verification” AI skill (final name TBD) that systematically encodes common contribution workflows and makes them reliably executable and testable by AI agents.
Goal
The overarching goal is to make AI tools:
Breeze-aware: able to detect whether they are running inside or outside Breeze and act accordingly.
In practice, this means that for a typical contributor PR, an AI agent can:
- Run the right static checks.
- Run the right subset of tests in Breeze.
- Spin up Airflow and verify system behavior for a Dag representing the change (nice-to-have).
- Do all of the above while respecting host/container boundaries.
Additionally, the solution should be consistency-focused, meaning that we want to keep Breeze CLI as the single source of truth for agent skills. This can be achieved by auto-syncing CLI docstrings and behaviors into the AI skill using existing tooling (e.g. prek), ensuring that the skill definitions always reflect the current state of the Breeze CLI.
Core Tasks
1. Environment Awareness & Detection
- Design and implement a simple, robust mechanism for the agent skills to detect:
- “Host” vs “inside Breeze container”.
- Relevant environment variables, markers, or file paths that indicate context.
- Encode decision logic for when to run:
- Host-only commands (e.g. breeze shell, breeze start-airflow, git operations).
- Container-only commands (e.g. pytest, airflow ...).
- Provide a clear API/contract that AI tools can call to query current context and get recommended commands.
Note: Maybe we need to add some explicit markers, files in the repo, or write a small helper script that can be called to determine context in a reliable way. Or maybe we can rely on existing environment variables or filesystem cues. This is an open design question to explore.
2. Modeling Core Contributor Workflows as Skills
Based on the three scenarios described, define and implement skills that represent common contribution flows:
Scenario 1: Static checks pass
- Stage changes (git add ...).
- Run prek.
- Collect and surface failures in a structured way so that an agent can fix them.
Scenario 2: Unit tests in Breeze
- Start or attach to a Breeze container with breeze shell or breeze exec.
- Run pytest with a targeted module/test path (not the whole suite).
- Then the agent can inspect results and decide on next steps (e.g. fix code, exit Breeze).
3. Syncing with Breeze CLI as Source of Truth (via prek)
- Investigate existing Breeze CLI docstrings and structure.
- Define a mapping from Breeze commands (and their docstrings) to skill definitions, paths, and parameters.
- Implement a prek hook that:
- Generates or updates the agent skills definition files from Breeze CLI docstrings.
- Fails when drift is detected (e.g. a command changed but the skill spec was not updated).
- Integrate these checks into existing static check pipelines so the skills stay in sync automatically.
4. Evaluation & Test Harness
- Design a testable user scenario or “exam” that simulates a typical contribution workflow (e.g. fixing a simple bug, adding a small feature) to verify that the added skills work as intended.
- Add unit tests for any additional scripts or helper functions created.
5. Documentation & Developer Guide
- Add or extend documentation (e.g. AGENTS.md, Breeze docs) to:
- Describe the new Breeze-aware skills.
- Show example workflows for human contributors and AI tools.
- Document how other tools can integrate with the skills (e.g. path to spec file, key commands).
Advanced Tasks (Optional / Stretch Goals)
Scenario: System behavior verification
- Write a Dag representing the feature/bugfix being contributed (or use an existing one).
- Run breeze start-airflow (with --integration when needed).
- Trigger the Dag via CLI (instead of UI) and wait for completion.
- Inspect logs/status to determine success/failure from the TaskInstance logs.
- Inspect logs/status from all the component services (scheduler, api-server, triggerer, etc) to determine if there are any underlying issues.
- The agent can then decide to fix code, fix the Dag, or exit Breeze based on the results.
Expected Outcome
By the end of the project, we expect:
- A Breeze-aware AI skill that can:
- Detect host vs. container context.
- Choose appropriate commands and environment transitions.
- The AI toolings will be "smart-enough" to handle the core workflows for contributions, including:
- Static checks with prek.
- Targeted unit tests in Breeze.
- Continue iterating based on results (e.g. fix code, fix tests, exit).
- A sync mechanism (likely using prek) that:
- Keeps Breeze CLI and the skill definitions in sync.
- Fails CI when they diverge, ensuring Breeze remains the single source of truth.
- Initial evaluation “exam(s)” and test harnesses that:
- Verify that an implementation of the skill behaves correctly on at least the core scenarios.
- Updated documentation explaining how contributors and AI tools can make use of the new capability.
A successful project will make it much easier for future AI tooling (IDEs, CLIs, bots) to interact with Breeze in a reliable and Airflow-native way, increasing contributor productivity and lowering the barrier to entry.
Recommended Skills
- Programming & Tooling
- Solid Python skills (CLI tools, packaging, basic testing).
- Familiarity with Docker and containerized development environments.
- Experience with writing or using CLIs and handling subprocesses.
- Dev Workflow & CI
- Understanding of typical open source contribution workflows (git, PRs, static checks, unit tests, pre-commit).
- Exposure to CI systems and concepts of reproducible environments.
- AI/Agents
- Interest in or experience with AI coding assistants, Agent Skills, tool-calling, or agent frameworks.
- Comfort reasoning about what “smart enough” means in terms of concrete, testable behaviors.
- Airflow/Breeze (Nice to Have)
- Basic knowledge of Apache Airflow concepts (Dags, tasks, operators).
- Prior use of Breeze for development or testing is a plus, but not strictly required.
Motivation to work at the intersection of developer experience, tooling, and AI is more important than prior deep expertise in all of these areas.
Mentors
Jason Liu (GitHub: @jason810496, Slack: Zhe-You(Jason) Liu)Jarek Potiuk (GitHub: @potiuk, Slack: Jarek Potiuk)- #gsoc Slack Channel in Apache Airflow workspace: https://apache-airflow.slack.com/archives/CSC0FLNJF
Learning Materials
- Airflow Breeze documentation: https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst
- Recent Airflow Dev Mailing List discussion regarding Agent Skills/ Agents:
- Airflow prek (pre-commit) hooks entrypoint: https://github.com/apache/airflow/blob/main/.pre-commit-config.yaml
- Modern Python monorepo for Apache Airflow (by Jarek): https://medium.com/apache-airflow/modern-python-monorepo-for-apache-airflow-part-1-1fe84863e1e1
- pre-commit: https://pre-commit.com/
- prek: https://github.com/j178/prek
Tracked GitHub Issue
Apache HTTP Server
OpenSSL 3.2+ brought native QUIC to the world’s most popular security library, yet integration into established web servers remains experimental. This project aims to stabilize the openssl-h3-examples repository and, crucially, advance the development of a prototype Apache httpd module (mod_h3). The work will focus on solving the architectural mismatch between Apache’s TCP-based workers and QUIC’s UDP-based streams, using OpenSSL and nghttp3.
Apache Iceberg
Apache Iceberg Make Spark Readears Async in File Opening
Project size missing! Please add appropriate label (small/medium/large)
Iceberg's Spark readers currently process scan tasks sequentially — each file is opened, fully consumed, and closed before moving to the next. For workloads with hundreds or thousands of small files (5 KB–1 MB), this creates significant overhead, especially on object stores with per-request latency. We would like to introduce an opt-in async mode that opens multiple small-file tasks concurrently and buffers their output into a shared iterator.
We have already been in discussion with a proposed contributor for this new feature
- GitHub issue: #15287
- WIP PR: #15341
- Design doc: Google Doc
HugeGraph
[GSoC][HugeGraph] HugeGraph Query Engine Upgrade & Adaptation
Project size missing! Please add appropriate label (small/medium/large)
Apache HugeGraph is a fast-speed and highly-scalable graph database/computing/AI ecosystem. Billions of vertices and edges can be easily stored into and queried from HugeGraph due to its excellent OLTP ability.
Description
Currently, the HugeGraph core query engine is built on Java 11 + TinkerPop 3.5.x + Groovy 3. While this stack provides fundamental graph query capabilities, it lags behind in security, performance optimization, and support for modern features. Specifically, the built-in Groovy engine relies on complex, high-maintenance black/whitelist mechanisms for script security, which poses potential bypass risks.
The goal of this task is to comprehensively upgrade HugeGraph's underlying dependencies to Java 17 + TinkerPop 3.7/3.8 + Groovy 4. This is not just a version iteration, but a modern architectural transformation:
- Groovy 4 & TinkerPop 3.7/3.8: Introduce improved syntax features and security designs. We aim to refactor HugeGraphSecurity using native, efficient sandboxing mechanisms to replace the legacy blacklist logic.
- Java 17/21 Support: Adapt to the new JDK to fully leverage features like ZGC/Shenandoah GC, Records, and Virtual Threads, significantly improving throughput and reducing long-tail latency in large-scale graph queries.
Applicants are expected to handle the full lifecycle, from dependency upgrades and code refactoring to unit test fixes and final performance benchmarking.
Recommended Skills
- Java Core: Proficiency in Java development with a solid understanding of Java 17+ new features.
- HugeGraph Architecture: Basic understanding of HugeGraph's storage structure (KV Store), Schema design, and specifically the Gremlin query execution flow.
- Graph Computing & Compilers: Familiarity with the TinkerPop Gremlin framework architecture; knowledge of AST (Abstract Syntax Tree) parsing or Functional Programming (FP) mindset is a plus.
- AI Coding: Proficiency in using AI Coding tools (e.g., Codex, Claude Code, Copilot) to assist in code refactoring, test case optimization, and source code interpretation is highly preferred.
- Security Awareness: Awareness of code security, understanding of how to prevent Script Injection, and experience designing secure sandbox environments.
💡 Important Notes for Applicants
- Authenticity Matters: While we encourage the use of AI for coding efficiency, please strictly control and reasonably limit the use of LLMs when writing your project proposal/emails. We value genuine communication and mutual respect.
- Proactive Engagement: We highly recommend participating in community Mini Tasks early. Demonstrating your hands-on ability within the community will significantly increase your chances of selection and help build trust with mentors.
Task List
- Dependency Analysis & Upgrade:
- Analyze Breaking Changes from TinkerPop 3.5 to 3.7/3.8.
- Complete core dependency version upgrades and API adaptations following mentor confirmation.
- Java 17 Environment Adaptation:
- Resolve compile-time and runtime compatibility issues (e.g., reflection restrictions, module access) to ensure the Server module runs correctly on Java 17 (Java 21 is even better).
- Update Docker configurations to migrate the default runtime to Java 17 (while exploring backward compatibility with Java 11).
- PD & Store Module Upgrade (New):
- Extend the upgrade scope to the PD (Placement Driver) and Store modules after completing the core Server upgrade.
- Ensure these modules are adapted to Java 17 to unify the runtime environment across the HugeGraph ecosystem.
- Security Module Refactoring:
- Refactor the HugeGraphSecurity component based on Groovy 4 features.
- Design a lightweight, secure script execution strategy and remove the performance-heavy legacy blacklist logic.
- Testing & Fixes:
- Fix Unit Test (UT) failures caused by the upgrade.
- Ensure all core functions (CRUD, complex Gremlin queries) pass verification.
- Performance Benchmarking:
- Produce a performance comparison report: Java 11 (Old) vs. Java 17 (New) using the Twitter-14B public dataset.
- Quantify improvements in Latency reduction and Throughput increases.
References
- New Contributor Guide: HugeGraph Contribution Guide (Issue #2212) - Environment setup & basics.
- Upgrade Docs: TinkerPop Upgrade Documentation
- Reference Implementation: JanusGraph Upgrade PR (For reference only)
- Gremlin Learning: Practical Gremlin Guide
- Project Wiki: HugeGraph Deepwiki
Project Size
- Difficulty: Medium (Similar references available)
- Estimated Time: ~250 Hours (~15 Weeks)
Mentors
- Yan Zhang: vaughn@apache.org (Apache HugeGraph PMC)
- Imba Jin: jin@apache.org (Apache HugeGraph PMC)
Apache Fluss
Apache Fluss (Incubating) Native RoaringBitmap Integration for Apache Fluss
Project size missing! Please add appropriate label (small/medium/large)
Synopsis
Apache Fluss currently incorporates the BITMAP data type within its metadata layer, but it remains inaccessible to end-users as it is trapped in the UnsupportedKeyword enum. While the aggregation merge engine in Fluss 0.9 supports rbm32/rbm64 at the storage level, BITMAP is not yet a first-class type. Users must currently declare bitmap columns as BYTES.
This GSoC project aims to enable end-to-end native support for the BITMAP data type to allow efficient server-side unique counting. By shifting the computational burden from the client side to the storage side, we can reduce network I/O and CPU utilization for high-cardinality DISTINCT-style aggregations. The project will introduce a proper BITMAP DDL type, SQL functions, and pushdown optimization via applyAggregates().
Benefits to Community
1. Network I/O Efficiency: With bitmap pushdown, only one serialized RoaringBitmap is transferred per group instead of all raw rows, reducing network cost from O(N) to O(G).
2. CPU Utilization Reduction: Heavy unique counting computation is offloaded to the Fluss TabletServer's native merge engine, reducing Flink TaskManager CPU overhead.
3. Ecosystem Interoperability: By using the standard RoaringBitmap binary serialization format, Fluss ensures bitmap data remains accessible to downstream consumers such as Flink, StarRocks, and Doris without requiring proprietary Fluss-specific headers or custom decoders.
4. UV Analytics Optimization: Enables efficient Unique Visitor analytics workflows with pre-aggregated bitmap fragments that can be efficiently merged on the storage side.
Deliverables
The student will deliver the following components:
1. Type System Enablement (fluss-common)
- Introduce BitmapType as a new logical type in fluss-common
- Extend DataTypeParser to support the BITMAP keyword in CREATE TABLE statements
- Define BITMAP type properties (nullable, not orderable, cannot be used as primary key or partition key)
2. Server-Side Aggregation Integration (fluss-server)
- Wire FieldRoaringBitmap32Agg to the new BITMAP logical type
- Extend FieldRoaringBitmap32AggFactory to accept DataTypeRoot.BITMAP in addition to DataTypeRoot.BYTES
- Update AggFunctionType.getSupportedDataTypeRoots() accordingly
- Perform comprehensive audit of AggregationMergeEngine for BITMAP type handling
3. Flink Connector Bridge (fluss-flink)
- Implement SQL UDFs: BITMAP_BUILD_AGG, BITMAP_OR_AGG, BITMAP_CARDINALITY, BITMAP_FROM_BYTES, BITMAP_TO_BYTES
- Extend PbDataTypeRoot with BITMAP = 16 for RPC message support
- Implement applyAggregates() pushdown optimization for BITMAP_OR_AGG
- Handle graceful fallback to Flink-side aggregation when pushdown is not applicable
4. Testing & Documentation
- Functional unit tests for BitmapType and FieldRoaringBitmap32Agg
- End-to-end integration tests (BitmapPushdownITCase) in flink-flink-common module
- Performance benchmarks measuring network I/O and CPU utilization improvements
- User documentation and SQL reference guides
Required Skills
- Proficiency in Java programming
- Understanding of distributed systems and data processing concepts
- Familiarity with Apache Flink or similar stream processing frameworks
- Knowledge of SQL and query optimization is a plus
- Experience with bitmap data structures (RoaringBitmap) is advantageous
- Good communication skills for community collaboration
Difficulty Level
Medium to Hard
This project requires understanding of multiple layers in the Fluss stack (common, server, flink connector) and involves type system changes, aggregation engine integration, and query optimization. A working prototype demonstrating BitmapType integration with FieldRoaringBitmap32Agg is available to help the student get started.
Mentors
- Giannis Polyzos (ipolyzos@apache.org)
Future Work (Stretch Goals)
- Native 64-bit support (BITMAP64 type) for IDs exceeding 32-bit range
- Advanced conversion functions: BITMAP_TO_ARRAY, BITMAP_TO_STRING, BITMAP_XOR_AGG
- Reverse materialization: UNNEST_BITMAP to explode bitmaps back into individual integer rows
Name and Contact Information
Project: Apache Fluss (Incubating)
Website: https://fluss.apache.org
Mailing List: dev@fluss.apache.org
GitHub: https://github.com/apache/fluss
httpd server Improve a prototype of mod_h3 using openssl and nghttp3
Project size missing! Please add appropriate label (small/medium/large)
OpenSSL 3.2+ brought native QUIC to the world’s most popular security library, yet integration into established web servers remains experimental. This project aims to stabilize the openssl-h3-examples repository and, crucially, advance the development of a prototype Apache httpd module (mod_h3). The work will focus on solving the architectural mismatch between Apache’s TCP-based workers and QUIC’s UDP-based streams, using OpenSSL and nghttp3.
Apache Iceberg
Apache Iceberg Make Spark Readears Async in File Opening
Project size missing! Please add appropriate label (small/medium/large)
Iceberg's Spark readers currently process scan tasks sequentially — each file is opened, fully consumed, and closed before moving to the next. For workloads with hundreds or thousands of small files (5 KB–1 MB), this creates significant overhead, especially on object stores with per-request latency. We would like to introduce an opt-in async mode that opens multiple small-file tasks concurrently and buffers their output into a shared iterator.
We have already been in discussion with a proposed contributor for this new feature
- GitHub issue: #15287
- WIP PR: #15341
- Design doc: Google Doc
HugeGraph
[GSoC][HugeGraph] HugeGraph Query Engine Upgrade & Adaptation
Project size missing! Please add appropriate label (small/medium/large)
Apache HugeGraph is a fast-speed and highly-scalable graph database/computing/AI ecosystem. Billions of vertices and edges can be easily stored into and queried from HugeGraph due to its excellent OLTP ability.
Description
Currently, the HugeGraph core query engine is built on Java 11 + TinkerPop 3.5.x + Groovy 3. While this stack provides fundamental graph query capabilities, it lags behind in security, performance optimization, and support for modern features. Specifically, the built-in Groovy engine relies on complex, high-maintenance black/whitelist mechanisms for script security, which poses potential bypass risks.
The goal of this task is to comprehensively upgrade HugeGraph's underlying dependencies to Java 17 + TinkerPop 3.7/3.8 + Groovy 4. This is not just a version iteration, but a modern architectural transformation:
- Groovy 4 & TinkerPop 3.7/3.8: Introduce improved syntax features and security designs. We aim to refactor HugeGraphSecurity using native, efficient sandboxing mechanisms to replace the legacy blacklist logic.
- Java 17/21 Support: Adapt to the new JDK to fully leverage features like ZGC/Shenandoah GC, Records, and Virtual Threads, significantly improving throughput and reducing long-tail latency in large-scale graph queries.
Applicants are expected to handle the full lifecycle, from dependency upgrades and code refactoring to unit test fixes and final performance benchmarking.
Recommended Skills
- Java Core: Proficiency in Java development with a solid understanding of Java 17+ new features.
- HugeGraph Architecture: Basic understanding of HugeGraph's storage structure (KV Store), Schema design, and specifically the Gremlin query execution flow.
- Graph Computing & Compilers: Familiarity with the TinkerPop Gremlin framework architecture; knowledge of AST (Abstract Syntax Tree) parsing or Functional Programming (FP) mindset is a plus.
- AI Coding: Proficiency in using AI Coding tools (e.g., Codex, Claude Code, Copilot) to assist in code refactoring, test case optimization, and source code interpretation is highly preferred.
- Security Awareness: Awareness of code security, understanding of how to prevent Script Injection, and experience designing secure sandbox environments.
💡 Important Notes for Applicants
- Authenticity Matters: While we encourage the use of AI for coding efficiency, please strictly control and reasonably limit the use of LLMs when writing your project proposal/emails. We value genuine communication and mutual respect.
- Proactive Engagement: We highly recommend participating in community Mini Tasks early. Demonstrating your hands-on ability within the community will significantly increase your chances of selection and help build trust with mentors.
Task List
- Dependency Analysis & Upgrade:
- Analyze Breaking Changes from TinkerPop 3.5 to 3.7/3.8.
- Complete core dependency version upgrades and API adaptations following mentor confirmation.
- Java 17 Environment Adaptation:
- Resolve compile-time and runtime compatibility issues (e.g., reflection restrictions, module access) to ensure the Server module runs correctly on Java 17 (Java 21 is even better).
- Update Docker configurations to migrate the default runtime to Java 17 (while exploring backward compatibility with Java 11).
- PD & Store Module Upgrade (New):
- Extend the upgrade scope to the PD (Placement Driver) and Store modules after completing the core Server upgrade.
- Ensure these modules are adapted to Java 17 to unify the runtime environment across the HugeGraph ecosystem.
- Security Module Refactoring:
- Refactor the HugeGraphSecurity component based on Groovy 4 features.
- Design a lightweight, secure script execution strategy and remove the performance-heavy legacy blacklist logic.
- Testing & Fixes:
- Fix Unit Test (UT) failures caused by the upgrade.
- Ensure all core functions (CRUD, complex Gremlin queries) pass verification.
- Performance Benchmarking:
- Produce a performance comparison report: Java 11 (Old) vs. Java 17 (New) using the Twitter-14B public dataset.
- Quantify improvements in Latency reduction and Throughput increases.
References
- New Contributor Guide: HugeGraph Contribution Guide (Issue #2212) - Environment setup & basics.
- Upgrade Docs: TinkerPop Upgrade Documentation
- Reference Implementation: JanusGraph Upgrade PR (For reference only)
- Gremlin Learning: Practical Gremlin Guide
- Project Wiki: HugeGraph Deepwiki
Project Size
- Difficulty: Medium (Similar references available)
- Estimated Time: ~250 Hours (~15 Weeks)
Mentors
- Yan Zhang: vaughn@apache.org (Apache HugeGraph PMC)
- Imba Jin: jin@apache.org (Apache HugeGraph PMC)
Apache Fluss
Airflow
Apache Airflow Contribution & Verification Agent Skills
Background
Apache Airflow’s Breeze environment is the de facto way to reproduce CI, run tests, and verify changes locally. It encapsulates complex tooling (Docker, integrations, static checks, tests, system verification) behind a single, consistent developer interface.
However, modern AI coding tools (e.g. Claude Code, Gemini CLI, GitHub Copilot–style agents) currently treat Airflow’s repo like any generic Python project. They rarely:
- Understand whether they are running inside or outside Breeze.
- Choose the correct commands for host vs. container.
- Follow the same workflows that Airflow contributors actually use (e.g. prek, breeze shell, breeze start-airflow).
We already expose some information through docs (e.g. AGENTS.md), but this mostly inflates the context window rather than giving agents a structured, machine-usable interface to Breeze.
This project aims to bridge that gap by creating an “Airflow Breeze Contribution / Contribution Verification” AI skill (final name TBD) that systematically encodes common contribution workflows and makes them reliably executable and testable by AI agents.
Goal
The overarching goal is to make AI tools:
Breeze-aware: able to detect whether they are running inside or outside Breeze and act accordingly.
In practice, this means that for a typical contributor PR, an AI agent can:
- Run the right static checks.
- Run the right subset of tests in Breeze.
- Spin up Airflow and verify system behavior for a Dag representing the change (nice-to-have).
- Do all of the above while respecting host/container boundaries.
Additionally, the solution should be consistency-focused, meaning that we want to keep Breeze CLI as the single source of truth for agent skills. This can be achieved by auto-syncing CLI docstrings and behaviors into the AI skill using existing tooling (e.g. prek), ensuring that the skill definitions always reflect the current state of the Breeze CLI.
Core Tasks
1. Environment Awareness & Detection
- Design and implement a simple, robust mechanism for the agent skills to detect:
- “Host” vs “inside Breeze container”.
- Relevant environment variables, markers, or file paths that indicate context.
- Encode decision logic for when to run:
- Host-only commands (e.g. breeze shell, breeze start-airflow, git operations).
- Container-only commands (e.g. pytest, airflow ...).
- Provide a clear API/contract that AI tools can call to query current context and get recommended commands.
Note: Maybe we need to add some explicit markers, files in the repo, or write a small helper script that can be called to determine context in a reliable way. Or maybe we can rely on existing environment variables or filesystem cues. This is an open design question to explore.
2. Modeling Core Contributor Workflows as Skills
Based on the three scenarios described, define and implement skills that represent common contribution flows:
Scenario 1: Static checks pass
- Stage changes (git add ...).
- Run prek.
- Collect and surface failures in a structured way so that an agent can fix them.
Scenario 2: Unit tests in Breeze
- Start or attach to a Breeze container with breeze shell or breeze exec.
- Run pytest with a targeted module/test path (not the whole suite).
- Then the agent can inspect results and decide on next steps (e.g. fix code, exit Breeze).
3. Syncing with Breeze CLI as Source of Truth (via prek)
- Investigate existing Breeze CLI docstrings and structure.
- Define a mapping from Breeze commands (and their docstrings) to skill definitions, paths, and parameters.
- Implement a prek hook that:
- Generates or updates the agent skills definition files from Breeze CLI docstrings.
- Fails when drift is detected (e.g. a command changed but the skill spec was not updated).
- Integrate these checks into existing static check pipelines so the skills stay in sync automatically.
4. Evaluation & Test Harness
- Design a testable user scenario or “exam” that simulates a typical contribution workflow (e.g. fixing a simple bug, adding a small feature) to verify that the added skills work as intended.
- Add unit tests for any additional scripts or helper functions created.
5. Documentation & Developer Guide
- Add or extend documentation (e.g. AGENTS.md, Breeze docs) to:
- Describe the new Breeze-aware skills.
- Show example workflows for human contributors and AI tools.
- Document how other tools can integrate with the skills (e.g. path to spec file, key commands).
Advanced Tasks (Optional / Stretch Goals)
Scenario: System behavior verification
- Write a Dag representing the feature/bugfix being contributed (or use an existing one).
- Run breeze start-airflow (with --integration when needed).
- Trigger the Dag via CLI (instead of UI) and wait for completion.
- Inspect logs/status to determine success/failure from the TaskInstance logs.
- Inspect logs/status from all the component services (scheduler, api-server, triggerer, etc) to determine if there are any underlying issues.
- The agent can then decide to fix code, fix the Dag, or exit Breeze based on the results.
Expected Outcome
By the end of the project, we expect:
- A Breeze-aware AI skill that can:
- Detect host vs. container context.
- Choose appropriate commands and environment transitions.
- The AI toolings will be "smart-enough" to handle the core workflows for contributions, including:
- Static checks with prek.
- Targeted unit tests in Breeze.
- Continue iterating based on results (e.g. fix code, fix tests, exit).
- A sync mechanism (likely using prek) that:
- Keeps Breeze CLI and the skill definitions in sync.
- Fails CI when they diverge, ensuring Breeze remains the single source of truth.
- Initial evaluation “exam(s)” and test harnesses that:
- Verify that an implementation of the skill behaves correctly on at least the core scenarios.
- Updated documentation explaining how contributors and AI tools can make use of the new capability.
A successful project will make it much easier for future AI tooling (IDEs, CLIs, bots) to interact with Breeze in a reliable and Airflow-native way, increasing contributor productivity and lowering the barrier to entry.
Recommended Skills
- Programming & Tooling
- Solid Python skills (CLI tools, packaging, basic testing).
- Familiarity with Docker and containerized development environments.
- Experience with writing or using CLIs and handling subprocesses.
- Dev Workflow & CI
- Understanding of typical open source contribution workflows (git, PRs, static checks, unit tests, pre-commit).
- Exposure to CI systems and concepts of reproducible environments.
- AI/Agents
- Interest in or experience with AI coding assistants, Agent Skills, tool-calling, or agent frameworks.
- Comfort reasoning about what “smart enough” means in terms of concrete, testable behaviors.
- Airflow/Breeze (Nice to Have)
- Basic knowledge of Apache Airflow concepts (Dags, tasks, operators).
- Prior use of Breeze for development or testing is a plus, but not strictly required.
Motivation to work at the intersection of developer experience, tooling, and AI is more important than prior deep expertise in all of these areas.
Mentors
Jason Liu (GitHub: @jason810496, Slack: Zhe-You(Jason) Liu)Jarek Potiuk (GitHub: @potiuk, Slack: Jarek Potiuk)- #gsoc Slack Channel in Apache Airflow workspace: https://apache-airflow.slack.com/archives/CSC0FLNJF
Learning Materials
- Airflow Breeze documentation: https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst
- Recent Airflow Dev Mailing List discussion regarding Agent Skills/ Agents:
- Airflow prek (pre-commit) hooks entrypoint: https://github.com/apache/airflow/blob/main/.pre-commit-config.yaml
- Modern Python monorepo for Apache Airflow (by Jarek): https://medium.com/apache-airflow/modern-python-monorepo-for-apache-airflow-part-1-1fe84863e1e1
- pre-commit: https://pre-commit.com/
- prek: https://github.com/j178/prek
Tracked GitHub Issue
Apache Fluss (Incubating) Native RoaringBitmap Integration for Apache Fluss
Project size missing! Please add appropriate label (small/medium/large)
Synopsis
Apache Fluss currently incorporates the BITMAP data type within its metadata layer, but it remains inaccessible to end-users as it is trapped in the UnsupportedKeyword enum. While the aggregation merge engine in Fluss 0.9 supports rbm32/rbm64 at the storage level, BITMAP is not yet a first-class type. Users must currently declare bitmap columns as BYTES.
This GSoC project aims to enable end-to-end native support for the BITMAP data type to allow efficient server-side unique counting. By shifting the computational burden from the client side to the storage side, we can reduce network I/O and CPU utilization for high-cardinality DISTINCT-style aggregations. The project will introduce a proper BITMAP DDL type, SQL functions, and pushdown optimization via applyAggregates().
Benefits to Community
1. Network I/O Efficiency: With bitmap pushdown, only one serialized RoaringBitmap is transferred per group instead of all raw rows, reducing network cost from O(N) to O(G).
2. CPU Utilization Reduction: Heavy unique counting computation is offloaded to the Fluss TabletServer's native merge engine, reducing Flink TaskManager CPU overhead.
3. Ecosystem Interoperability: By using the standard RoaringBitmap binary serialization format, Fluss ensures bitmap data remains accessible to downstream consumers such as Flink, StarRocks, and Doris without requiring proprietary Fluss-specific headers or custom decoders.
4. UV Analytics Optimization: Enables efficient Unique Visitor analytics workflows with pre-aggregated bitmap fragments that can be efficiently merged on the storage side.
Deliverables
The student will deliver the following components:
1. Type System Enablement (fluss-common)
- Introduce BitmapType as a new logical type in fluss-common
- Extend DataTypeParser to support the BITMAP keyword in CREATE TABLE statements
- Define BITMAP type properties (nullable, not orderable, cannot be used as primary key or partition key)
2. Server-Side Aggregation Integration (fluss-server)
- Wire FieldRoaringBitmap32Agg to the new BITMAP logical type
- Extend FieldRoaringBitmap32AggFactory to accept DataTypeRoot.BITMAP in addition to DataTypeRoot.BYTES
- Update AggFunctionType.getSupportedDataTypeRoots() accordingly
- Perform comprehensive audit of AggregationMergeEngine for BITMAP type handling
3. Flink Connector Bridge (fluss-flink)
- Implement SQL UDFs: BITMAP_BUILD_AGG, BITMAP_OR_AGG, BITMAP_CARDINALITY, BITMAP_FROM_BYTES, BITMAP_TO_BYTES
- Extend PbDataTypeRoot with BITMAP = 16 for RPC message support
- Implement applyAggregates() pushdown optimization for BITMAP_OR_AGG
- Handle graceful fallback to Flink-side aggregation when pushdown is not applicable
4. Testing & Documentation
- Functional unit tests for BitmapType and FieldRoaringBitmap32Agg
- End-to-end integration tests (BitmapPushdownITCase) in flink-flink-common module
- Performance benchmarks measuring network I/O and CPU utilization improvements
- User documentation and SQL reference guides
Required Skills
- Proficiency in Java programming
- Understanding of distributed systems and data processing concepts
- Familiarity with Apache Flink or similar stream processing frameworks
- Knowledge of SQL and query optimization is a plus
- Experience with bitmap data structures (RoaringBitmap) is advantageous
- Good communication skills for community collaboration
Difficulty Level
Medium to Hard
This project requires understanding of multiple layers in the Fluss stack (common, server, flink connector) and involves type system changes, aggregation engine integration, and query optimization. A working prototype demonstrating BitmapType integration with FieldRoaringBitmap32Agg is available to help the student get started.
Mentors
- Giannis Polyzos (ipolyzos@apache.org)
Future Work (Stretch Goals)
- Native 64-bit support (BITMAP64 type) for IDs exceeding 32-bit range
- Advanced conversion functions: BITMAP_TO_ARRAY, BITMAP_TO_STRING, BITMAP_XOR_AGG
- Reverse materialization: UNNEST_BITMAP to explode bitmaps back into individual integer rows
Name and Contact Information
Project: Apache Fluss (Incubating)
Website: https://fluss.apache.org
Mailing List: dev@fluss.apache.org
GitHub: https://github.com/apache/fluss
Spark
SPIP Client-Side Metadata Caching for Spark Connect
Project size missing! Please add appropriate label (small/medium/large)
This SPIP proposes adding a client-side schema cache for Spark Connect DataFrames.
Currently, every call to df.columns or df.schema triggers a synchronous gRPC analysis request to the server. While these are local and near-instant in Spark Classic, in Connect they average 277 ms on standard cloud setups (like AWS t3.medium). This makes iterative work extremely slow; we've measured a 13-second lag for 50 metadata calls in a typical ETL pipeline.
This delay is forcing developers to use a "Shadow Schema" pattern, where they manually track column names in local lists to avoid the RPC overhead. Since Spark DataFrames are immutable, we can fix this by caching the resolved schema on the client after the first request. Our POC shows this reduces the 13-second lag to about 250 ms (a 51× speedup) without breaking the core Spark Connect model.
I have followed the official SPIP template for the detailed breakdown below.
SIP
https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?tab=t.0
Benchmark - https://docs.google.com/document/d/1ebX8CtTHN3Yf3AWxg7uttzaylxBLhEv-T94svhZg_uE/edit?tab=t.0
Note for GSoC -
To set clear expectations for your GSoC timeline, and as a heads-up to the broader Spark developer community:
Because the underlying SPIP (SPARK-55163) is still actively being discussed and has not yet received formal PMC approval, your GSoC project will function purely as an experimental prototype.
Your open Pull Requests will be used by mentors to evaluate your GSoC deliverables and milestones. However, please be aware that your code will not be merged into the mainline Apache Spark repository during the GSoC program. Successfully completing your GSoC project and passing the evaluations is tied to the quality of your prototype and testing, not to getting the code merged.
Your prototype will be incredibly valuable in helping the community benchmark the latency improvements for Spark Connect. I look forward to reviewing your finalized proposal!