DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
This page is auto-generated! Please do NOT edit it, all changes will be lost on next update
Contents
APISIX
JSON Schema to Form UI for APISIX Dashboard
APISIX plugins ship JSON Schema definitions for their configuration. If APISIX Dashboard can render plugin configuration forms directly from JSON Schema, developer experience improves significantly and reduces manual UI maintenance.
Goals (Deliverables)
Must-have:
- A reusable SchemaForm (or equivalent) that renders basic types: string/number/integer/boolean/object/array.
- enum support (Select/Radio etc), defaults, required fields, basic constraints (min/max, pattern, etc) where feasible.
- Support at least the key complex patterns used by APISIX plugin schemas:
- oneOf (select one option and render corresponding fields)
- dependencies / conditional fields
- (Stretch) anyOf if present in target schemas
- Validation pipeline: validate form values against schema (AJV) and show errors in UI consistently.
- Documentation + developer guide: how to add/extend schema-to-widget mapping.
- Tests (unit + minimal integration) to prevent regressions for schema parsing and conditional rendering.
Apache AsterixDB
NL2SQL++ assistant
This project aims to develop a modular, extensible NL2SQL component for AsterixDB that translates natural language prompts into executable SQL++ queries. The system will leverage recent advances in Large Language Models (LLMs) to enable users to express complex analytical questions without writing formal queries. It will follow best practices by exposing an OpenAPI-based interface that connects to external LLMs through frameworks such as LangChain4j while remaining model-agnostic. The component will also support locally-hosted LLMs to reduce operating costs and maintain privacy.
LLM Agent Protocols/Memory
This feature adds agent compatibility to AsterixDB by implementing standard agent protocols and agentic memory capabilities. It involves implementing two emerging standards: the Model Context Protocol (MCP) for tool exposure and structured capability discovery, and the Agent-to-Agent (A2A) protocol for multi-agent coordination. MCP will allow AsterixDB to describe its capabilities, datasets, functions, and safe operations to AI agents. The project also utilizes AsterixDB to provide persistent agentic memory that tracks an agent's query sessions, enabling agents to recall and build on previous interactions.
Backup/restore utility for AsterixDB
In order to backup and restore a database, one common pattern is to use a tool that takes the current state of the database and generates a set of DDL statements which, when executed, will create the existing state of the database. Currently this is not possible in AsterixDB for DDL statements- you would have to remember which ones you issued to create Types, Datasets, and so on. Therefore having a tool that can take the current state of the Metadata dataverse and craft a set of DDL statements that would create that state, and then for each dataset dump its contents into an INSERT statement, would be a great addition.
In-browser packaging of AsterixDB
AsterixDB since its inception has always been a distributed system. This has historically led to some friction for new users who simply want to try out the system to get a sense of the language and features. It simply isn't necessary for them to deploy the system as it would be for handling large amounts of data, however the deployment and packaging has to assume someone wants to do this. Therefore it has always been a balance between configurability and simplicity.
With the advancement of WASM and Javascript in general, there now exist versions of other databases, which were previously only run locally, which are adapted and targeted to a WASM or JS environment. This lets the user simply open a browser and get a fully-functioning instance of a real database like they would if it was installed locally or on a server somewhere. Given that AsterixDB is written purely in Java, it should in principle be possible to run AsterixDB on a JVM which can target WASM as an architecture, with WASI or some other platform. Having something similar for AsterixDB would be an amazing tool to help further the adoption of AsterixDB, and SQL++ in general.
Top K Nearest Queries Support
AsterixDB currently lacks native support for Top-K-Nearest queries, which return the K tuples whose attribute values are closest to a given reference value or point. Examples include: the five employees whose salaries are closest to the CEO's salary or the five buildings closest to the White House. This project involves designing and implementing efficient Top-K-Nearest query processing within AsterixDB's execution engine (Hyracks), including optimizer support to avoid full scans and to leverage existing indexes where possible. The implementation should integrate cleanly with SQL++.
Dynamic Memory Management
AsterixDB currently uses a static approach for memory allocation in memory-intensive operators, where each operator is assigned a fixed memory budget, either user-provided or derived from defaults. Static budgeting can lead to several issues. Long-running queries may hold large memory allocations for extended periods, reducing concurrency and blocking other queries. In addition, memory estimation errors can result in over-allocation that wastes resources or under-allocation that causes spills and performance degradation.
This project will make key memory-intensive operators dynamically adaptive to memory reallocation requests from a resource broker. The broker will adjust operator memory budgets at runtime based on system conditions and workload objectives, such as improving fairness across concurrent queries, increasing overall throughput, and maintaining predictable performance under contention. The expected outcome is a coordinated memory management loop where operators expose safe resizing hooks and the broker uses feedback signals to rebalance memory across running queries.
Apache Fineract
Loan Origination POC
Note: GSOC applicants - this is a "draft concept". Do not work on your proposal until we kick off the process at Fineract for evaluating. We may significantly edit this concept or create new ones to replace it.
No one should work on this specific ticket unless assigned - the GSOC candidate we choose will be assigned this ticket.
For more information, you should be reviewing emails on this subject and following the Wiki pages.
https://lists.apache.org/list.html?dev@fineract.apache.org
https://cwiki.apache.org/confluence/display/FINERACT/GSOC+Program+at+Fineract
LOAN ORIGINATION CONTEXT
Fineract has some loan origination functionality but it is not robust enough for many operations. Several vendors, working with Fineract have created new Loan Origination plug ins.
There is also a major enhancement underway that would build out a full Loan Origination flow by supporting the backend needs of data storage for such LOS. See ticket https://issues.apache.org/jira/browse/FINERACT-2418 .
The GSOC student would be expected to propose something as a POC (proof of concept) that would either - use the developed Fienract backend solution, or build a new component outside of Fineract to create the flows that would demonstrate the LOS functionality.
That is, this is a moving target, and we would need different proposals from prospective candidates to explore the area of Loan Origination. This may require expertise in risk assessment, loan origination models and business acumen. There will not be much more explanation that this available. The student would be expected to be a self starter.
The mentor for this would need to be an expert at risk modeling, understand Loan Origination, and support a conceptual basis that may involve some things internal to Fineract and some processing elements outside of Fineract. Please comment below if you are an existing Fineract contributor with this expertise.
To try to illustrate: one possible GSOC Proposal archtype we could accept would be a survey of Loan Origination Models, their strengths and weaknesses and to identify commonalities for the community to focus on. This would thus be a Requirements exercise and may help identify future roadmap concepts. In this case, the code to be developed may just expose a few APIs into different screen flows. Thus, perhaps FIGMA flows (or similar) connecting to a set of APIs on the backend.
If those new LOS APIs are existing in June 2026 (ticket 2418 resolved), then those APIs are to be used. if they are NOT there in Fineract, then the student would be requested to create a fork and to implement the POC outside of the main Dev branch.
I welcome additions to this write up. jdailey
Front end application MVP (POC)
Note: GSOC applicants - this is a "draft concept". Do not work on your proposal until we kick off the process at Fineract for evaluating. We may significantly edit this concept or create new ones to replace it.
No one should work on this specific ticket unless assigned - the GSOC candidate we choose will be assigned this ticket.
For more information, you should be reviewing emails on this subject and following the Wiki pages.
https://lists.apache.org/list.html?dev@fineract.apache.org
https://cwiki.apache.org/confluence/display/FINERACT/GSOC+Program+at+Fineract
Build a simple self-service front end that talks to the Self-Service API
We need a new, user-friendly front end app that connects to our Backend for Front end (Self-Service API component) This will be the “customer portal” experience where users can log in, see their accounts, and check recent activity. It should be straightforward, easy to use, and a good reference example for others to build on.
Functionality needed would include:
- Login
- Check balances
- Transfer between accounts owned by the same customer.
- Submit application for a new loan
Testing end to end required.
Solid UI design
Modern app framework
Documentation
Create a new backend for front end component POC
Note: GSOC applicants - this is a "draft concept". Do not work on your proposal until we kick off the process at Fineract for evaluating. We may significantly edit this concept or create new ones to replace it.
No one should work on this specific ticket unless assigned - the GSOC candidate we choose will be assigned this ticket.
For more information, you should be reviewing emails on this subject and following the Wiki pages.
https://lists.apache.org/list.html?dev@fineract.apache.org
https://cwiki.apache.org/confluence/display/FINERACT/GSOC+Program+at+Fineract
Build a Self-Service API Component that Connects to Apache Fineract
When the project removed self-service APIs in 2025, it did so understanding that we would need an outside component to make that connection as part of an overall solution.
This project is to create - as a Proof of Concept (POC) - a new dedicated Self-Service API component or service that integration with Fineract backend. It will need to expose APIs to consumer facing applications for typical activities like viewing account balances, transaction initiation, loan application, etc.
The idea is for GSOC candidates to propose a design and build the POC.
Minimal criteria include testing, authentication methodology, documentation.
Not included in this GSOC would be the end consumer APP, although that may be undertaken by another project and coordination would be needed.
BI connector and demonstration
Note: GSOC applicants - this is a "draft concept". Do not work on your proposal until we kick off the process at Fineract for evaluating. We may significantly edit this concept or create new ones to replace it.
No one should work on this specific ticket unless assigned - the GSOC candidate we choose will be assigned this ticket.
For more information, you should be reviewing emails on this subject and following the Wiki pages.
https://lists.apache.org/list.html?dev@fineract.apache.org
https://cwiki.apache.org/confluence/display/FINERACT/GSOC+Program+at+Fineract
The idea is to create a connector and a demonstration of analytics that would consume and organize data from Fineract.
For example, create a way to pull data out of Fineract and make it easy to use in common analytics such as Power BI or Tableau or, better yet, an open source variant. The data should probably go to a Data Warehouse.
Start by proposing and exploring different options and write up the pros and cons.
Create a demonstration project that takes into account security, levels of access, and security of PII data if it exists.
fineract-client-feign usage for integration tests
Note: GSOC applicants - this is a "draft concept". Do not work on your proposal until we kick off the process at Fineract for evaluating. We may significantly edit this concept or create new ones to replace it.
No one should work on this specific ticket unless assigned - the GSOC candidate we choose will be assigned this ticket.
For more information, you should be reviewing emails on this subject and following the Wiki pages.
https://lists.apache.org/list.html?dev@fineract.apache.org
https://cwiki.apache.org/confluence/display/FINERACT/GSOC+Program+at+Fineract
"Moving away from RestAssured (low-level) API calls in integration tests and rather use fineract-client-feign would be a great improvement"
Summary (with some assist from chatgpt for clarity)
Apache Fineract has a large set of REST APIs and many integration tests currently call those APIs using RestAssured(low-level HTTP requests). This ticket is to help modernize the tests by switching them to use fineract-client-feign, which is Fineract’s higher-level API client.
Goal
Create a simple migration approach and then migrate a small set of integration tests from RestAssured to fineract-client-feign.
Why we’re doing this
- Makes tests easier to read and maintain (less raw HTTP code).
- Encourages consistent API usage across tests.
- Reduces duplicated request-building logic (headers, base URLs, auth, etc.).
Scope of Work
1) Create a short migration plan
Write a short note (in the Jira ticket comments or a small doc) that answers:
- Where are the current RestAssured-based integration tests located?
- What’s the recommended pattern for using fineract-client-feign in tests?
- What should be migrated first (start small)?
2) Pick a small “starter set” of tests
Identify 2–5 integration tests that:
- Are simple (e.g., create/read/update a resource)
- Don’t involve complicated multi-step workflows
- Run reliably in CI
3) Implement the migration for the starter set
For each selected test:
- Replace RestAssured calls with fineract-client-feign client calls
- Keep the same assertions (same expected behavior)
- Ensure the tests still pass locally and in CI
4) Document the new pattern
Add a short README note or comments in the test code showing:
- How to initialize/configure the Feign client for tests
- How auth/session is handled
- A small “before vs after” explanation (1 paragraph is enough)
Acceptance Criteria
- A brief migration plan is written and linked in the ticket.
- At least 2 integration tests have been converted to use fineract-client-feign.
- All tests pass (locally and/or in CI).
- A short note exists explaining how to write future integration tests using fineract-client-feign.
Notes / Hints for a beginner
- Start by converting just one very small test to learn the pattern.
- Keep changes small and easy to review (one test per commit is ideal).
- If something is unclear (e.g., how auth is set up), add a comment in the ticket describing what you found.
Out of Scope (for this ticket)
- Migrating all integration tests across the repo
- Refactoring production API code
- Changing API behavior—this is only a test client swap
Add DPG Badge to Fineract website and README
As requested by James Dailey on the mailing list, add the Digital Public Goods (DPG) badge to the project website and the main repository README following the DPG Alliance usage guidelines.
Apache NuttX
Create a NuttX Distribution with Dynamic Binary (ELF) Loading
NuttX is very Unix/Linux-like RTOS for microcontrollers and it supports dynamic loading of binaries and libraries. It makes perfect sense to have the possibilities to create a NuttX Distros similar to what exists for Linux.
In fact there is already a proposal here: https://github.com/apache/nuttx/issues/17351
Goals:
1) Test ELF Loading in the current NuttX mainline
2) Create an application that will be downloaded and updated the existing version on the board
3) Add Library support on NuttX/NuttX-Apps (use Android Makefile Library building as reference)
Micro-ROS integration on NuttX
Micro-ROS (https://micro.ros.org) is a ROS2 support to Microcontrollers. Initially the project was developed over NuttX by Bosch and other EU organizations. Later on they added support to FreeRTOS and Zephyr. After that NuttX support started ageing and we didn't get anyone working to fix it (with few exceptions like Roberto Bucher work to test it with pysimCoder).
Add X11 graphic support on NuttX using NanoX
NanoX/Microwindows is a small graphic library what allow Unix/Linux X11 application to run on embedded systems that cannot support X-Server because it is too big. Add it to NuttX will allow many applications to be ported to NuttX. More importantly: it will allow FLTK 1.3 run on NuttX and that could big Dillo web browser.
Wireguard port to NuttX
Wireguard is a light VPN solution for Linux and microcontrollers.
Porting wireguard for NuttX will allow remote and secure access to NuttX devices.
Projects to be used as reference:
TinyGL support on NuttX
TinyGL is a small 3D graphical library created by Fabrice Bellard (same creator of QEMU) designed for embedded system. Currently NuttX RTOS doesn´t have a 3D library and this could enable people to add more 3D programs on NuttX.
Add multi-user support for NuttX
Currently NuttX only support a single user. Also there is no file mode and file owner support.
In fact file mode is already defined in some places in the fs/ but it is not used.
This feature will make NuttX even yet more Unix/Linux-like.
NXBoot algorithm extension for two partitions
Currently NuttX bootloader NXBoot requires three partitions to function properly. This is a trade of between better update speed and higher external memory capacity requirements.
The algorithm isn't suited for devices with small or even none external memory. A different algorithm that uses just two partitions (primary which runs the image) and update (where the update is uploaded) could be used for devices that use only internal memory. It would result in slower update process, but save memory space.
Analog (ADC/DAC) interfaces unification and better API
The issue was discussed and is tracked in a GitHub issue https://github.com/apache/nuttx/issues/16916
Add support to ESP Hosted on NuttX
ESP Hosted is a firmware that allows ESP32xx modules shared WiFi and BLE with the host OS, like Linux, RTOS or even some baremetal MCU.
Add ESP Hosted support on NuttX will allow any platform supported by NuttX to WiFi and/or BLE from ESP32xx modules.
More info: https://github.com/espressif/esp-hosted
Dropbear port (or other SSH Server/Client) to NuttX
NuttX doesn't have a SSH Client/Server support yet.
Supporting a SSH server will open doors to let NuttX boards in the fields to be access remotely for maintenance
Adding support to SSH client will let low cost boards powered by NuttX and LVGL to become a remote console control for more advanced Linux server.
Apache Wayang
Support for a Dataframes API
Background
Apache Wayang is a cross-platform data processing framework that lets users write data analytics tasks once and execute them efficiently across diverse execution engines such as Apache Spark, Apache Flink, relational databases, and others. It abstracts heterogeneous backends and can enable efficient hybrid execution across different execution engines.
Currently, Wayang supports dataflow-style APIs in Java, Scala, and Python and an SQL API. However, there is no high-level DataFrame API — a programmatic abstraction widely used in modern data processing ecosystems (e.g., Spark DataFrames, Pandas, R DataFrames) — that lets users express relational transformations over structured datasets in a fluent, tabular style.
A DataFrame API for Wayang would dramatically improve usability for data engineers and scientists, making Wayang accessible to users familiar with DataFrame programming paradigms while preserving its powerful cross-platform optimization capabilities.
Project Goal
Implement a DataFrame API for Apache Wayang that:
- Represents structured data in a tabular abstraction (rows & columns),
- Supports common relational and analytical operations (select, filter, join, groupBy, aggregate, etc.),
- Can compile DataFrame operations into Wayang plans executed across backends transparently,
- Includes comprehensive documentation and examples.
Outcomes & Impact
By the end of GSoC, Wayang will have its first robust DataFrame API — a major usability milestone that bridges structured analytics with cross-platform execution. This will enhance adoption, unlock new classes of applications, and position Wayang as a friendly high-level programming environment in addition to its optimizer backend strengths.
Difficulty: Medium
Project size: ~350 hours (Large)
Potential mentors:
- Zoi Kaoudi — zkaoudi (at) apache.org
Implement a JDBC driver for Wayang
Background
Apache Wayang is a cross-platform data processing framework that enables users to write data analytics tasks once and execute them across multiple heterogeneous execution engines (e.g., Spark, Flink, Java Streams, and others). In addition, Wayang optimizes execution plans across platforms and can split pipelines to be executed among multiple backends to optimize performance.
Currently, Wayang provides programmatic APIs (Java/Scala) and SQL support. However, it does not expose a standard JDBC interface that would allow external tools to connect to Wayang as if it were a relational database.
Many analytics tools rely on JDBC to communicate with query engines. Implementing a JDBC driver for Wayang would allow users to issue SQL queries to Wayang using standard database tooling.
Project Goal
Design and implement a JDBC driver for Apache Wayang that allows users to:
- Establish a JDBC connection to a Wayang instance,
- Submit SQL queries via standard JDBC interfaces,
- Retrieve results using ResultSet,
- Access metadata through DatabaseMetaData,
- Integrate Wayang with existing SQL-based tools and BI platforms.
The driver should delegate incoming SQL queries to the SQL api provided by Wayang.
Difficulty: Minor
Project size: ~90 hours (small)
Potential mentors:
- Zoi Kaoudi — zoka (at) apache.org
Make Wayang more datalake-friendly
Background
Apache Wayang is a cross-platform data processing framework that allows users to execute analytics pipelines across multiple heterogeneous execution engines such as Apache Spark, Apache Flink, and relational database systems. Wayang’s optimizer automatically selects where to execute a pipeline and enables hybrid pipelines where part of it can be executed in one platform and part of it in another.
Wayang’s architecture is built around a pluggable backend model. Each execution engine is integrated via a dedicated backend implementation that translates Wayang’s logical operators into engine-specific physical operators.
Current execution engines (platforms) that Wayang supports include: JDBC-based databases, Spark, Flink, Tensorflow, Giraph.
Project Goal
Design and implement one or more new execution engine backends to enable Apache Wayang to work in data lake environments.
Potential target engines include (depending on feasibility and community discussion):
- Apache Datafusion
- Trino / Presto
- Dremio
- BigQuery
The project includes:
- Implementing the backend abstraction layer,
- Mapping Wayang logical operators to the new engine’s execution model,
- Integrating cost estimation for the optimizer.
Difficulty: Medium
Project size: Depends on the number of platforms. It can be 175 (medium) or ~350 hours (large)
Potential mentors:
- Zoi Kaoudi — zkaoudi (at) apache.org
- Juri Petersen — juri (at) apache.org
- Community — dev (at) wayang.apache.org
IoTDB
Implement Trino-IoTDB Plugin to enable OLAP on time-series data
Background
Apache IoTDB (Internet of Things Database) is a high-performance, open-source time-series database optimized for data management and analysis in IoT scenarios. Trino (formerly PrestoSQL) is a fast distributed SQL query engine designed for running interactive analytic queries against data sources of all sizes.
Currently, while IoTDB provides strong capabilities for writing and querying time-series data, integrating it with the broader big data ecosystem for complex OLAP (Online Analytical Processing) remains a demand. A dedicated Trino connector for IoTDB will allow users to query IoTDB data using standard SQL via Trino and perform federated queries with other data sources (like Hive, MySQL, or Iceberg).
Goal
The goal of this project is to implement a trino-iotdb connector plugin based on the Trino SPI (Service Provider Interface). This connector will enable Trino to read data directly from IoTDB, supporting schema mapping, data projection, and predicate pushdown or maybe aggregate pushdown.
Core Tasks(Mandatory)
Project Scaffolding: Set up the Maven project structure for the trino-iotdb plugin and integrate the IoTDB JDBC API.
Metadata Implementation: Implement ConnectorMetadata to map IoTDB’s Table Mode (relational view) to Trino’s relational metadata model:
Map IoTDB databases to Trino Schemas.
Map IoTDB Tables to Trino Tables.
Map IoTDB Data Type to Trino Data Type.
Column Pruning (Projection Pushdown): Ensure the connector strictly fetches only the requested columns (measurements) from IoTDB, avoiding SELECT * overhead.
Predicate Pushdown: Implement optimization rules to push down SQL filters (especially time range filters and value filters) to the IoTDB engine to minimize data transfer.
Limit & Offset Pushdown: Map Trino’s LIMIT and OFFSET clauses to IoTDB’s native query pagination to prevent fetching excessive data during preview or pagination queries.
Integration Testing: Provide Docker-based integration tests to verify correctness using Trino's testing framework.
Advanced Tasks (Optional)
Aggregation Pushdown: Implement the applyAggregation method in the connector SPI.
Goal: Map Trino’s aggregate functions (e.g., COUNT, AVG, SUM, MIN, MAX) directly to IoTDB’s native aggregation queries.
Benefit: Instead of fetching raw data to Trino for calculation, the connector leverages IoTDB's pre-calculated statistics or downsampling capabilities, significantly reducing network overhead and latency.
Deliverables
A fully functional trino-iotdb connector source code.(a pull request to Trino Repo)
Comprehensive integration tests covering data types and query patterns.
User documentation explaining how to configure and use the connector.
Recommended Skills
Java: Proficiency in Java programming (Trino and IoTDB are both Java-based).
Database Internals: Basic understanding of SQL execution, schema design, and database connectors.
Maven: Experience with Java build systems.
Nice to have: Familiarity with Trino SPI or IoTDB Session API.
Learning Material
Apache IoTDB: https://iotdb.apache.org/
Trino Connector Developer Guide: https://trino.io/docs/current/develop/connectors.html
Trino PG Plugin: https://github.com/trinodb/trino/tree/master/plugin/trino-postgresql/src/main/java/io/trino/plugin/postgresql
IoTDB Java JDBC API: https://iotdb.apache.org/UserGuide/latest/API/Programming-JDBC_apache.html
IoTDB Table Model Concepts: https://iotdb.apache.org/UserGuide/latest/Background-knowledge/Data-Model-and-Terminology_apache.html
IoTDB Table Model Query Syntax: https://iotdb.apache.org/UserGuide/latest-Table/SQL-Manual/overview_apache.html
Difficulty: medium
Mentor: Yuan Tian (Apache IoTDB PMC Member) (jackietien@apache.org)
Compatible with TPU & integrate SOTA time series foundation models for IoTDB-AINode
Background
Apache IoTDB is a high-performance, IoT-native time-series database designed to manage massive volumes of time-series data generated by industrial IoT devices. It addresses challenges including high ingestion rates, complex out-of-order data handling, and real-time analytical requirements. IoTDB-AINode represents an endogenous node type in the IoTDB ecosystem, extending the database with native machine learning capabilities. IoTDB-AINode enables seamless integration of time series machine learning algorithms directly within the database engine, allowing users to register, manage, and execute inference tasks using simple SQL statements (e.g., CREATE MODEL ..., SELECT * FROM FORECAST (...)). This architecture eliminates costly data migration to external ML platforms, accelerates processing pipelines, and enhances data security by keeping computations close to the data. Currently, AINode includes built-in time series foundation models such as the Timer and Chronos for time series forecasting task.
Tensor Processing Units (TPUs) are Google-developed AI accelerators specifically designed for neural network computations. Offering high-throughput matrix operations and energy efficiency, TPUs provide a compelling alternative to GPUs for deploying large foundation models. PyTorch/XLA enables PyTorch models to leverage TPU hardware through the XLA (Accelerated Linear Algebra) compiler, supporting both single-device and distributed training scenarios.
Time Series Foundation Models have emerged as powerful tools for temporal analysis. These models demonstrate superior performance across diverse domains—from industrial sensor data to financial forecasting—making them ideal candidates for integration into IoTDB's analytical pipeline.
Goal
This project aims to enhance IoTDB-AINode with TPU hardware acceleration capabilities and integrate cutting-edge time series foundation models into the database's model inference pipeline. Specifically, the project will:
- Enable IoTDB-AINode to recognize and leverage Google TPU devices for model deployment and inference.
- Adapt the AINode packaging and compilation workflow (Maven/Java and Poetry/Python) to support TPU-specific releases.
- Survey and integrate 1-2 SOTA time series foundation models (e.g., TimesFM) into AINode's SQL-accessible model registry.
- Establish comprehensive CI pipelines for TPU environments to ensure long-term maintainability.
The ultimate outcome will empower IoTDB users to execute high-performance time series analysis on TPU hardware using state-of-the-art foundation models through simple SQL interfaces, significantly enhancing the database's analytical capabilities for industrial AI applications.
Core Tasks(Mandatory)
- TPU Adaptation. Implement TPU device recognition and tensor management within the AINode Python runtime. This involves:
- Integrating PyTorch/XLA (torch_xla) to detect available TPU devices during AINode initialization.
- Implementing device abstraction layers to handle model loading and tensor operations on TPU hardware.
- Ensuring automatic fallback mechanisms to CPU/GPU when TPU is unavailable.
- Packaging for TPU Version. Extend the existing build infrastructure to support TPU-enabled distributions:
- Update Poetry configuration to manage PyTorch/XLA and TPU-specific Python dependencies.
- Create automated packaging scripts that bundle XLA compilers and TPU runtime libraries.
- Ensure the TPU version can be deployed directly in Google Cloud TPU environments and on-premise TPU pods without manual dependency resolution.
- Model Survey. Conduct a comprehensive technical survey of SOTA time series foundation models available at project commencement. The deliverable will be a technical document analyzing each model's architecture, input requirements, computational complexity, zero-shot capabilities, and suitability for IoTDB's SQL-based inference pipeline. The survey will conclude with a justified selection of 1–2 models for integration based on deployability, inference latency, licensing, and compatibility with IoTDB’s SQL-based workflow.
- Model Integration. Integrate 1-2 selected foundation models into IoTDB-AINode's model inference framework:
- Implement model wrappers conforming to AINode's model registration interface.
- Adapt models to process IoTDB's time series data format.
- Ensure compatibility with AINode's inference pipeline, supporting SQL syntax such as SELECT * FROM FORECAST (...).
- Support both built-in model usage and custom model registration for integrated architectures.
- Integration Testing & CI. Establish robust testing infrastructure for TPU functionality:
- Design and implement integration tests covering device detection, model loading, tensor operations, and end-to-end inference workflows.
- Build TPU-specific CI environments using Google Cloud TPUs or TPU simulators.
Advanced Tasks (Optional)
- Distributed Large Model Deployment. As an optional stretch goal, this task explores distributed deployment of large time series foundation models across multiple TPU devices. This involves:
- Enabling distributed inference where large models are partitioned across TPU pods.
- Developing SQL extensions to specify distributed compute resources (e.g., LOAD MODEL ... TO DEVICES ...).
- Optimizing communication patterns between DataNodes and AINode for high-throughput industrial scenarios involving thousands of time series streams.
Deliverables
- Fully Functional Source Code.
- Pull requests to Apache IoTDB repository containing TPU adaptation modules.
- Integration code for SOTA time series foundation models.
- Extended build configurations (Maven/Poetry/PyInstaller) supporting TPU distributions.
- Comprehensive Integration Tests.
- Automated test suites for TPU device detection and model execution.
- CI pipeline configurations for TPU environments.
- User Documentation.
- Deployment guide for TPU-enabled AINode (e.g. Google Cloud TPU).
- SQL reference extensions for new model types and TPU-specific configuration options.
- Tutorial documentation demonstrating time series analysis workflows using the integrated foundation models.
Recommended Skills
- Python >= 3.11. Including asynchronous programming and ML pipeline development.
- Poetry & PyInstaller. Experience with Python dependency management and executable packaging.
- PyTorch. Known about the PyTorch/XLA integration for TPU support.
- Java & Maven. Knowledge of multi-module Java projects, build profiles, and dependency management.
Learning Material
- Apache IoTDB. https://iotdb.apache.org/
- Time series forecasting models in HuggingFace. https://huggingface.co/models?pipeline_tag=time-series-forecasting&sort=trending
- PyTorch TPU support. https://docs.pytorch.org/xla/master/accelerators/tpu.html
Difficulty: medium
Mentor: Yongzao Dan (Apache IoTDB PMC Member) (yongzao@apache.org)
[GSoC] Flink connector for IoTDB 2.X Table Mode
Background
Apache IoTDB is an open-source IoT-native time-series database designed for high-performance storage, ingestion, and analysis of massive time-series data from IoT devices. It supports deep integration with big data ecosystems like Apache Hadoop, Spark, and Flink, enabling seamless data processing workflows. IoTDB traditionally uses a tree-based data model for organizing time-series data hierarchically (e.g., root.group.device.sensor), which is efficient for device-centric IoT scenarios.
Starting with IoTDB 2.0, a dual-mode SQL architecture was introduced, adding a table mode alongside the tree mode. The table mode allows users to manage time-series data using SQL-like table structures, where each table represents a device type, with columns for timestamps, tags, and fields (e.g., measurements like temperature or humidity). This mode enhances flexibility for data analysis, supports standard SQL queries, and improves interoperability with relational tools. It is particularly useful for scenarios involving heterogeneous devices or advanced analytics, as it supports table-level schema management and retention-related configurations (e.g., TTL).
Apache Flink is a powerful stream and batch processing framework for real-time data analytics. IoTDB already provides a Flink connector (flink-iotdb-connector) for reading from and writing to IoTDB using the tree mode, including IoTDBSource for data ingestion and IoTDBSink for output. There is also a Flink SQL connector (flink-sql-iotdb-connector) for SQL-based interactions and change data capture (CDC). However, these connectors primarily target the tree mode and lack full support for the table mode's features, such as table-specific metadata handling, SQL table mappings in Flink Table API, and optimized read/write operations for table-structured data. As a result, Flink users cannot natively treat IoTDB table-mode data as first-class tables in Flink SQL or the Table API. This gap limits the ability to leverage Flink's processing capabilities with IoTDB's modern table mode, especially in real-time IoT applications like predictive maintenance or anomaly detection.
This project aims to bridge this gap by developing a dedicated Flink connector for IoTDB's 2.X table mode, enabling efficient, real-time integration between Flink and IoTDB tables.
Goal
The primary goal is to create a robust, production-ready Flink connector that supports reading from and writing to IoTDB tables using the 2.X table mode. This will allow Flink users to process IoT time-series data stored in table format, perform transformations, aggregations, and joins in real-time, and sink results back into IoTDB tables. The connector should align with Flink's DataStream and Table APIs, support fault tolerance, and handle table-specific features like tags, fields, and TTL. Ultimately, this will enhance IoTDB's ecosystem integration, making it easier for developers to build scalable IoT data pipelines.
Core Tasks (Mandatory)
- Research and Design: Analyze the existing flink-iotdb-connector and flink-sql-iotdb-connector to identify limitations with the table mode. Design the connector architecture, including schema and type mappings between Flink Table/RowData and IoTDB table-mode concepts (e.g., time column, tags, and fields). Define APIs for source and sink functions compatible with Flink 1.18+.
- Implement IoTDB Table Source: Develop a Flink source connector (e.g., IoTDBTableSource) that reads data from IoTDB tables. Support filtering by time ranges, tags, and fields using IoTDB's SQL interface. Ensure it handles schema inference and dynamic table changes.
- Implement IoTDB Table Sink: Create a Flink sink connector (e.g., IoTDBTableSink) for writing processed data back to IoTDB tables. Support batch and streaming modes, automatic schema creation (if enabled in IoTDB), and error handling for constraints like TTL or data types.
- Testing and Documentation: Write unit and integration tests using Flink's testing utilities and IoTDB test clusters. Document usage examples, configuration options, and deployment guides in the IoTDB repository.
- Community Contributions: Submit pull requests to upstream repositories for any required changes, and create example Flink jobs demonstrating the use cases.
Advanced Tasks (Optional)
- Performance Optimization: Implement optimizations like parallel reading/writing.
- Benchmarking and Comparison: Develop benchmarks comparing the new connector's performance with the existing tree-mode connector, focusing on throughput, latency, and resource usage in IoT scenarios.
Deliverables
Source code for the Flink connector for IoTDB table mode, including Maven artifacts (e.g., flink-iotdb-table-connector).
Comprehensive documentation, including API references, setup guides, and usage examples integrated into the IoTDB website.
Test suites covering core functionality, edge cases, and integration with Flink.
A demo application showcasing a complete Flink pipeline reading from/writing to IoTDB tables.
Optimization reports, benchmarks, and any upstream PRs.
Recommended Skills
- Programming Language: Proficiency in Java, as both Flink and IoTDB are primarily Java-based projects.
- Database Knowledge: Understanding of SQL and fundamental database concepts. Knowledge of time-series data is a plus.
- System Integration: Interest or experience in connecting different systems and understanding data flows.
- Learning and Communication: Ability to quickly understand the codebases of two open-source projects and willingness to actively collaborate with community mentors and members.
Learning Material
Apache IoTDB Official Website: https://iotdb.apache.org/
Apache Flink Official Documentation: https://flink.apache.org/
Integrated Reference: https://github.com/apache/iotdb-extras/tree/master/connectors/flink-iotdb-connector
IoTDB Table Mode Concepts: https://iotdb.apache.org/UserGuide/latest/Background-knowledge/Data-Model-and-Terminology_apache.html
IoTDB Table Mode Query Syntax: https://iotdb.apache.org/UserGuide/latest-Table/SQL-Manual/overview_apache.html
Difficulty: medium
Mentor: Haonan Hou (Apache IoTDB PMC member) (haonan@apache.org)
Enhancing ThingsBoard Integration with IoTDB 2.X Table Mode
Background
Apache IoTDB is a high-performance, open-source time-series database optimized for data management and analysis in Internet of Things (IoT) scenarios, while ThingsBoard is an open-source IoT platform for device management, data visualization, and rule-based automation.
With the release of IoTDB 2.X introducing a dual-mode architecture (tree and table), significant opportunities arise to enhance this integration. The table mode supports standard SQL syntax, JOIN operations, and user-defined functions, enabling more complex queries and analytics. This project proposes to develop an enhanced storage backend for ThingsBoard based on IoTDB's 2.X table mode, providing improved flexibility and performance for IoT data storage and analysis.
Goal
The primary goal of this project is to design and implement a new, enhanced storage backend for ThingsBoard that strategically leverages key features of Apache IoTDB 2.X’s table mode to improve flexibility, query expressiveness, and performance for core IoT telemetry workloads. This enhancement aims to provide ThingsBoard users with more powerful SQL querying capabilities (including complex multi-device joins and time-window aggregations) and improved performance for specific workloads. Furthermore, the project seeks to strengthen the open-source ecosystem by providing a deeper, more capable integration between ThingsBoard and the Apache IoTDB project, resulting in a more robust end-to-end IoT solution for the community.
Core Tasks (Mandatory)
- In-depth Analysis and Design: Conduct a thorough analysis of the existing ThingsBoard-IoTDB integration architecture and ThingsBoard's storage backend interfaces (e.g., TimeseriesDao). Then, design an optimal strategy for mapping the ThingsBoard data model (devices, assets, telemetry, attributes, labels) to the IoTDB 2.X table mode. A key focus will be utilizing IoTDB's TAGS column to efficiently store and manage static device attributes (e.g., location, device type), enabling flexible device filtering and grouping based on these tags .
- Implementation of Storage Backend Connector:
- Data Access Layer: Based on the design, implement the relevant ThingsBoard storage backend interfaces to connect with IoTDB.
- Write Path: Develop efficient data writing logic that transforms device telemetry data received by ThingsBoard and performs batch writes to the corresponding tables in IoTDB.
- Read/Query Path: Implement query interfaces that translate data requests from ThingsBoard dashboards or the rule engine into efficient SQL queries that take full advantage of IoTDB 2.X table mode features.
- Performance Benchmarking and Comparison: Design and execute standardized performance test cases (e.g., high-concurrency data ingestion, complex conditional queries, large-scale range queries). Produce a detailed performance comparison report between the new IoTDB 2.X table mode-based backend and ThingsBoard's existing data storage options, This report should quantify improvements in metrics like write throughput and query latency.
- Testing and Documentation: Write comprehensive integration tests to ensure the correctness and stability of the new functionality. Create detailed user documentation, including installation/configuration instructions, data model explanations, API usage guidelines, and best practices.
- Community Collaboration and Upstream Contribution: Actively communicate with the ThingsBoard open-source community at key project milestones to discuss designs and gather feedback. Submit high-quality Pull Requests (PRs) to the official ThingsBoard repository, adhering to its coding standards, with the goal of getting the implementation merged.
Advanced Tasks (Optional)
- Leverage IoTDB UDFs: Explore the integration of IoTDB's User-Defined Functions (UDFs) within ThingsBoard's rule engine. This could allow for performing more complex data processing and analysis (e.g., anomaly detection) directly within the database before data is pulled into ThingsBoard.
- Enhanced Data Modeling for Assets: Extend the data mapping design to optimally support ThingsBoard's assets and the relations between entities (devices, assets, customers), exploiting the relational capabilities of the IoTDB table mode for more complex queries.
- Comprehensive Dashboard Demo: Build a detailed ThingsBoard dashboard that showcases the advanced querying capabilities made possible by the new integration, such as visualizations based on multi-device joins or complex aggregations.
Deliverables
- A fully functional storage backend plugin/implementation, including source code, build scripts, and configuration examples.
- A detailed design document explaining the data mapping and integration architecture between ThingsBoard and the IoTDB 2.X table mode.
- A comprehensive performance benchmark report comparing the new solution with existing options.
- Complete user and developer documentation.
- A Pull Request submitted to the ThingsBoard community containing the implementation, tests, and relevant documentation.
- A final project report summarizing work, technical challenges, learnings, and future possibilities.
Recommended Skills
- Programming Language: Proficiency in Java, as both ThingsBoard and IoTDB are primarily Java-based projects.
- Database Knowledge: Understanding of SQL and fundamental database concepts. Knowledge of time-series data is a plus.
- System Integration: Interest or experience in connecting different systems and understanding data flows.
- Learning and Communication: Ability to quickly understand the codebases of two open-source projects and willingness to actively collaborate with community mentors and members.
Learning Material
- Apache IoTDB Official Website: https://iotdb.apache.org/
- ThingsBoard Official Documentation: https://thingsboard.io/docs/
- Integrated Reference: https://github.com/thingsboard/thingsboard/pull/11476
- IoTDB Table Mode Concepts: https://iotdb.apache.org/UserGuide/latest/Background-knowledge/Data-Model-and-Terminology_apache.html
- IoTDB Table Mode Query Syntax: https://iotdb.apache.org/UserGuide/latest-Table/SQL-Manual/overview_apache.html
Difficulty: medium
Mentor: Xuan Wang (Apache IoTDB Committer) (critas@apache.org
)
Mahout
Add ZZFeatureMap Encoding for QDP
Backgroud
ZZFeatureMap is the most widely-used data encoding in quantum machine learning. It's the default in Qiskit and PennyLane for quantum kernel methods and variational classifiers.
QDP currently supports amplitude, angle, basis, and IQP encodings. Adding ZZFeatureMap completes our QML encoding suite.
What is ZZFeatureMap?
Maps classical features to quantum states using:
1. Hadamard gates (superposition)
2. RZ gates (single-qubit rotations)
3. ZZ interactions (two-qubit entanglement)
4. Repetition layers for expressivity
Tracked github issue
Apache Mahout Automated API Documentation Pipeline for Qumat & QDP
Summary
Implement an automated API documentation pipeline that generates and publishes API reference documentation from the Python (Qumat, QDP) and Rust (qdp-core) codebases, integrated into the project's Docusaurus website and CI.
Background
- Apache Mahout exposes two main API surfaces:
- Qumat: Python library for quantum circuits (backends: Qiskit, Cirq, Amazon Braket).
- QDP (Quantum Data Plane): GPU-accelerated encoding (Rust core + PyO3 Python bindings, qumat.qdp / _qdp).
- Manual doc updates are error-prone and don’t scale. Automating from source keeps docs accurate and reduces maintainer burden.
Current state
- QuMat API is maintained by hand and can drift from code.
- QDP API is waiting for new website migration to be finished.
- Rust (qdp-core) has extensive doc comments but no published rustdoc in the website.
Goals
1. Generate API reference from source for Python (Qumat).
2. Integrate generated docs into the existing Docusaurus site.
3. Automate the pipeline in CI so doc builds run on changes.
4. Define conventions (docstrings, public API) for future contributors.
Deliverables
- Python API doc pipeline for qumat and QDP.
- QuMat API reference either generated or explicitly linked.
- Rust (qdp-core) rustdoc built and linked from the website.
- CI job(s) that build Python API docs and rustdoc and fail on errors.
- Short contribution guide on docstring style and how to update API docs.
Tracked github issue
https://github.com/apache/mahout/issues/1012
Note
Please email me(jiekaichang@apache.org) your proposal first and show me different types of approaches you considered and why you decided to do it this way.
Beam
Apache Beam Add Kafka Streams Runner
Sketch a working skeleton of portable Kafka Streams Runner for Apache Beam. The runner should be able to run basic portable pipelines and be a baseline implementation for further development, feature additions and performance optimization.
A more detailed design document shall be attached to the github tracking issue.
A learning path to using accelerators with Beam
The Beam project has a few examples where hardware accelerators can be used to run models. See https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/dataflow_tpu_examples.ipynb
This project is to improve on the available set of examples by building starter examples that allow a user to write code that slowly builds up to using these hardware accelerators. The idea would be:
- A simple python script that runs slowly without HW accelerators
- A script that shows improvements when using them
- A training job that uses accelerators
- A Beam pipeline that can train multiple models in parallel using accelerators
- A blog post that can serve as a guide for anyone learning to use hardware accelerators
These would run continuously to ensure their freshness.
Simplify management of Beam infrastructure, access control and permissions via Platform features
This project consists in a series of tasks that build a sort of 'infra platform' for Beam. Some tasks include:
- Automated cleaning of infrastructure: [Task]: Build a cleaner for assets in the GCP test environment #33644
- Implement Infra-as-code for Beam infrastructure
- Implement access permissions using IaC: [Task]: Build a cleaner for assets in the GCP test environment #33644
- Implement drift detection for IaC resources for Beam
- Implement 'best-practice' key management for Beam (i.e. force key rotation for service account keys, and store in secret manager secrets)
A quality proposal will include a series of features beyond the ones listed above. Some ideas:
- Detection of policy breakages, and nagging to fix
- Security detections based on cloud logging
- others?
Apache Beam Python SDK native streaming transforms
Background
Apache Beam is a unified programming model for user developing data processing pipelines capable running in distributed systems. Apache Beam SDK officially supports Java, Python, and Go. While Java SDK was historically dominant, Python SDK is increasingly popular thanks to Beam ML. Python APIs are crucial for developers. We plan to port highly anticipated basic streaming transforms made convenient for Beam Python developers.
Tasks
1. Python UnboundedSource (https://github.com/apache/beam/issues/19137)
While Splittable DoFn has been introduced as a Beam primitive transform handling IO sources, UnboundedSource arguably remains an easier API for users to author their own IOs. In the Java SDK, UnboundedSource/UnboundedReader has been (re)implemented as a wrapper of Splittable DoFn, we can follow the Java implementation and add it to Python.
Stretch goal: implement a native Python streaming IO based on UnboundedSource.
2. Python Watch Transform (https://github.com/apache/beam/issues/21521)
Currently we have a Watch transform in the Java SDK that is very useful when periodically polling for new input to a pipeline. We would like a parallel transform in Python.
Stretch goal: Update Python FileIO.readContinuously to use watch transform
Deliverables
- Implementation of Python UnboundedSource: A functional wrapper API for UnboundedSource and UnboundedReader built on Splittable DoFn (a merged pull request to the Apache Beam repo).
- Implementation of Python Watch Transform: A parallel transform to the Java Watch API for periodic polling (a merged pull request to the Apache Beam repo).
- Unit and Integration Tests: tests for both features, specifically covering watermarks, checkpointing, and polling termination conditions.
- User Documentation: Updated SDK guides and Docstrings explaining how to author custom IOs using UnboundedSource and how to use the Watch transform in pipelines.
- Refactored FileIO.readContinuously (Stretch Goal): A pull request updating FileIO.read_continuously to utilize the new Watch transform logic.
Recommended Skills
- Proficiency in Python, experience with pytest
- Java-to-Python Porting: Ability to read and interpret Java source code
- Version control: Git, development with GitHub
- nice to have: exposure to streaming data processing tools (e.g. Apache Beam/Flink/Spark, etc)
DolphinScheduler
Apache DolphinScheduler Embedding the AlertServer into the API Server
Apache DolphinScheduler
Apache DolphinScheduler is a distributed and extensible workflow scheduler platform with powerful DAG visual interfaces, dedicated to solving complex job dependencies in the data pipeline and providing various types of jobs available out of box.
Website: https://dolphinscheduler.apache.org/en-us/index.html
GitHub: https://github.com/apache/dolphinscheduler
Linked GitHub Issue: https://github.com/apache/dolphinscheduler/issues/8975
Background
Currently, DolphinScheduler requires a separate alert-server to handle workflow and task alerts. Although the alert-server is lightweight, maintaining and deploying it separately adds operational complexity.
We aim to remove the standalone alert-server and embed its alerting functionality directly into the API server.
Task
Integrate the alert-server functionality into the API server so that it can handle workflow and task alerts natively.
Deliverables
- Remove the standalone alert-server.
- Enable the API server to handle all alerting tasks.
- Add Integration test case.
Recommended Skills
- Proficiency in Java.
- Familiarity with microservice, e.g. spring-boot.
- Familiarity with DolphinScheduler’s architecture and alerting mechanisms is a plus.
Mentors
- Wenjun Ruan(Apache DolphinScheduler PMC member), wenjun@apache.org

- Zihao Xiang(Apache DolphinScheduler PMC member), zihaoxiang@apache.org

SkyWalking
Apache SkyWalking BanyanDB Native Data Export/Import Utility
Background
BanyanDB is the native storage engine for Apache SkyWalking, designed specifically for observability data (Traces, Metrics, and Logs). As BanyanDB matures into a production-ready storage backend, data portability becomes critical. Users need the ability to move datasets between environments (e.g., from production to staging for debugging) or export data for external analysis in tools like Python/Pandas, Spark, or specialized AI training pipelines.
Currently, BanyanDB supports disaster recovery backups and simple CSV dumps for specific models. This project aims to build a high-performance, comprehensive Export/Import Utility that supports multiple formats and ensures data integrity.
Tasks
- Multi-Format Support: Implement export/import functionality for:
- Native Binary: High-performance format for BanyanDB-to-BanyanDB migration.
- Plain Text/Standard: Support for Parquet (optimized for metrics/measures) and JSON/CSV (for human readability).
- Batch & Stream Processing: Ensure the tool can handle massive datasets by implementing chunked data reading and writing to avoid memory bottlenecks.
- Schema Evolution Handling: Implement logic to handle cases where the schema in the exported file differs slightly from the target server's schema.
- Integration with bydbctl: Expose these capabilities through a user-friendly CLI command suite (e.g., bydbctl data export --group=user_logs --format=parquet).
Requirements
- Strong knowledge of Go and concurrency patterns.
- Experience with data serialization formats (Protobuf, Parquet, Apache Arrow).
- Familiarity with gRPC-based API communication.
Apache SkyWalking Natural Language to BydbQL
Background
BanyanDB is the native storage engine for Apache SkyWalking, designed specifically for observability data (Traces, Metrics, and Logs). It utilizes its own query language, BydbQL, which is SQL-like but optimized for time-series and observability schemas. While BydbQL is powerful, non-expert users or SREs in high-pressure situations may find it difficult to construct complex queries for specific traces or aggregated metrics.
The goal of this project is to build an Intelligent Query Agent that leverages Large Language Models (LLMs) to translate Natural Language (NL) into valid BydbQL.
Tasks
- Schema-Aware Prompting: Develop a mechanism to extract BanyanDB metadata (Groups, Streams, Measures, Tag Families) and feed it into the LLM context (RAG - Retrieval-Augmented Generation).
- N2SQL Implementation: Adapt state-of-the-art "Natural Language to SQL" (NL2SQL) techniques to the specific syntax and constraints of BydbQL.
- Verification Loop: Integrate the agent with the existing BydbQL parser to validate generated queries before execution.
- CLI/UI Integration: Implement a "chat" interface or an --ask flag in bydbctl (the BanyanDB CLI tool) to allow users to query data via plain English (e.g., "Show me the top 5 slowest services in the last hour").
Requirements
- Proficiency in Go (BanyanDB's primary language).
- Experience with LLM APIs (OpenAI, Gemini, or local models via Ollama) and orchestration frameworks (LangChain, LangGraph).
- Understanding of Compiler Front-ends (Lexing, Parsing, AST).
Seata
GSoC 2026 - Apache Seata(Incubating)Enhance the Seata framework Golang SDK’s support for multiple databases
Project Overview
Title
Enhance the Seata framework Golang SDK’s support for multiple databases
Abstract
Apache Seata(incubating) is a popular distributed transaction solution, providing solutions like AT, TCC, and XA for ensuring data consistency in microservice architectures.
The AT mode (Automatic Transaction) provides applications with non-intrusive distributed transaction capabilities by proxying SQL statements and parsing protocols. Although Seata-go currently supports MySQL and has initial compatibility with PostgreSQL, it still falls short in covering commonly used production databases, and precise compatibility with Oracle and MariaDB is an urgent need.
This project aims to align with the mature ecosystem of Seata Java and introduce AT mode support for Oracle and MariaDB in Seata-go. This not only involves parsing and adapting SQL dialects, but also includes metadata management, handling differences in Undo Log serialization, and integrating with the specific locking mechanisms of each database. It is a critical step in expanding the capability boundaries of Seata-go.
Detailed Description Objectives
- Console Metrics Visualization: Develop functionality to view various metrics related to the connection pool in the Seata console. The metrics should be displayed based on IP/connection pool granularity, helping users easily identify resource allocation and utilization.
- Metrics Control via Console: Allow users to control various aspects of the connection pools directly from the Seata console. This includes the ability to adjust minimum and maximum connection counts, configure connection acquisition timeout, and manage connection pool keep-alive settings.
Deliverables
- Complete MariaDB AT Mode Support (Priority P0)
- Implement MariaDB driver adapter layer (seata-at-mariadb Driver)
- Implement MariaDB TableMetaCache and Trigger
- Implement MariaDB UndoLogManager
- Handle dialect differences between MariaDB and MySQL (e.g., RETURNING clause, system variables)
- Complete Oracle AT Mode Support (Priority P1)
- Implement Oracle driver adapter layer (seata-at-oracle Driver)
- Implement Oracle metadata query adaptation (based on ALL_TAB_COLUMNS, ALL_INDEXES system views)
- Implement Oracle data type to JDBC type mapping (NUMBER, VARCHAR2, CLOB, DATE, etc.)
- Implement Oracle UndoLogManager with Undo Log serialization differences
- Adapt Oracle SQL dialect (ROWNUM pagination, Sequence retrieval, DUAL table, etc.)
- Integration Testing and Validation (Priority P1)
- Write comprehensive unit tests and integration tests covering single-table CRUD, multi-table join operations, and transaction rollback scenarios
- Validate accuracy of Before/After Image generation
- Validate correctness of global locks and Undo Log
- Samples and Documentation (Priority P2)
- Add Oracle and MariaDB usage demos in seata-go-samples
- Write a technical blog "Seata-Go Multi-Database Adaptation Design" explaining design concepts and implementation details
Implementation Plan
Phase 1: Requirement Analysis and Design
- Align scope and acceptance criteria with mentors/community: prioritize MariaDB (P0) and Oracle (confirm final priority as per topic), and define the must-cover SQL/transaction scenarios (CRUD, rollback, lock conflict, batch ops, joins where applicable).
- Study and benchmark Seata Java AT implementation: produce a gap list for Dialect, TableMeta, UndoLog, Before/After Image, and global lock integration, and decide what to port vs. re-design for Go.
- Design a pluggable multi-database architecture for Seata-Go: define clear interfaces (Dialect, MetaQuery, TypeMapper, UndoLogManager, DriverAdapter) and module boundaries for MariaDB/Oracle implementations; write a short design spec.
- Prepare baseline environments and regression safety: stand up MariaDB/Oracle test environments (local and/or CI) and create baseline test cases to ensure existing MySQL AT behavior does not regress.
Phase 2: MariaDB AT Mode Support (P0)
- Implement seata-at-mariadb driver adapter: integrate with database/sql, hook into key execution points, and ensure Seata AT context is correctly propagated.
- MariaDB dialect adaptation: handle MariaDB vs. MySQL differences (syntax/behaviors such as RETURNING-related cases, system variables, and any MariaDB-specific edge cases affecting parsing and image SQL).
- Metadata and caching: implement MariaDB TableMetaCache and metadata queries (columns, primary keys, indexes) with robust caching/invalidations as needed.
- MariaDB UndoLogManager: implement undo log write/read/delete and serialization strategy consistent with Seata-Go conventions; ensure rollback works for common and edge data types.
- Scenario-driven hardening: validate with integration tests covering single-table DML, unique key updates, batch updates, idempotent rollback, and lock conflict/retry behaviors.
Phase 3: Oracle AT Mode Support (P0/P1)
- Implement seata-at-oracle driver adapter: adapt to the chosen Oracle driver (godror / go-ora, per community decision), addressing bind variables, result set handling, and transaction boundary behaviors.
- Oracle metadata adaptation: implement metadata queries using Oracle system views (e.g., ALL_TAB_COLUMNS, ALL_INDEXES) and cache results effectively.
- Oracle type mapping: map Oracle types to Seata-Go internal types (NUMBER, VARCHAR2, CLOB, DATE, TIMESTAMP, etc.) to ensure image capture and undo serialization are consistent.
- Oracle dialect adaptation: support Oracle-specific SQL behaviors (ROWNUM pagination patterns, sequences, DUAL table usage, locking semantics where relevant).
- Oracle UndoLogManager: implement Oracle-compatible undo log persistence and serialization differences; validate large objects and time types in rollback.
Phase 4: Testing, Samples, and Documentation
- Testing: add unit tests (Dialect/TypeMapper/MetaQuery/UndoLog) and integration tests (real DB) to verify:
- Before/After Image correctness
- Global lock correctness (conflicts, concurrency, retries)
- Rollback correctness and idempotency
- No regressions for existing MySQL AT
- CI enablement (if feasible): make MariaDB/Oracle tests repeatable in CI or provide a documented script-based workflow for contributors.
- Samples: add full MariaDB and Oracle examples to incubator-seata-go-samples (config, schema, demo transactions, rollback demos).
- Documentation/blog: write “Seata-Go Multi-Database Adaptation Design” and user/developer docs covering configuration, driver selection, supported SQL patterns, and known limitations.
Required Skills
- Have Go language development experience, familiar with database/sql standard library and common database drivers (e.g., go-sql-driver/mysql, godror, go-ora)
- Proficient in SQL syntax, deep understanding of relational database principles, familiar with transaction isolation levels and row/table locking mechanisms
- Understand the core principles of Seata AT mode (Before/After Image, Undo Log, Global Lock)
- Have experience with Oracle or MariaDB databases, understand their dialect differences from MySQL
- Possess good documentation habits and code standards awareness, able to read Seata Java source code for reference
Benefits to Apache Seata
- Broader database coverage for Seata-Go AT mode: enables production adoption in enterprises that rely on MariaDB and Oracle.
- Lower migration and adoption cost: users can extend distributed transaction capability beyond MySQL with minimal application changes.
- Better maintainability and extensibility: a clean, interface-driven design (Dialect/Meta/UndoLog/TypeMapping) reduces future effort to add more databases.
- Higher reliability through verification: comprehensive integration tests and samples make correctness measurable and reduce regressions.
- Stronger community value: aligning with Seata Java’s proven approach and producing clear docs/design guidance improves contributor productivity and ecosystem confidence.
Conclusion
This project strengthens Seata-Go AT mode by adding robust MariaDB and Oracle support aligned with Seata Java’s mature implementation. By delivering dialect adaptation, metadata management, undo log handling, type mapping, and thorough testing plus samples and documentation, it significantly expands Seata-Go’s multi-database capabilities and improves its production readiness for real-world enterprise environments.
Useful Link
https://github.com/apache/incubator-seata-go
https://github.com/apache/incubator-seata-go-samples
Contact Information
- Mentor Name: FengZhang [zfeng@apache.org], Apache Seata(incubating) PPMC member
GSoC 2026 - Apache Seata(Incubating)Enhance the Seata framework Golang SDK’s multi-registry support and seata-ctl capability
Project Overview
Title
Enhance Seata-Go Multi-Registry Support and seata-ctl Diagnostic Tool Capability
Abstract
Apache Seata (incubating) is a popular distributed transaction solution for ensuring data consistency in microservice architectures. Seata-Go, as its Go language SDK, is responsible for implementing core TM/RM functionalities in the Go ecosystem.
Currently, Seata-Go lags behind the Java version in terms of registry support richness at the infrastructure layer, and its production-level transaction troubleshooting and operational toolchain (seata-ctl) is still in its early stages. This results in limited options for users in non-Etcd/Raft scenarios and high troubleshooting costs when transaction anomalies occur.
This project aims to align with Seata's infrastructure ecosystem by introducing support for four mainstream registries: Nacos, ZooKeeper, Consul, and Redis to Seata-Go. Additionally, it will significantly enhance seata-ctl's diagnostic capabilities through full-chain environment checks, transaction state insights, and an interactive terminal interface, reducing the operational threshold for distributed transactions.
Detailed Description / Objectives
- Infrastructure Alignment: Ensure Seata-Go can seamlessly integrate into existing enterprise-level microservice governance systems by implementing adapters for various mainstream registries.
- Operational Efficiency Improvement: Build a complete diagnostic command set enabling developers to quickly locate network, database, and transaction state anomalies, and simplify operation workflows through an interactive interface.
- Community Ecosystem Contribution: Produce high-quality design documents and technical blogs to help community users understand Seata-Go's underlying governance logic and operational best practices.
Deliverables
1. Multi-Registry Cluster Support (Priority P0)
- Mainstream Registry Adapter Implementation:
- Implement Nacos and ZooKeeper registry adapters with support for service instance subscription, real-time listening, and multi-tenant isolation configuration.
- Implement Consul and Redis adapters with support for service registration/discovery and heartbeat monitoring mechanisms.
- Bug fixes for Seata NamingServer Golang SDK.
- Ensure service registration path formats for all registries are fully compatible with Java version Seata.
- Configuration and Initialization System Integration:
- Extend configuration structure to standardize registry-specific configuration parameters.
- Optimize factory initialization logic to support smooth registry type switching via configuration files.
2. seata-ctl Diagnostic Tool Enhancement (Priority P1)
- Full-Chain Self-Check Functionality:
- Implement automated environment checks covering network connectivity verification with the server.
- Implement database-level health checks including connection availability and validation of transaction core system table structures.
- Implement configuration file format and required field legality validation.
- Transaction State Insight Capability:
- Implement real-time query functionality for active transaction lists.
- Implement query functionality for resource lock records corresponding to specific transaction identifiers (XID).
- Support structured output formats (e.g., table, JSON, YAML).
- Interactive Terminal Interface (TUI):
- Introduce a visual interactive mode for the tool, simplifying complex command input through interface guidance to enhance operational experience.
3. Testing, Samples, and Community Output (Priority P2)
- Testing and Validation:
- Write unit tests and integration tests for each registry adapter to verify node change awareness capabilities.
- Validate diagnostic tool accuracy across different database dialects.
- Samples and Documentation:
- Add complete multi-registry integration examples in seata-go-samples.
- Write technical articles: "Seata-Go Registry Extension Design and Practice Guide" and "Distributed Transaction Troubleshooting in Practice: Quickly Locating Anomalies with Diagnostic Tools".
Implementation Plan
- Phase 1: Research and Architecture Design
- Research existing Seata-Go registry implementations and study Seata NamingServer implementation logic.
- Design diagnostic tool's interaction logic and command set architecture, ensuring tool extensibility.
- Phase 2: Registry Adapter Development (P0)
- Prioritize completion of core functionality implementation and compatibility testing for Nacos and ZooKeeper.
- Perform bug fixes for Seata NamingServer to optimize its stability in the Go SDK.
- Integrate Consul and Redis support and unify configuration initialization entry points.
- Phase 3: Diagnostic Tool and Interactive Interface Development (P1)
- Develop core logic for environment checks and transaction state queries.
- Build interactive terminal interface (TUI), encapsulating underlying commands into intuitive visual operations.
- Phase 4: Testing Validation and Community Promotion (P2)
- Improve test cases to ensure stability across different registry environments.
- Complete community technical article output and submit related sample code.
Required Skills
- Have Go language development experience, familiar with concurrent programming and network communication.
- Understand service discovery principles, familiar with mainstream registries (e.g., Nacos, ZooKeeper).
- Understand basic distributed transaction principles, familiar with Seata's interaction architecture (TM/RM/TC).
- Familiar with command-line tool development, possess good code standards awareness and documentation writing skills.
Benefits to Apache Seata
- Expand Infrastructure Boundaries: Enable Seata-Go to adapt to more diverse enterprise production environments, eliminating selection barriers.
- Improve Operational Convenience: Fill the gap in operational diagnostic tools for the Go version, significantly reducing user learning and maintenance costs.
- Enhance Ecosystem Interoperability: Ensure consistency in governance between Go and Java versions, supporting Seata's unified multi-language ecosystem.
Conclusion
This project addresses Seata-Go's shortcomings in infrastructure adaptation and operational troubleshooting by enhancing multi-registry support and diagnostic tool capabilities. This not only improves Seata-Go's production readiness but also strengthens the Apache Seata community ecosystem through user-friendly interactive tools and comprehensive technical documentation.
Useful Link
- https://seata.apache.org/
- https://github.com/apache/incubator-seata-go
- https://github.com/apache/incubator-seata-go-samples
- https://github.com/apache/incubator-seata-ctl
Contact Information
- Mentor Name: TunGuo [tew@apache.org], Apache Seata(incubating) Committer
HugeGraph
[GSoC][HugeGraph] HugeGraph Query Engine Upgrade & Adaptation
Description
Currently, the HugeGraph core query engine is built on Java 11 + TinkerPop 3.5.x + Groovy 3. While this stack provides fundamental graph query capabilities, it lags behind in security, performance optimization, and support for modern features. Specifically, the built-in Groovy engine relies on complex, high-maintenance black/whitelist mechanisms for script security, which poses potential bypass risks.
The goal of this task is to comprehensively upgrade HugeGraph's underlying dependencies to Java 17 + TinkerPop 3.7/3.8 + Groovy 4. This is not just a version iteration, but a modern architectural transformation:
- Groovy 4 & TinkerPop 3.7/3.8: Introduce improved syntax features and security designs. We aim to refactor HugeGraphSecurity using native, efficient sandboxing mechanisms to replace the legacy blacklist logic.
- Java 17/21 Support: Adapt to the new JDK to fully leverage features like ZGC/Shenandoah GC, Records, and Virtual Threads, significantly improving throughput and reducing long-tail latency in large-scale graph queries.
Applicants are expected to handle the full lifecycle, from dependency upgrades and code refactoring to unit test fixes and final performance benchmarking.
Recommended Skills
- Java Core: Proficiency in Java development with a solid understanding of Java 17+ new features.
- HugeGraph Architecture: Basic understanding of HugeGraph's storage structure (KV Store), Schema design, and specifically the Gremlin query execution flow.
- Graph Computing & Compilers: Familiarity with the TinkerPop Gremlin framework architecture; knowledge of AST (Abstract Syntax Tree) parsing or Functional Programming (FP) mindset is a plus.
- AI Coding: Proficiency in using AI Coding tools (e.g., Codex, Claude Code, Copilot) to assist in code refactoring, test case optimization, and source code interpretation is highly preferred.
- Security Awareness: Awareness of code security, understanding of how to prevent Script Injection, and experience designing secure sandbox environments.
💡 Important Notes for Applicants
- Authenticity Matters: While we encourage the use of AI for coding efficiency, please strictly control and reasonably limit the use of LLMs when writing your project proposal/emails. We value genuine communication and mutual respect.
- Proactive Engagement: We highly recommend participating in community Mini Tasks early. Demonstrating your hands-on ability within the community will significantly increase your chances of selection and help build trust with mentors.
Task List
- Dependency Analysis & Upgrade:
- Analyze Breaking Changes from TinkerPop 3.5 to 3.7/3.8.
- Complete core dependency version upgrades and API adaptations following mentor confirmation.
- Java 17 Environment Adaptation:
- Resolve compile-time and runtime compatibility issues (e.g., reflection restrictions, module access) to ensure the Server module runs correctly on Java 17 (Java 21 is even better).
- Update Docker configurations to migrate the default runtime to Java 17 (while exploring backward compatibility with Java 11).
- PD & Store Module Upgrade (New):
- Extend the upgrade scope to the PD (Placement Driver) and Store modules after completing the core Server upgrade.
- Ensure these modules are adapted to Java 17 to unify the runtime environment across the HugeGraph ecosystem.
- Security Module Refactoring:
- Refactor the HugeGraphSecurity component based on Groovy 4 features.
- Design a lightweight, secure script execution strategy and remove the performance-heavy legacy blacklist logic.
- Testing & Fixes:
- Fix Unit Test (UT) failures caused by the upgrade.
- Ensure all core functions (CRUD, complex Gremlin queries) pass verification.
- Performance Benchmarking:
- Produce a performance comparison report: Java 11 (Old) vs. Java 17 (New) using the Twitter-14B public dataset.
- Quantify improvements in Latency reduction and Throughput increases.
References
- New Contributor Guide: HugeGraph Contribution Guide (Issue #2212) - Environment setup & basics.
- Upgrade Docs: TinkerPop Upgrade Documentation
- Reference Implementation: JanusGraph Upgrade PR (For reference only)
- Gremlin Learning: Practical Gremlin Guide
- Project Wiki: HugeGraph Deepwiki
Project Size
- Difficulty: Medium (Similar references available)
- Estimated Time: ~250 Hours (~15 Weeks)
Mentors
- Yan Zhang: vaughn@apache.org (Apache HugeGraph PMC)
- Imba Jin: jin@apache.org (Apache HugeGraph PMC)
Apache Fory
Apache Fory Implement Fory Python gRPC Integration
Description:
Apache Fory can generate high-performance Python model code from IDL, but Python gRPC stubs are not generated yet.
This project will implement Python gRPC integration in the Fory compiler by generating *_service.py and *_grpc.py
artifacts, using Fory serialization only (without protobuf runtime payload types).
The implementation should align with Fory’s compiler conventions and keep runtime overhead low.
Potential Outcomes:
- Generate *_service.py base interfaces and *_grpc.py gRPC binding files from service definitions.
- Generate servicer base classes and stub classes using gRPC unary/stream APIs.
- Wire request/response serialization via generated Fory serializer/deserializer callables.
- Implement zero-copy deserialization buffer support for inbound gRPC payloads, with a safe fallback path when zero-copy cannot be applied.
- Add golden codegen tests for output file names and key signatures.
- Provide a runnable Python server/client example with generated stubs and Fory codec.
- Update compiler documentation for Python gRPC code generation usage and constraints.
Skills:
Python, gRPC (grpcio), compiler/code generation, serialization internals, testing, performance optimization.
Difficulty: Medium
Project size: 175 hours
Potential mentors: Chaokun Yang, Weipeng Wang
Source links:
- https://github.com/apache/fory/issues/3273
- https://fory.apache.org/docs/next/compiler/compiler_guide
- https://github.com/apache/fory/tree/main/compiler
- https://github.com/apache/fory/tree/main/python
- https://fory.apache.org/docs/guide/python/
Apache Fory Implement Fory Java gRPC Integration
Description
Apache Fory can already generate high-performance Java model code from IDL, but Java gRPC stubs are not generated
yet.
This project will implement Java gRPC integration in the Fory compiler by generating *Service.java and *Grpc.java artifacts, using Fory serialization only (no protobuf runtime types for payload encoding).
compiler conventions, and avoid adding gRPC runtime dependencies into Fory core.
- Generate *Service.java interfaces and *Grpc.java binding classes from service definitions.
- Generate gRPC MethodDescriptors, server binders, and client stubs (including unary/streaming modes supported by
IR). - Implement generated Fory-based Marshaller logic for request/response types.
- Add golden codegen tests for output file names and key method signatures.
- Provide a runnable Java server/client example using generated stubs and Fory codec.
- Update compiler documentation for Java gRPC code generation usage and limits.
Skills
Java, gRPC Java, compiler/code generation, serialization internals, testing.
Difficulty:Medium
Project size:175 hours
Potential mentors: Chaokun Yang, Weipeng Wang
Source links:
Apache Fory Implement Fory C++ gRPC integration
Description:
Apache Fory can generate high-performance C++ model code from IDL, but it does not yet generate C++ gRPC service
bindings.
This project will add C++ gRPC code generation to the Fory compiler. For each service definition, the compiler should
generate service.h (service API), service.grpc.h (gRPC declarations), and service.grpc.cc (gRPC implementations),
using a Fory codec instead of protobuf runtime payload types.
The implementation should follow Fory compiler conventions and prioritize low-overhead, performance-first behavior.
Potential Outcomes:
- Parse service IR and generate C++ gRPC outputs for unary and streaming methods.
- Generate service.h, service.grpc.h, and service.grpc.cc with clear separation between API abstractions and transport
bindings. - Generate abstract service interfaces and client stub classes compatible with gRPC C++.
- Implement Fory-based request/response serialization hooks for gRPC marshalling.
- Implement zero-copy deserialization buffer support for inbound gRPC payloads, with a safe fallback path when zero-
copy cannot be applied. - Add golden codegen tests for generated file names and key signatures.
- Provide a runnable C++ server/client example using generated bindings and Fory codec.
- Update compiler documentation for C++ gRPC code generation usage and constraints.
Skills:
C++ 17, gRPC, compiler/code generation, serialization internals, testing, performance optimization.
Difficulty: Medium
Project size: 175 hours
Potential mentors: Chaokun Yang, Weipeng Wang
Source links: * https://github.com/apache/fory/issues/3276
Apache Fory Implement Fory Rust gRPC Integration
Description:
Apache Fory can generate high-performance Rust model code from IDL, but it does not yet generate Rust gRPC service bindings.
This project will add Rust gRPC code generation to the Fory compiler using tonic. For each service definition, the compiler should generate service.rs (service API traits/modules) and service_grpc.rs (tonic server/client transport
The implementation should follow Fory compiler conventions and prioritize low-overhead, allocation-conscious runtime behavior.
Potential Outcomes:
- Generate service.rs and service_grpc.rs from service definitions, with clear separation between API traits and
transport bindings. - Generate tonic-compatible async server/client wrappers.
- Implement a custom tonic codec backed by Fory serialization/deserialization.
- Implement zero-copy deserialization buffer support for inbound gRPC payloads, with a safe fallback path when zero-
copy cannot be applied. - Add golden codegen tests for generated file names and key signatures.
- Provide a runnable Rust server/client example using tonic + generated Fory codec bindings.
- Update compiler documentation for Rust gRPC code generation usage and constraints.
Skills:
Rust, gRPC (tonic), compiler/code generation, serialization internals, async Rust, testing, performance optimization.
Difficulty: Medium
Project size: 175 hours
Potential mentors: Chaokun Yang, Weipeng Wang
Source links:
Apache Fory Implement Fory Go gRPC integration
Description:
Apache Fory can generate high-performance Go model code from IDL, but it does not yet generate Go gRPC service
This project will add Go gRPC code generation to the Fory compiler. For each service definition, the compiler should
generate *_service.go (service interfaces) and *_grpc.go (gRPC transport bindings), using a Fory codec instead of
The implementation should follow Fory compiler conventions and prioritize low-overhead, performance-first behavior.
Potential Outcomes:
- Parse service IR and generate Go gRPC outputs for unary and streaming methods.
- Generate ServiceDesc, server interfaces, and client wrappers compatible with grpc-go.
- Register and use a custom Fory codec through grpc.CallOption for request/response serialization.
- Implement zero-copy deserialization buffer support for inbound gRPC payloads, with a safe fallback path when zero-copy cannot be applied.
- Add golden codegen tests for generated file names and key signatures.
- Provide a runnable Go server/client example using generated bindings and Fory codec.
- Update compiler documentation for Go gRPC code generation usage and constraints.
Skills:
Go, gRPC (grpc-go), compiler/code generation, serialization internals, testing, performance optimization.
Difficulty: Medium
Project size: 175 hours
Potential mentors: Chaokun Yang, Weipeng Wang
Source links:
Apache Fory Implement Fory JavaScript gRPC integration
Description:
Apache Fory currently lacks JavaScript/TypeScript gRPC service binding generation.
The generated service interface and gRPC binding outputs should follow existing JS/TS generator naming and layout conventions, stay dependency-light in Fory runtime, and keep runtime overhead low.
Potential Outcomes:
- Generate JS/TS service interface and gRPC binding outputs from service definitions, aligned with existing generator
layout conventions. - Wire request/response payload handling through generated Fory serializer/deserializer functions.
copy cannot be applied. - Coordinate with JS/TS type generation so emitted message/enum/union types are directly usable by generated gRPC
stubs. - Add golden codegen tests for generated file names and key signatures.
- Provide a runnable JS/TS server-client example using generated bindings and Fory codec.
- Update compiler documentation for JS/TS gRPC code generation usage and constraints.
Skills:
JavaScript/TypeScript, Node.js, gRPC (@grpc/grpc-js), compiler/code generation, serialization internals, testing, performance optimization.
Difficulty: Medium
Project size: 175 hours
Potential mentors: Chaokun Yang, Weipeng Wang
Source links:
Apache Fory Implement Fory Dart gRPC integration
Description
Apache Fory does not yet generate Dart gRPC service bindings.
This project will add Dart gRPC code generation to the Fory compiler. For each service definition, the compiler should generate Dart service interfaces and gRPC transport bindings that follow the existing Dart generator layout and use a Fory codec instead of protobuf runtime payload types.
The implementation must keep the Fory runtime free of gRPC dependencies. Any required gRPC glue should be emitted as generated helper code. Runtime behavior should remain low-overhead and allocation-conscious.
Potential Outcomes
- Generate Dart service interface and gRPC binding outputs from service definitions, aligned with current Dart generator conventions.
- Generate Dart gRPC server and client stubs for unary and streaming RPCs using Dart gRPC APIs.
- Wire request/response handling through generated Fory serializer and deserializer functions.
- Implement zero-copy deserialization buffer support for inbound gRPC payloads, with a safe fallback path when zero-copy cannot be applied.
- Coordinate with Dart type generation so emitted message, enum, and union types are directly usable by generated gRPC stubs.
- Add golden codegen tests for generated file names and key signatures.
- Provide a runnable Dart server/client example using generated bindings and the Fory codec.
- Update compiler documentation for Dart gRPC code generation usage and constraints.
Skills:Dart, gRPC (`grpc`), compiler/code generation, serialization internals, async programming, testing, performance optimization.
Difficulty: Medium
Project size:175 hours
Potential mentors:Chaokun Yang, Weipeng Wang
Source links:
- https://github.com/apache/fory/issues/3279
- https://github.com/apache/fory/issues/3281
- https://fory.apache.org/docs/next/compiler/compiler_guide
- https://github.com/apache/fory/tree/main/compiler
- https://github.com/apache/fory/tree/main/dart
- https://github.com/apache/fory/blob/main/dart/README.md
- https://github.com/apache/fory/tree/main/dart/packages/fory
Spark
SPIP Client-Side Metadata Caching for Spark Connect
This SPIP proposes adding a client-side schema cache for Spark Connect DataFrames.
Currently, every call to df.columns or df.schema triggers a synchronous gRPC analysis request to the server. While these are local and near-instant in Spark Classic, in Connect they average 277 ms on standard cloud setups (like AWS t3.medium). This makes iterative work extremely slow; we've measured a 13-second lag for 50 metadata calls in a typical ETL pipeline.
This delay is forcing developers to use a "Shadow Schema" pattern, where they manually track column names in local lists to avoid the RPC overhead. Since Spark DataFrames are immutable, we can fix this by caching the resolved schema on the client after the first request. Our POC shows this reduces the 13-second lag to about 250 ms (a 51× speedup) without breaking the core Spark Connect model.
I have followed the official SPIP template for the detailed breakdown below.
SIP
https://docs.google.com/document/d/1xTvL5YWnHu1jfXvjlKk2KeSv8JJC08dsD7mdbjjo9YE/edit?tab=t.0
Benchmark - https://docs.google.com/document/d/1ebX8CtTHN3Yf3AWxg7uttzaylxBLhEv-T94svhZg_uE/edit?tab=t.0