Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere 

Background

Currently we try to promote StorageNode as major CRD to represent a set of storage units for ShardingSphere.

Task

The elementary task is that the storage node controller could manage the lifecycle of  a set of storage units, like PostgreSQL, in kubernetes. 

We don't hope to create another wheel like pg-operator. So consider using a predefined parameter group to generate the target CRD.

• [ ] Generate DistSQL according to the Golang struct `EncryptionRule`
• [ ] Generate DistSQL according to the Golang struct `ShardingRule`
• [ ] Generate DistSQL according to the Golang struct `ReadWriteSplittingRule`
• [ ] Generate DistSQL according to the Golang struct `MaskRule`
• [ ] Generate DistSQL according to the Golang struct `ShadowRule`

  Relevant Skills

1. Master Go language, Ginkgo test framework
2. Have a basic understanding of Apache ShardingSphere Concepts and DistSQL

Targets files

DistSQL Converter - https://github.com/apache/shardingsphere-on-cloud/blob/main/shardingsphere-operator/pkg/distsql/converter.go, etc.

Example

A struct defined as below:

```golang
type EncryptRule struct{}
func (t EncryptRule) ToDistSQL() string {}
```
While invoking ToDistSQL() it will generate a DistSQL regarding a EncryptRule like:

```SQL
CREATE ENCRYPT RULE t_encrypt (....
```

References:

• https://shardingsphere.apache.org/document/current/en/user-manual/shardingsphere-proxy/distsql/syntax/rdl/rule-definition/encrypt/create-encrypt-rule/

Mentor
Liyao Miao, Committer of Apache ShardingSphere, miaoliyao@apache.org

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache

Background

SkyWalking BanyanDB is an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data.

Objectives

1. Support EXPLAIN[1] for both measure query and stream query
2. Add self-observability including trace and metrics for query subsystem
3. Support EXPLAIN in the client SDK & CLI and add query plan visualization in the UI

[1]: EXPLAIN in MySQL

Recommended Skills

1. Familiar with Go
2. Have a basic understanding of database query engine
3. Have an experience of Apache SkyWalking or other APMs

Mentor

Mentor: Jiajing Lu, Apache SkyWalking PMC, lujiajing@apache.org^{Image Removed}

Mentor: Hongtao Gao, Apache SkyWalking PMC, Apache ShardingSphere PMC, hanahmily@apache.org^{Image Removed}

.org

Background

SkyWalking BanyanDB is an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data.

Objectives

1. Fully unify/merge the query planner and executor for Measure and TopN

Recommended Skills

1. Familiar with Go
2. Have a basic understanding of database query engine
3. Have an experience of Apache SkyWalking

Mentor

• Mentor: Jiajing Lu, Apache SkyWalking PMC, lujiajing@apache.org^{Image Removed}
• Mentor: Hongtao Gao, Apache SkyWalking PMC, Apache ShardingSphere PMC, hanahmily@apache.org^{Image Removed}
• Mailing List: dev@skywalking.apache.org

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere 

Background

There is a proposal about new CRD Cluster and ComputeNode as belows:

• #167
• #166

Currently we try to promote StorageNode as major CRD to represent a set of storage units for ShardingSphere.

Task

The elementary task is that the storage node controller could manage the lifecycle of a set of storage units, like PostgreSQL, in kubernetes.

We don't hope to create another wheel like pg-operator. So consider using a predefined parameter group to generate the target CRD.

• [ ] Create a PostgreSQL cluster while a StorageNode with pg parameters is created
• [ ] Update the PostgreSQL cluster while updated StorageNode
• [ ] Delete the PostgreSQL cluster while deleted StorageNode. Notice this may need a deletion strategy.
• [ ] Reconciling StorageNode according to the status of PostgreSQL cluster.
• [ ] The status of StorageNode would be consumed by common storage units related DistSQLs

Relevant Skills

1. Master Go language, Ginkgo test framework
2. Have a basic understanding of Apache ShardingSphere Concepts
3. Be familiar with Kubernetes Operator, kubebuilder framework

Targets files

StorageNode Controller - https://github.com/apache/shardingsphere-on-cloud/blob/main/shardingsphere-operator/pkg/controllers/storagenode_controller.go

Mentor

Liyao Miao, Committer of Apache ShardingSphere,  miaoliyao@apache.org^{Image Added}

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org^{Image Added}

Now the deployment methods for SkyWalking are limited, we only have Helm Chart for users to deploy in Kubernetes, other users that are not using Kubernetes have to do all the house keeping stuffs to set up SkyWalking on, for example, VM.

This issue aims to add a Terraform provider, so that users can conveniently  spin up a cluster for demonstration or testing, we should evolve the provider and allow users to customize as their need and finally users can use this in their production environment.

In this task, we will mainly focus on the support for AWS. In the Terraform provider, users need to provide their access key / secret key, and the provider does the rest stuffs: create VMs, create database/OpenSearch or RDS, download SkyWalking tars, configure the SkyWalking, and start the SkyWalking components (OAP/UI), create public IPs/domain name, etc.

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere

Background

There is a proposal about the background of ChaosEngineering as belows:

Introduce ChaosEngineering for ShardingSphere #32
And we also proposed a generic controller for ShardingSphereChaos as belows:

[GSoC 2023] Introduce New CRD ShardingSphereChaos #272
The ShardingSphereChaos controller is aiming at different chaos tests. This JVMChaos is an important one.

Task

Write several scripts to implement different JVMChaos for main features of ShardingSphere. The specific case list is as follows.

• Add scripts injecting chaos to DataSharding
• Add scripts injecting chaos to ReadWritingSplitting
• Add scripts injecting chaos to DatabaseDiscovery
• Add scripts injecting chaos to Encryption
• Add scripts injecting chaos to Mask
• Add scripts injecting chaos to Shadow
  Basically, these scripts will cause unexpected behaviour while executing the related. DistSQL.

Relevant Skills

• Master Go language, Ginkgo test framework
• Have a deep understanding of Apache ShardingSphere concepts and practices.
• JVM byte mechanisms like ByteMan, ByteBuddy.

Targets files

JVMChaos Scripts - https://github.com/apache/shardingsphere-on-cloud/chaos/jvmchaos/scripts/

Mentor
Liyao Miao, Committer of Apache ShardingSphere, miaoliyao@apache.org
Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org

Background

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance. Currently, ShenYu has good usability and performance in microservice scenarios. However, ShenYu's support for Kubernetes is still relatively weak.

Tasks

1. Support the registration of microservices deployed in K8s Pod to shenyu-admin and use K8s as the register center.
2. Discuss with mentors, and complete the requirements design and technical design of Shenyu K8s Register Center.
3. Complete the initial version of Shenyu K8s Register Center.
4. Complete the CI test of Shenyu K8s Register Center, verify the correctness of the code.
5. Write the necessary documentation, deployment guides, and instructions for users to connect microservices running inside the K8s Pod to ShenYu

Relevant Skills

1. Know the use of Apache ShenYu, especially the register center
2. Familiar with Java and Golang
3. Familiar with Kubernetes and can use Java or Golang to develop

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org/
Github: https://github.com/apache/shardingsphere 

Background

There is a proposal about the background of ChaosEngineering as belows:

• Introduce ChaosEngineering for ShardingSphere · Issue #32 · apache/shardingsphere-on-cloud (github.com)

The ShardingSphereChaos controller is aiming at different chaos tests. 

Task

Propose a generic controller for ShardingSphereChaos, which reconcile CRD ShardingSphereChaos, prepare, execute and verify test.

• [ ] Support common ShardingSphere features, prepare test rules and dataset
• [ ] Generating chaos type according to the backend implementation
• [ ] Verify testing result with DistSQL or other tools

Relevant Skills

1. Master Go language, Ginkgo test framework
2. Have a deep understanding of Apache ShardingSphere concepts and practices.
3. Kubernetes operator pattern, kube-builder 

Targets files

ShardingSphereChaos Controller -

Background

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance. Currently, ShenYu has good usability and performance in microservice scenarios. However, ShenYu's support for Kubernetes is still relatively weak.

Tasks

1. Discuss with mentors, and complete the requirements design and technical design of shenyu-ingress-controller.
2. Complete the initial version of shenyu-ingress-controller, implement the reconcile of k8s ingress api, and make ShenYu as the ingress gateway of k8s.
3. Complete the ci test of shenyu-ingress-controller, verify the correctness of the code.

Relevant Skills

1. Know the use of Apache ShenYu
2. Familiar with Java and Golang
3. Familiar with Kubernetes and can use java or golang to develop Kubernetes Controller

Description

Issues ： https://github.com/apache/shenyu/issues/4438
website ： https://shenyu.apache.org/shardingsphere-on-cloud/shardingsphere-operator/pkg/controllers/chaos_controller.go, etc.

Mentor

Liyao Miao, Committer of Apache ShardingSphere,  miaoliyao@apache.org^{Image Added}

Chuxin Chen, Committer of Apache ShardingSphere, tuichenchuxin@apache.org^{Image Added}

Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures. This year we will proceed on log clustering implementation with a revised architecture and this task will require student to focus on algorithm optimiztion for the clustering technique.

Background

At present, shenyu needs to manually check whether the license is correct one by one when releasing the version.

Tasks

1. Discuss with the tutor to complete the requirement design and technical design of the scanning license.
2. Finished scanning the initial version of the license.
3. Complete the corresponding test.

Relevant Skills

1. Familiar with Java.

Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures. This task is about enhancing Python agent performance, the tracking issue can be seen here -< https://github.com/apache/skywalking/issues/10408

Apache SkyWalking is an application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures. This year we will proceed on log clustering implementation with a revised architecture and this task will require student to focus on Flink and its integration with SkyWalking OAP.

Background

SkyWalking BanyanDB is an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data.

Objectives

1. Support EXPLAIN[1] for both measure query and stream query
2. Add self-observability including trace and metrics for query subsystem
3. Support EXPLAIN in the client SDK & CLI and add query plan visualization in the UI

[1]: EXPLAIN in MySQL

Recommended Skills

1. Familiar with Go
2. Have a basic understanding of database query engine
3. Have an experience of Apache SkyWalking or other APMs

Mentor

• Mentor: Jiajing Lu, Apache SkyWalking PMC, lujiajing@apache.org^{Image Added}
• Mentor: Hongtao Gao, Apache SkyWalking PMC, Apache ShardingSphere PMC, hanahmily@apache.org^{Image Added}
• Mailing List: dev@skywalking.apache.org

Background

SkyWalking BanyanDB is an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data.

Objectives

1. Fully unify/merge the query planner and executor for Measure and TopN

Recommended Skills

1. Familiar with Go
2. Have a basic understanding of database query engine
3. Have an experience of Apache SkyWalking

Mentor

• Mentor: Jiajing Lu, Apache SkyWalking PMC, lujiajing@apache.org^{Image Added}
• Mentor: Hongtao Gao, Apache SkyWalking PMC, Apache ShardingSphere PMC, hanahmily@apache.org^{Image Added}
• Mailing List: dev@skywalking.apache.org

Now the deployment methods for SkyWalking are limited, we only have Helm Chart for users to deploy in Kubernetes, other users that are not using Kubernetes have to do all the house keeping stuffs to set up SkyWalking on, for example, VM.

This issue aims to add a Terraform provider, so that users can conveniently  spin up a cluster for demonstration or testing, we should evolve the provider and allow users to customize as their need and finally users can use this in their production environment.

In this task, we will mainly focus on the support for AWS. In the Terraform provider, users need to provide their access key / secret key, and the provider does the rest stuffs: create VMs, create database/OpenSearch or RDS, download SkyWalking tars, configure the SkyWalking, and start the SkyWalking components (OAP/UI), create public IPs/domain name, etc.

Background

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance. Currently, ShenYu has good usability and performance in microservice scenarios. However, ShenYu's support for Kubernetes is still relatively weak.

Tasks

1. Support the registration of microservices deployed in K8s Pod to shenyu-admin and use K8s as the register center.
2. Discuss with mentors, and complete the requirements design and technical design of Shenyu K8s Register Center.
3. Complete the initial version of Shenyu K8s Register Center.
4. Complete the CI test of Shenyu K8s Register Center, verify the correctness of the code.
5. Write the necessary documentation, deployment guides, and instructions for users to connect microservices running inside the K8s Pod to ShenYu

Relevant Skills

1. Know the use of Apache ShenYu, especially the register center
2. Familiar with Java and Golang
3. Familiar with Kubernetes and can use Java or Golang to develop

Background

Apache ShenYu is a Java native API Gateway for service proxy, protocol conversion and API governance. Currently, ShenYu has good usability and performance in microservice scenarios. However, ShenYu's support for Kubernetes is still relatively weak.

Tasks

1. Discuss with mentors, and complete the requirements design and technical design of shenyu-ingress-controller.
2. Complete the initial version of shenyu-ingress-controller, implement the reconcile of k8s ingress api, and make ShenYu as the ingress gateway of k8s.
3. Complete the ci test of shenyu-ingress-controller, verify the correctness of the code.

Relevant Skills

1. Know the use of Apache ShenYu
2. Familiar with Java and Golang
3. Familiar with Kubernetes and can use java or golang to develop Kubernetes Controller

Description

Issues ： https://github.com/apache/shenyu/issues/4438
website ： https://shenyu.apache.org/

Background

At present, shenyu needs to manually check whether the license is correct one by one when releasing the version.

Tasks

1. Discuss with the tutor to complete the requirement design and technical design of the scanning license.
2. Finished scanning the initial version of the license.
3. Complete the corresponding test.

Relevant Skills

1. Familiar with Java.

Background

Shenyu is a native API gateway for service proxy, protocol translation and API governance. It can manage and maintain the API through Shenyu-admin, and support internationalization in Chinese and English. Unfortunately, Shenyu-admin is only internationalized on the front end. The message prompt returned by the back-end interface is still in English. Therefore, we need to implement internationalization support for the back-end interface.This will lay a good foundation for shenyu to move towards more language support.

Relevant skills

• Related skills spring resources
• Spring Internationalization
• Front-end react framework

API reference

            java.util.Locale;
            org.springframework.context.MessageSource;
            org.springframework.context.support.ResourceBundleMessageSource; 

Interface effect example

            ## zh request example
            POST http://localhost:9095/plugin
            Content-Type: application/json
            Location: cn-zh
            X-Access-Token: xxx
            {
            "name": "test-create-plugin",
            "role": "test-create-plugin",
            "enabled": true,
            "sort": 

Background

Shenyu is a native API gateway for service proxy, protocol translation and API governance. It can manage and maintain the API through Shenyu-admin, and support internationalization in Chinese and English. Unfortunately, Shenyu-admin is only internationalized on the front end. The message prompt returned by the back-end interface is still in English. Therefore, we need to implement internationalization support for the back-end interface.This will lay a good foundation for shenyu to move towards more language support.

Relevant skills

• Related skills spring resources
• Spring Internationalization
• Front-end react framework

API reference

            java.util.Locale;
            org.springframework.context.MessageSource;
            org.springframework.context.support.ResourceBundleMessageSource; 

Interface effect example

            ## zh request example
            POST http://localhost:9095/plugin
            Content-Type: application/json
            Location: cn-zh
            X-Access-Token: xxx
            {
            "name": "test-create-plugin",
            "role": "test-create-plugin",
            "enabled": true,
            "sort": 100
            }
            Respone
            {
            "code": 600,
            "message": "未登录"
            }
            
            ### en request example
            POST http://localhost:9095/plugin
            Content-Type: application/json
            Location: en
            X-Access-Token: xxx
            {
            "name": "test-create-plugin",
            "role": "test-create-plugin",
            "enabled": true,
            "sort": 100
            }
            Respone
            {
            "code": 600,
             "code": 600,
            "message": "token is error"
            } 

Task List

 "message": "token is error"
            } 

Task List

• The task discussed with the tutor how to achieve the internationalization of shenyu-admin background
• Some prompt message translation
• Get through the internationalization of front-end, obtain the client region information through http protocol, support the language of the corresponding region.
• Leave the extension of other multi-language internationalization support interface, so as to facilitate the localization transformation of subsequent users.

Background:

Shenyu is a native API gateway for service proxy, protocol translation and API governance. but Shenyu lack of End-To-End Tests.

Relevant skills：

1.Understand the architecture of ShenYu

2.Understand SpringCloud micro-service and ShenYu SpringCloud proxy plugin.

3.Understand ShenYu e2e framework and architecture.

How to coding

1.please refer to org.apache.shenyu.e2e.testcase.plugin.DividePluginCases

How to test

1.start shenyu admin in docker

2.start shenyu boostrap in docker

3.run test case org.apache.shenyu.e2e.testcase.plugin.PluginsTest#testDivide

Task List

1.develop e2e tests of the springcloud plug-in.

2.write shenyu e2e springcloud plugin documentation in shenyu-website.

3.refactor the existing plugin test cases.

Links:

website： https://shenyu.apache.org/

issues: https://github.com/apache/shenyu/issues/4474

Background
Apache Traffic Control is a Content Delivery Network (CDN) control plane for large scale content distribution.

Traffic Control currently requires Apache Traffic Server as the underlying cache. Help us expand the scope by integrating with the very popular Varnish Cache.

There are multiple aspects to this project:

• Configuration Generation: Write software to build Varnish configuration files (VCL). This code will be implemented in our Traffic Ops and cache client side utilities, both written in Go.

• Health Monitoring: Implement monitoring of the Varnish cache health and performance. This code will run both in the Traffic Monitor component and within Varnish. Traffic Monitor is written in Go and Varnish is written in C.

• Testing: Adding automated tests for new code

Skills:

• Proficiency in Go is required
• A basic knowledge of HTTP and caching is preferred, but not required for this project.

Background:

Shenyu is a native API gateway for service proxy, protocol translation and API governance. but Shenyu lack of End-To-End Tests.

Relevant skills：

1.Understand the architecture of ShenYu

2.Understand SpringCloud micro-service and ShenYu SpringCloud proxy plugin.

3.Understand ShenYu e2e framework and architecture.

How to coding

1.please refer to org.apache.shenyu.e2e.testcase.plugin.DividePluginCases

How to test

1.start shenyu admin in docker

2.start shenyu boostrap in docker

3.run test case org.apache.shenyu.e2e.testcase.plugin.PluginsTest#testDivide

Task List

1.develop e2e tests of the springcloud plug-in.

2.write shenyu e2e springcloud plugin documentation in shenyu-website.

3.refactor the existing plugin test cases.

Links:

website： https://shenyu.apache.org/

issues: https://github.com/apache/shenyutrafficcontrol/issues/44747076

Background
Apache Traffic Control is a Content Delivery Network (CDN) control plane for large scale content distribution.

Traffic Control currently requires Apache Traffic Server as the underlying cache. Help us expand the scope by integrating with the very popular Varnish Cache.

There are multiple aspects to this project:

• Configuration Generation: Write software to build Varnish configuration files (VCL). This code will be implemented in our Traffic Ops and cache client side utilities, both written in Go.

• Health Monitoring: Implement monitoring of the Varnish cache health and performance. This code will run both in the Traffic Monitor component and within Varnish. Traffic Monitor is written in Go and Varnish is written in C.

• Testing: Adding automated tests for new code

Skills:

• Proficiency in Go is required
• A basic knowledge of HTTP and caching is preferred, but not required for this project.

Apache Doris
Apache Doris is a real-time analytical database based on MPP architecture. As a unified platform that supports multiple data processing scenarios, it ensures high performance for low-latency and high-throughput queries, allows for easy federated queries on data lakes, and supports various data ingestion methods.
Page: https://doris.apache.org

Github: https://github.com/apache/doris

Background

Apache Doris accelerates high-concurrency queries utilizing page cache, where the decompressed data is stored.
Currently, the page cache in Apache Doris uses a simple LRU algorithm, which reveals a few problems: 

• Hot data will be phased out in large queries
• The page cache configuration is immutable and does not support GC.

Task

• Phase One: Identify the impacts on queries when the decompressed data is stored in memory and SSD, respectively, and then determine whether full page cache is required.

• Phase Two: Improve the cache strategy for Apache Doris based on the results from Phase One.

Learning Material

Page: https://doris.apache.org
Github: https://github.com/apache/doris

Mentor

• Mentor: Yongqiang Yang, Apache Doris PMC member & Committer, yangyongqiang@apache.org ^{Image Added}
• Mentor: Haopeng Li, Apache Doris PMC member & Committer, lihaopeng@apache.org^{Image Added}  
• Mailing List: dev@doris.apache.org

https://github.com/apache/trafficcontrol/issues/7076