https://www.mail-archive.com/server-dev@james.apache.org/msg71462.html

A good long term objective for the PMC is to drop RabbitMQ in
favor of pulsar (third parties could package their own components using
RabbitMQ if they wishes...)

This means:

• Solve the bugs that were found during the Pulsar MailQueue review
• Pulsar MailQueue need to allow listing blobs in order to be
  deduplication friendly.
• Provide an event bus based on Pulsar
• Provide a task manager based on Pulsar
• Package a distributed server backed by pulsar, deprecate then replace
  the current one.
• (optionally) support mail queue priorities

While contributions would of course be welcomed on this topic, we could
offer it as part of GSOC 2022, and we could co-mentor it with mentors of
the Pulsar community (see [3])

[3] https://lists.apache.org/thread/y9s7f6hmh51ky30l20yx0dlz458gw259

Would such a plan gain traction around here ?

Apache ShardingSphere
Apache ShardingSphere is a distributed database middleware ecosystem, including 2 independent products, ShardingSphere JDBC and ShardingSphere Proxy presently. They all provide functions of data sharding, distributed transaction, and database orchestration.
Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere

 Background

ShardingSphere parser engine helps users parse a SQL to get the AST (Abstract Syntax Tree) and visit this tree to get SQLStatement (Java Object). At present, this parser engine can handle SQLs for `MySQL`, `PostgreSQL`, `SQLServer`, `openGauss` and `Oracle`, which means we have to understand different database dialect SQLs.
More details:
https://shardingsphere.apache.org/document/current/en/reference/sharding/parse/

Task

This issue is to solve the unsupported postgres sql about alter in this file . * ALTER OPERATOR

• ALTER POLICY

• ALTER PUBLICATION

• ALTER ROUTINE

• ALTER RULE

• ALTER SCHEMA

• ALTER SEQUENCE

• ALTER SERVER

• ALTER STATISTICS

• ALTER SUBSCRIPTION

• ALTER TABLE

• ALTER TEXT SEARCH

• ALTER TRIGGER

• ALTER TYPE

• ALTER VIEW

You can learn more here. *
You may need to try to get why it's not supported.(antlr4 grammar? or not implement visit method) You can use antlr4 plugins to help you to analyze. You may need to visit an official doc to check the grammar.

• 
  After you fix it, remember to add a new corresponding SQL case in SQL Cases and the expected parsed result in Expected Statment XML.

• Run SQLParserParameterizedTest and UnsupportedSQLParserParameterizedTest to make sure no exceptions.

Notice, these issues can be a good example.
support alter foreign table for pg/og
support alter materialized view for pg/og.

Relevant Skills

1. Master JAVA language
2. Have a basic understanding of Antlr g4 file
3. Be familiar with Postgres SQLs

Targets files

1. Postgres SQLs g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-postgresql/src/main/antlr4/org/apache/shardingsphere/sql/parser/autogen/PostgreSQLStatement.g4

Mentor

Trista Pan, PMC of Apache ShardingSphere, https://tristazero.github.io

Zhengqiang Duan, Committer of ApacheShardingSphere, https://github.com/strongduanmu

Apache ShardingSphere

 
Apache ShardingSphere is a distributed database middleware ecosystem, including 2 independent products, ShardingSphere JDBC and ShardingSphere Proxy presently. They all provide functions of data sharding, distributed transaction, and database orchestration.
Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere 

Background

ShardingSphere parser engine helps users parse a SQL to get the AST (Abstract Syntax Tree) and visit this tree to get SQLStatement (Java Object). At present, this parser engine can handle SQLs for `MySQL`, `PostgreSQL`, `SQLServer`, `openGauss` and `Oracle`, which means we have to understand different database dialect SQLs.
 
More details:
https://shardingsphere.apache.org/document/current/en/reference/sharding/parse/ 

Task

This issue is to solve the unsupported postgres sql about alter in this file . * CALL

• CHECKPOINT

• CLOSE

• CLUSTER

• COMMENT

• COPY

• CREATE ACCESS METHOD

• CREATE AGGREGATE

• CREATE CAST

• CREATE COLLATION

• CREATE EVENT TRIGGER

• CREATE FOREIGN DATA WRAPPER

• CREATE FOREIGN TABLE

• CREATE GROUP

• CREATE MATERIALIZED VIEW

• CREATE OPERATOR

• CREATE POLICY

• CREATE PUBLICATION

 
You can learn more here. *
You may need to try to get why it's not supported.(antlr4 grammar? or not implement visit method) You can use antlr4 plugins to help you to analyze. You may need to visit an official doc to check the grammar.

• After you fix it, remember to add a new corresponding SQL case in SQL Cases and expected parsed result in Expected Statment XML.

• Run SQLParserParameterizedTest and UnsupportedSQLParserParameterizedTest to make sure no exceptions.

 
Notice, these issues can be a good example.
support alter foreign table for pg/og
support alter materialized view for pg/og.

Relevant Skills

 
1. Master JAVA language
2. Have a basic understanding of Antlr g4 file
3. Be familiar with Postgres SQLs 

Targets files

 
1. Postgres SQLs g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-postgresql/src/main/antlr4/org/apache/shardingsphere/sql/parser/autogen/PostgreSQLStatement.g4

Mentor

Zhengqiang Duan, Committer of Apache ShardingSphere, duanzhengqiang@apache.org
Haoran Meng, PMC of Apache ShardingSphere, menghaoran@apache.org

Apache ShardingSphere

Apache ShardingSphere is a distributed database middleware ecosystem, including 2 independent products, ShardingSphere JDBC and ShardingSphere Proxy presently. They all provide functions of data sharding, distributed transaction, and database orchestration.
Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere

Background

ShardingSphere parser engine helps users parse a SQL to get the AST (Abstract Syntax Tree) and visit this tree to get SQLStatement (Java Object). At present, this parser engine can handle SQLs for `MySQL`, `PostgreSQL`, `SQLServer`, `openGauss` and `Oracle`, which means we have to understand different database dialect SQLs.
 
More details:
https://shardingsphere.apache.org/document/current/en/reference/sharding/parse/

Task

This issue is to solve the unsupported postgres sql about alter in this file . * ALTER OPERATOR

• ALTER POLICY

• ALTER PUBLICATION

• ALTER ROUTINE

• ALTER RULE

• ALTER SCHEMA

• ALTER SEQUENCE

• ALTER SERVER

• ALTER STATISTICS

• ALTER SUBSCRIPTION

• ALTER TABLE

• ALTER TEXT SEARCH

• ALTER TRIGGER

• ALTER TYPE

• ALTER VIEW

 
You can learn more here. *
You may need to try to get why it's not supported.(antlr4 grammar? or not implement visit method) You can use antlr4 plugins to help you to analyze. You may need to visit an official doc to check the grammar.

• After you fix it, remember to add a new corresponding SQL case in SQL Cases and expected parsed result in Expected Statment XML.

• Run SQLParserParameterizedTest and UnsupportedSQLParserParameterizedTest to make sure no exceptions.

 
Notice, these issues can be a good example.
support alter foreign table for pg/og
support alter materialized view for pg/og.

Relevant Skills

1. Master JAVA language
2. Have a basic understanding of Antlr g4 file
3. Be familiar with Postgres SQLs

Targets files

1. Postgres SQLs g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-postgresql/src/main/antlr4/org/apache/shardingsphere/sql/parser/autogen/PostgreSQLStatement.g4

Mentor

Zhengqiang Duan, Committer of Apache ShardingSphere, duanzhengqiang@apache.org
Haoran Meng, PMC of Apache ShardingSphere, menghaoran@apache.org

Apache ShenYu (incubating)

A High-performance,multi-protocol,extensible,responsive API Gateway. Compatible with a variety of mainstream framework systems, support hot plug, users can customize the development, meet the current situation and future needs of users in a variety of scenarios, experienced the temper of large-scale scenes

Website: https://shenyu.apache.org

GitHub: https://github.com/apache/incubator-shenyu

Linked GitHub Issue: https://github.com/apache/incubator-shenyu/issues/2890

Background

1. ShenYu is still vacant with helm deployment, so we need to write charts for it, and then complete the integration test.

1. Shenyu already has a relatively complete integration testing framework, but some plug-ins have not been tested, and some tests are not perfect.

Task

• Write helm chart for Apache ShenYu
• Complete the integration test of deploying Apache ShenYu with helm in Kubernetes
• Documentation for helm deployment
• Complete the integration test of the Oauth2 plugin
• Improve the integration test of other existing plugin

Recommended Skills

Familiar with Java

Know the usage of spring-framework

Have some knowledge about Kubernetes and Docker

Mentor

Kunshuai Zhu, Committer of Apache ShenYu, https://github.com/JooKS-me, jooks@apache.org

Placeholder for tasks that could be undertaken in this year's GSoC.

Ideas (extracted from the "dev" ML):

1. Redesign and modularize the "ml" package
   -> main goal: enable multi-thread usage.
2. Abstract the linear algebra utilities
   -> main goal: allow switching to alternative implementations.
3. Redesign and modularize the "random" package
   -> main goal: general support of low-discrepancy sequences.
4. Refactor and modularize the "special" package
   -> main goals: ensure accuracy and performance and better API,
   add other functions.
5. Upgrade the test suite to Junit 5
   -> additional goal: collect a list of "odd" expectations.

Other suggestions welcome, as well as

• delineating additional and/or intermediate goals,
• signalling potential pitfalls and/or alternative approaches to the intended goal(s).

The existing compaction strategies have a number of drawbacks that make all three unsuitable as a general use compaction strategy, for example STCS creates giant files that are hard to back up, mess with read performance and the page cache, and led to many of the early re-open bugs. LCS improved dramatically on this but also has various issues e.g. lack of performant full compaction or due to the strict leveling with e.g. bulk loading when writes exceed the rate we can do the L0 - L1 promotion.

In this talk I proposed a novel compaction strategy that aims to expose a single tunable that the user can control for the read amplification. Raise the min_threshold_levels and you tradeoff read/space performance for write performance. Since then a proof of concept patch has been published along with some rudimentary documentation but this is still not tracked in Jira.

The remaining work here is

1. Validate the algorithm is correct via test suites and performance testing stress testing and benchmarking with OSS tools (e.g. cassandra-stress, tlp-stress, or ndbench). When issues are found (there likely will be issues as the patch is a PoC), devise how to adjust the algorithm and implementation appropriately. Key metric of success is we can run Cassandra stably for more than 24 hours while applying sustained load, with minimal compaction load (and also compaction can keep up).

2. Do more in depth experiments measuring performance across a wide range of workloads (e.g. write heavy, read heavy, balanced, time series, register update, etc ...) and in comparison with LCS (leveled), STCS (size tiered), and TWCS (time window). Key metrics of success are establishing that as we tune max_read_per_read we should get more predictable read latency under low system load (ρ < 30%) while not degrading at high system load (ρ > 70%), and we should match LCS performance under low load while doing better at high load.

Once this is validated a Cassandra blog post reporting on the findings (positive or negative) may be advisable.

It would be desirable to develop a Beam Sink that supports all of the 'best practices' like throttling, auto-sharding, exactly-once capabilities, etc.

A design proposal is here: https://docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org if you have questions, or comment here.

In Stream processing, we have many methods, starting from the primitive checkpoint-and-replay to a more fancy version of reconfiguration and reinitiation of stream workloads. We aim to find a way to find the most effective and efficient way of reconfiguring stream workloads. Sub-issues are to be created later on.

Nemo has a policy layer that allows powerful optimization with configurable runtime modules. In terms of caching, it is possible to identify frequently used data and decide to cache them in-memory ahead of execution, without user annotation.

Implementation would need:

• On policy layer, build compile-time pass that identify reused data and mark them as cached
• On runtime, design and implement caching mechanism that manages per-executor cached data and discard them when these are no longer used.

Currently, Nemo doesn't have a spill mechanism. This makes executors prone to memory problems such as OOM(Out Of Memory) or GC when task data is large. For example, handling skewed shuffle data in Nemo results in OOM and executor failure, as all data has to be handled in-memory.

We need to spill in-memory data to secondary storage when there are not enough memory in executor.

In-memory caching and spilling are essential features in in-memory big data processing frameworks, and Nemo needs one.

• Identify and persist frequently used data and unpersist it when its usage ended
• Spill in-memory data to disk upon memory pressure