Contents
...
Debezium integration for live sync
Debezium provides a mean to transform changes from RDBMS, MongoDB and Cassandra into Kafka messages.
This tool can be leveraged to implement a listener-like approach to enable a "live-sync" mechanism from External Resources not requiring ConnId.
Synapse
Open Telemetry based Tracing for Apache Synapse
Currently, Apache Synapse does not have sophisticated support for modern tracing standardized. Therefore this new feature is intended to implement OpenTelemetery based tracing implementation for apache synapse.
This feature will include request-response training and inbound/outbound tracing at the transport level and the orchestration layer. Further, this also needs a really good investigation on Opentelemetry specification[1] and the Apache synapse transport component [1].
Relevant Skills
- JAVA language
- Understanding about observability
- Integration and Synapse configuration language.
[1]https://opentelemetry.io/
[2] http://synapse.apache.org/userguide/transports/pass_through.html
Containerization of Integration Framework
The world has been moved towards container friendly products, microservices, and cloud-native technologies. However, the integration problem solved by the ESB architecture still exists and is handled by different entities. Even though Synapse is an ESB, it has most of the qualities (ex: faster startup time, low resource consumption, etc.) that are important for containers and microservices integrations. So with a little effort, we could make Synapse a container friendly product that suits any architectural style for doing the integration. As a very first step, Making Synapse a container friendly framework is important.
Task
- Reduce memory footprint of Synapse
- Create docker, Kubernetes artifacts [https://issues.apache.org/jira/projects/SYNAPSE/issues/SYNAPSE-1111]
- Documentation
- Publish the containers in the docker-hub
Relevant Skills
- JAVA language
- Kubernetes
- Docker
- Integration and Synapse configuration language.
Possible Mentors
Vanjikumaran Sivajothy
Mohamad Anfar Mohamad Shafreen
Isuru Udana
Open Telemetry based Tracing for Apache Synapse
Containerization of Integration Framework
The world has been moved towards container friendly products, microservices, and cloud-native technologies. However, the integration problem solved by the ESB architecture still exists and is handled by different entities. Even though Synapse is an ESB, it has most of the qualities (ex: faster startup time, low resource consumption, etc.) that are important for containers and microservices integrations. So with a little effort, we could make Synapse a container friendly product that suits any architectural style for doing the integration. As a very first step, Making Synapse a container friendly framework is important.
Task
- Reduce memory footprint of Synapse
- Create docker, Kubernetes artifacts [https://issues.apache.org/jira/projects/SYNAPSE/issues/SYNAPSE-1111]
- Documentation
- Publish the containers in the docker-hub
Currently, Apache Synapse does not have sophisticated support for modern tracing standardized. Therefore this new feature is intended to implement OpenTelemetery based tracing implementation for apache synapse.
This feature will include request-response training and inbound/outbound tracing at the transport level and the orchestration layer. Further, this also needs a really good investigation on Opentelemetry specification[1] and the Apache synapse transport component [1].Relevant Skills
- JAVA language
- Kubernetes
- Docker
- Integration and Synapse configuration language.
[1]https://opentelemetry.io/
[2] http://synapse.apache.org/userguide/transports/pass_through.html
Possible Mentors
Vanjikumaran Sivajothy
Mohamad Anfar Mohamad Shafreen
Isuru Udana
StreamPipes
More powerful real-time visualizations for StreamPipes
Apache StreamPipes
Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.
Background
Current wrappers such as standalone (JVM, Siddhi) or distributed (Flink) already allow to develop new processors in the given runtime environment. The idea is to extend the list of standalone runtime wrappers to also support pure Python processors. We already got a minimal working version that however is pretty inflexible and still relies on Java as a proxy to the pipeline management in the backend service for the model declaration in the setup phase ( capabilities, requirements, static properties of a processor) as well as the actual invocation in the execution phase ( receiving specific configuration from pipeline management when pipeline is started). This issue is to track the status of the development.
Tasks
- Add API endpoints as an interface for registration/invocation ( partly done)
- Port relevant model classes over to Python (declaration + invocation descriptions)
- Implement support for various transport protocols and transport formats
- Implement dev friendly alternative to Java builder pattern for model declaration
- Implement overall runtime logic for Python wrapper
Relevant Skills
0. Don't be afraid! We'll guide you through your first steps with StreamPipes.
- Excellent Python skills
- Excellent understanding of stream processing paradigm incl. message broker such as Kafka, MQTT, etc
- Good Understanding of RESTful web services (HTTP, etc.)
- Basic Java skills to understand existing wrapper logic
Info
- SIP-02 to collect design decisions https://cwiki.apache.org/confluence/display/STREAMPIPES/SIP-02+Python+wrapper
- Current python runtime wrapper implementation: https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper-python
- POC example: https://github.com/apache/incubator-streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/python
Mentor
Patrick Wiener, PPMC Apache StreamPipes (wiener@apache.org)
Introduce event windowing to the StreamPipes core/sdk
Apache StreamPipes
Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.
Background
Currently, window logic can be individually defined per pipeline element. The whole windowing logic needs to be declared in the controller and runtime logic needs to be individually added based on the selected runtime wrapper (Java, Siddhi, Flink, etc...).
As many data processors benefit from using window-functions (i.e PEs such as Event Counter, Count Aggregation, Rate Limiter), windowing logic is often duplicated as it needs to be implemented for every new pipeline element. In addition, the feature set of supported window operators differs (and often depends on the developer) as it is unclear which windows and parameters should/can be offered.
Therefore, adding support for explicit window semantics to the SDK/Core would make implementing data processors and sinks using windows much easier and less error-prone.
Tasks
- Design and introduce new processor and controller classes for windowed event processors (e.g., WindowedDataProcessor) which handle the windowing logic internally and only expose the higher-level methods to users (i.e onCurrentEvent, onExpiredEvent, etc...).
- Implement internal logic for few window functions (i.e TimeWindow, LengthWindow, TimeBatchWindow, LengthBatchWindow, etc...)
- Write a few sample pipeline-elements using your new API!
Relevant Skills
- Basic knowledge in StreamPipes core (cloning the repo, going through the codebase/documents would do).
- Basic knowledge of stream analytics window functions (this is not a must, but it's awesome if you know your way around analytics window functions).
- Some Java experience.
Learning Material
For StreamPipes:
- https://streampipes.apache.org/docs/
- https://streampipes.apache.org/media.html
- https://github.com/apache/incubator-streampipes
- https://github.com/apache/incubator-streampipes-extensions
For Streaming Analytics:
- https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-window-functions
- https://www.mikulskibartosz.name/difference-between-tumbling-and-sliding-window/
- https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/developing-storm-applications/content/understanding_sliding_and_tumbling_windows.html
For the context for the issue:
Mentor
- Grainier Perera (grainier [at] apache.org).
Currently, the live dashboard (implemented in Angular) offers an initial set of simple visualizations, such as line charts, gauges, tables and single values. More advanced visualizations, especially those relevant for condition monitoring tasks (e.g., monitoring sensor measurements from industrial machines) is easy. Visualizations can be flexibly created by users and there is an SDK that allows to express requirements (e.g., based on data type or semantic type) for visualizations to better guide users through the creation process.
Tasks
- Extend the set of real-time visualizations in StreamPipes, e.g., by integrating existing visualizations from Apache ECharts.
- Improve the existing dashboard, e.g., by introducing better filtering or more advanced customization options.
Relevant Skills
0. Don't be afraid! We'll guide you through your first steps with StreamPipes.
- Angular
- Basic knowledge of Apache ECharts
Mentor
Dominik Riemer, PPMC Apache StreamPipes (riemer@apache.org)
Introduce event windowing to the StreamPipes core/sdk
Apache StreamPipes
Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.
Background
Currently, window logic can be individually defined per pipeline element. The whole windowing logic needs to be declared in the controller and runtime logic needs to be individually added based on the selected runtime wrapper (Java, Siddhi, Flink, etc...).
As many data processors benefit from using window-functions (i.e PEs such as Event Counter, Count Aggregation, Rate Limiter), windowing logic is often duplicated as it needs to be implemented for every new pipeline element. In addition, the feature set of supported window operators differs (and often depends on the developer) as it is unclear which windows and parameters should/can be offered.
Therefore, adding support for explicit window semantics to the SDK/Core would make implementing data processors and sinks using windows much easier and less error-prone.
Tasks
- Design and introduce new processor and controller classes for windowed event processors (e.g., WindowedDataProcessor) which handle the windowing logic internally and only expose the higher-level methods to users (i.e onCurrentEvent, onExpiredEvent, etc...).
- Implement internal logic for few window functions (i.e TimeWindow, LengthWindow, TimeBatchWindow, LengthBatchWindow, etc...)
- Write a few sample pipeline-elements using your new API!
Relevant Skills
- Basic knowledge in StreamPipes core (cloning the repo, going through the codebase/documents would do).
- Basic knowledge of stream analytics window functions (this is not a must, but it's awesome if you know your way around analytics window functions).
- Some Java experience.
Learning Material
For StreamPipes:
- https://streampipes.apache.org/docs/
- https://streampipes.apache.org/media.html
- https://github.com/apache/incubator-streampipes
- https://github.com/apache/incubator-streampipes-extensions
For Streaming Analytics:
- https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-window-functions
- https://www.mikulskibartosz.name/difference-between-tumbling-and-sliding-window/
- https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/developing-storm-applications/content/understanding_sliding_and_tumbling_windows.html
For the context for the issue:
Mentor
- Grainier Perera (grainier [at] apache.org).
New Python Wrapper
More powerful real-time visualizations for StreamPipes
Apache StreamPipes
Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.
Background
Currently, the live dashboard (implemented in Angular) offers an initial set of simple visualizations, such as line charts, gauges, tables and single values. More advanced visualizations, especially those relevant for condition monitoring tasks (e.g., monitoring sensor measurements from industrial machines) is easy. Visualizations can be flexibly created by users and there is an SDK that allows to express requirements (e.g., based on data type or semantic type) for visualizations to better guide users through the creation process.
Tasks
- Extend the set of real-time visualizations in StreamPipes, e.g., by integrating existing visualizations from Apache ECharts.
- Improve the existing dashboard, e.g., by introducing better filtering or more advanced customization options.
Relevant Skills
Current wrappers such as standalone (JVM, Siddhi) or distributed (Flink) already allow to develop new processors in the given runtime environment. The idea is to extend the list of standalone runtime wrappers to also support pure Python processors. We already got a minimal working version that however is pretty inflexible and still relies on Java as a proxy to the pipeline management in the backend service for the model declaration in the setup phase ( capabilities, requirements, static properties of a processor) as well as the actual invocation in the execution phase ( receiving specific configuration from pipeline management when pipeline is started). This issue is to track the status of the development.
Tasks
- Add API endpoints as an interface for registration/invocation ( partly done)
- Port relevant model classes over to Python (declaration + invocation descriptions)
- Implement support for various transport protocols and transport formats
- Implement dev friendly alternative to Java builder pattern for model declaration
- Implement overall runtime logic for Python wrapper
Relevant Skills
0. Don't be afraid! We'0. Don't be afraid! We'll guide you through your first steps with StreamPipes.
- Angular
- Basic knowledge of Apache ECharts
Mentor
Dominik Riemer, PPMC Apache StreamPipes (riemer@apache.org)
Pulsar
- Excellent Python skills
- Excellent understanding of stream processing paradigm incl. message broker such as Kafka, MQTT, etc
- Good Understanding of RESTful web services (HTTP, etc.)
- Basic Java skills to understand existing wrapper logic
Info
- SIP-02 to collect design decisions https://cwiki.apache.org/confluence/display/STREAMPIPES/SIP-02+Python+wrapper
- Current python runtime wrapper implementation: https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper-python
- POC example: https://github.com/apache/incubator-streampipes-examples/tree/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/python
Mentor
Patrick Wiener, PPMC Apache StreamPipes (wiener@apache.org)
Pulsar
Expose the broker level message metadata to the client
Improve the message backlogs for the topic
In Pulsar, the client usually sends several messages with a batch. From the broker side, the broker receives a batch and write the batch message to the storage layer.
The message backlog is maintaining how many messages should be handled for a subscription. But unfortunately, the current backlog is based on the batches, not the messages. This will confuse users that they have pushed 1000 messages to the topic, but from the subscription side, when to check the backlog, will return a value that lower than 1000 messages such as 100 batches. Not able to get the message based backlog is it's so expensive to calculate the number of messages in each batch.
PIP-70 https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-raw-Message-metadata Introduced a broker level entry metadata which can and already support add message index for a topic(or message offset of a topic). This will provide the ability to calculate the number of messages between a message index to another message index. So we can leverage PIP-70 to improve the message backlog implementation to able to get the message-based backlog.
For the Exclusive subscription or Failover subscription, it easy to implement by calculating the messages between the mark delete position and the LAC position. But for the Shared and Key_Shared subscription, the individual acknowledgment will bring some complexity. We can cache the individual acknowledgment count in the broker memory, so the way to calculate the message backlog for the Shared and Key_Shared subscription is `backlogOfTheMarkdeletePosition` - `IndividualAckCount`
and broker add a timestamp for the message.
But currently, the client can't get the broker level message metadata since the broker skip this information when dispatching messages to the client. Provide a way to expose the broker level message metadata to the client.
Integration with Apache Ranger
Currently, Pulsar supports resetting the cursor according to time and message-id, e.g. you can reset the cursor to 3 hours ago or reset the cursor to a specific message-id. For some cases that users want to reset to the 10,000 earlier messages, Pulsar has not supported this operation yetPIP-70 only supports store authorization policies under local-zookeeper. Is it possible to support [ranger](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-raw-Message-metadata Introduced a broker level entry metadata which can support message index for a topic(or message offset of a topic), this will provide the ability to support reset cursor according to the message indexranger), apache ranger can provide a framework for central administration of security policies and monitoring of user access.
Throttle the ledger rollover for the broker
In Pulsar, the ledger rollover is split the data of a topic into multiple segments. For each ledger roll over operation, the metadata of the topic needs to be updated in the ZookKeeper. High ledger rollover frequency may lead to the ZookKeeper cluster in heavy workload. In order to make the ZookKeeper run more stable, we should limit the ledger rollover rate.
Expose the broker level message metadata to the client.
PIP-70 https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-raw-Message-metadata Introduced a broker level entry metadata and already support add message index and broker add a timestamp for the message.
But currently, the client can't get the broker level message metadata since the broker skip this information when dispatching messages to the client. Provide a way to expose the broker level message metadata to the client.
Integration with Apache Ranger
Currently, Pulsar only supports store authorization policies under local-zookeeper. Is it possible to support [ranger](https://github.com/apache/ranger), apache ranger can provide a framework for central administration of security policies and monitoring of user access.
Throttle the ledger rollover for the broker
In Pulsar, the ledger rollover is split the data of a topic into multiple segments. For each ledger roll over operation, the metadata of the topic needs to be updated in the ZookKeeper. High ledger rollover frequency may lead to the ZookKeeper cluster in heavy workload. In order to make the ZookKeeper run more stable, we should limit the ledger rollover rate.
Support publish and consume avro objects in pulsar-perf
We should use perf tool to benchmark producing and consuming messages using Schema.
SkyWalking
Improve the message backlogs for the topic
In Pulsar, the client usually sends several messages with a batch. From the broker side, the broker receives a batch and write the batch message to the storage layer.
The message backlog is maintaining how many messages should be handled for a subscription. But unfortunately, the current backlog is based on the batches, not the messages. This will confuse users that they have pushed 1000 messages to the topic, but from the subscription side, when to check the backlog, will return a value that lower than 1000 messages such as 100 batches. Not able to get the message based backlog is it's so expensive to calculate the number of messages in each batch.
PIP-70 https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-raw-Message-metadata Introduced a broker level entry metadata which can support message index for a topic(or message offset of a topic). This will provide the ability to calculate the number of messages between a message index to another message index. So we can leverage PIP-70 to improve the message backlog implementation to able to get the message-based backlog.
For the Exclusive subscription or Failover subscription, it easy to implement by calculating the messages between the mark delete position and the LAC position. But for the Shared and Key_Shared subscription, the individual acknowledgment will bring some complexity. We can cache the individual acknowledgment count in the broker memory, so the way to calculate the message backlog for the Shared and Key_Shared subscription is `backlogOfTheMarkdeletePosition` - `IndividualAckCount`
Support reset cursor by message index
Currently, Pulsar supports resetting the cursor according to time and message-id, e.g. you can reset the cursor to 3 hours ago or reset the cursor to a specific message-id. For some cases that users want to reset to the 10,000 earlier messages, Pulsar has not supported this operation yet
PIP-70 https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-raw-Message-metadata Introduced a broker level entry metadata which can support message index for a topic(or message offset of a topic), this will provide the ability to support reset cursor according to the message index.
Support publish and consume avro objects in pulsar-perf
We should use perf tool to benchmark producing and consuming messages using Schema.
Apache SkyWalking: Python agent supports profiling
Apache SkyWalking [1] is an application performance monitor (APM) tool for distributed systems, especially designed for microservices, cloud native and container-based (Docker, K8s, Mesos) architectures.
SkyWalking is based on agent to instrument (automatically) monitored services, for now, we have many agents for different languages, Python agent [2] is one of them, which supports automatic instrumentations.
The goal of this project is to extend the agent's features by supporting profiling [3] a function's invocation stack, help the users to analyze which method costs the most major time in a cross-services call.
To complete this task, you must be comfortable with Python, have some knowledge of tracing system, otherwise you'll have a hard time coming up to speed..
[1] http://skywalking.apache.org
[2] http://github.com/apache/skywalking-python
[3] https://thenewstack.io/apache-skywalking-use-profiling-to-fix-the-blind-spot-of-distributed-tracing/
SkyWalking
Apache SkyWalking: Python agent
collects and reports PVM metrics to backendsupports profiling
Apache SkyWalking SkyWalking [1] is an application performance monitor (APM) tool for distributed systems, especially designed for microservices, cloud native and container-based (Docker, K8s, Mesos) architectures.
Tracing distributed systems SkyWalking is based on agent to instrument (automatically) monitored services, for now, we have many agents for different languages, Python agent [2] is one of the main features of SkyWalking, with those traces, it can analyze some service metrics such as CPM, success rate, error rate, apdex, etc. SkyWalking also supports receiving metrics from the agent side directly.
In this task, we expect the Python agent to report its Python Virtual Machine (PVM) metrics, including (but not limited to, whatever metrics useful are also acceptable) CPU usage (%), memory used (MB), (active) thread/coroutine counts, garbage collection count, etc.
them, which supports automatic instrumentations.
The goal of this project is to extend the agent's features by supporting profiling [3] a function's invocation stack, help the users to analyze which method costs the most major time in a cross-services call.
To complete this task, you must be comfortable with Python, have some knowledge of tracing system, To complete this task, you must be comfortable with Python and gRPC, otherwise you'll have a hard time coming up to speed..
[1] Live demo to play around: http://122skywalking.112.182.72:8080 (under reconstruction, maybe unavailable but latest demo address can be found at the GitHub index page apache.org
[2] http://github.com/apache/skywalking)-python
[13]httphttps://skywalkingthenewstack.apache.orgio/apache-skywalking-use-profiling-to-fix-the-blind-spot-of-distributed-tracing/
ShardingSphere
Apache
ShardingSphere: Proofread the SQL definitions for ShardingSphere ParserSkyWalking: Python agent collects and reports PVM metrics to backend
Apache SkyWalking [1] is an application performance monitor (APM) tool for distributed systems, especially designed for microservices, cloud native and container-based (Docker, K8s, Mesos) architectures.
Tracing distributed systems is one of the main features of SkyWalking, with those traces, it can analyze some service metrics such as CPM, success rate, error rate, apdex, etc. SkyWalking also supports receiving metrics from the agent side directly.
In this task, we expect the Python agent to report its Python Virtual Machine (PVM) metrics, including (but not limited to, whatever metrics useful are also acceptable) CPU usage (%), memory used (MB), (active) thread/coroutine counts, garbage collection count, etc.
To complete this task, you must be comfortable with Python and gRPC, otherwise you'll have a hard time coming up to speed.
Live demo to play around: http://122.112.182.72:8080 (under reconstruction, maybe unavailable but latest demo address can be found at the GitHub index page http
Apache ShardingSphere
Apache ShardingSphere is a distributed database middleware ecosystem, including 2 independent products, ShardingSphere JDBC and ShardingSphere Proxy presently. They all provide functions of data sharding, distributed transaction, and database orchestration.
Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere
Background
ShardingSphere parser engine helps users parse a SQL to get the AST (Abstract Syntax Tree) and visit this tree to get SQLStatement (Java Object). At present, this parser engine can handle SQLs for `MySQL`, `PostgreSQL`, `SQLServer` and `Oracle`, which means we have to understand different database dialect SQLs.
More details: https://shardingsphere.apache.org/document/current/en/features/sharding/principle/parse/
Task
This issue is to proofread the DML(SELECT/UPDATE/DELETE/INSERT) SQL definitions for Oracle. As we have a basic Oracle SQL syntax definitions but do not keep in line with Oracle DOC, we need you to find out the vague SQL grammar definitions and correct them referring to Oracle DOC.
Notice, when you review these DML(SELECT/UPDATE/DELETE/INSERT) SQLs, you will find that these definitions will involve some basic elements of Oracle SQL. No doubt, these elements are included in this task as well.
Relevant Skills
1. Master JAVA language
2. Have a basic understanding of Antlr g4 file
3. Be familiar with Oracle SQLs
Targets files
1. DML SQLs g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-oracle/src/main/antlr4/imports/oracle/DMLStatement.g4
2. Basic elements g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-oracle/src/main/antlr4/imports/oracle/BaseRule.g4
References
1. Oracle SQL quick reference: https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlqr/SQL-Statements.html#GUID-1FA35EAD-AED2-4619-BFEE-348FF05D1F4A
2. Detailed Oracle SQL info: https://docs.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/19/sqlqr&id=SQLRF008
Mentor
Juan Pan, PMC of Apache ShardingSphere, panjuan@apache.org
TrafficControl
TrafficControl
GSOC: Varnish Cache support in Apache Traffic Control
Background
Apache Traffic Control is a Content Delivery Network (CDN) control plane for large scale content distribution.
Traffic Control currently requires Apache Traffic Server as the underlying cache. Help us expand the scope by integrating with the very popular Varnish Cache.
There are multiple aspects to this project:
- Configuration Generation: Write software to build Varnish configuration files (VCL). This code will be implemented in our Traffic
GSOC: Varnish Cache support in Apache Traffic Control
Background
Apache Traffic Control is a Content Delivery Network (CDN) control plane for large scale content distribution.
Traffic Control currently requires Apache Traffic Server as the underlying cache. Help us expand the scope by integrating with the very popular Varnish Cache.
There are multiple aspects to this project:
- Configuration Generation: Write software to build Varnish configuration files (VCL). This code will be implemented in our Traffic Ops and cache client side utilities, both written in Go.
- Health Monitoring: Implement monitoring of the Varnish cache health and performance. This code will run both in the Traffic Monitor component and within Varnish. Traffic Monitor is written in Go and Varnish is written in C.
- Testing: Adding automated tests for new code
Skills:
- Proficiency in Go is required
- A basic knowledge of HTTP and caching is preferred, but not required for this project.
Cassandra
Allow table property defaults (e.g. compaction, compression) to be specified for a cluster/keyspace
During an IRC discussion in cassandra-dev it was proposed that we could have table property defaults stored on a Keyspace or globally within the cluster. For example, this would allow users to specify "All new tables on this cluster should default to LCS with SSTable size of 320MiB" or "all new tables in Keyspace XYZ should have Zstd commpression with a 8 KiB block size" or "default_time_to_live should default to 3 days" etc ... This way operators can choose the default that makes sense for their organization once (e.g. LCS if they are running on fast SSDs), rather than requiring developers creating the Keyspaces/Tables to make the decision on every creation (often without context of which choices are right).
A few implementation options were discussed including:
- A YAML option
- Schema provided at the Keyspace level that would be inherited by any tables automatically
- Schema provided at the Cluster level that would be inherited by any Keyspaces or Tables automatically
In IRC it appears that rough consensus was found in having global -> keyspace -> table defaults which would be stored in schema (no YAML configuration since this isn't node level really, it's a cluster level config).
Provide easy copypasta config formatting for nodetool get commands
Allow all nodetool commands which print out the state of the node or cluster to do so in a way that makes it easy to re-use or paste on other nodes or config files.
For example, the command getcompactionthroughput formats its output like this:
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput
Current compaction throughput: 64 MB/s
But with an --as-yaml option, it could do this instead:
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-yaml
compaction_throughput_mb_per_sec: 64
and with an --as-cli option, it could do this:
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-cli
./nodetool setcompactionthroughput 64
Any other nodetool standard options should simply be carried along to the --as-cli form, with the exception of -pw.
Any -pw options should be elided with a warning in comments, but -pwf options should be allowed. This would allow users using -pw to append a password at their discretion, but would allow -pwf to work as usual.
In the absence of either of the options above (--as-yaml or --as-cli) the formatting should not be changed to avoid breaking extant tool integrations.
ShardingSphere
Apache ShardingSphere: Proofread the DML SQL definitions for ShardingSphere Parser
Apache ShardingSphere
Apache ShardingSphere is a distributed database middleware ecosystem, including 2 independent products, ShardingSphere JDBC and ShardingSphere Proxy presently. They all provide functions of data sharding, distributed transaction, and database orchestration.
Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere
Background
ShardingSphere parser engine helps users parse a SQL to get the AST (Abstract Syntax Tree) and visit this tree to get SQLStatement (Java Object). At present, this parser engine can handle SQLs for `MySQL`, `PostgreSQL`, `SQLServer` and `Oracle`, which means we have to understand different database dialect SQLs.
More details: https://shardingsphere.apache.org/document/current/en/features/sharding/principle/parse/
Task
This issue is to proofread the DML(SELECT/UPDATE/DELETE/INSERT) SQL definitions for Oracle. As we have a basic Oracle SQL syntax definitions but do not keep in line with Oracle DOC, we need you to find out the vague SQL grammar definitions and correct them referring to Oracle DOC.
Notice, when you review these DML(SELECT/UPDATE/DELETE/INSERT) SQLs, you will find that these definitions will involve some basic elements of Oracle SQL. No doubt, these elements are included in this task as well.
Relevant Skills
1. Master JAVA language
2. Have a basic understanding of Antlr g4 file
3. Be familiar with Oracle SQLs
Targets files
1. DML SQLs g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-oracle/src/main/antlr4/imports/oracle/DMLStatement.g4
2. Basic elements g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-oracle/src/main/antlr4/imports/oracle/BaseRule.g4
References
1. Oracle SQL quick reference: https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlqr/SQL-Statements.html#GUID-1FA35EAD-AED2-4619-BFEE-348FF05D1F4A
2. Detailed Oracle SQL info: https://docs.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/19/sqlqr&id=SQLRF008
Mentor
Juan Pan, PMC of Apache ShardingSphere, panjuan@apache.org
Apache ShardingSphere: Proofread the DDL/TCL SQL definitions for ShardingSphere Parser
Apache ShardingSphere
Apache ShardingSphere is a distributed database middleware ecosystem, including 2 independent products, ShardingSphere JDBC and ShardingSphere Proxy presently. They all provide functions of data sharding, distributed transaction, and database orchestration.
Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere
Background
ShardingSphere parser engine helps users parse a SQL to get the AST (Abstract Syntax Tree) and visit this tree to get SQLStatement (Java Object). At present, this parser engine can handle SQLs for `MySQL`, `PostgreSQL`, `SQLServer` and `Oracle`, which means we have to understand different database dialect SQLs.
More details: https://shardingsphere.apache.org/document/current/en/features/sharding/principle/parse/
Task
This issue is to proofread the following definitions,
- All the DDL SQL definitions for Oracle except for ALTER, DROP, CREATE and TRUNCATE.
- All the TCL (Transaction Control Language) SQL definitions for Oracle
You can learn more here.
As we have a basic Oracle SQL syntax definitions but do not keep in line with Oracle DOC, we need you to find out the vague SQL grammar definitions and correct them referring to Oracle DOC.
Notice, when you review these target SQLs above, you will find that these definitions will involve some basic elements of Oracle SQL. No doubt, these elements are included in this task as well.
Relevant Skills
1. Master JAVA language
2. Have a basic understanding of Antlr g4 file
3. Be familiar with Oracle SQLs
Targets files
1. DDL SQLs g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-oracle/src/main/antlr4/imports/oracle/DDLStatement.g4
2. TCL SQLs g4 file:
https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-oracle/src/main/antlr4/imports/oracle/TCLStatement.g4
3. Basic elements g4 file: https://github.com/apache/shardingsphere/blob/master/shardingsphere-sql-parser/shardingsphere-sql-parser-dialect/shardingsphere-sql-parser-oracle/src/main/antlr4/imports/oracle/BaseRule.g4
References
1. Oracle SQL quick reference: https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlqr/SQL-Statements.html#GUID-1FA35EAD-AED2-4619-BFEE-348FF05D1F4A
2. Detailed Oracle SQL info: https://docs.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/19/sqlqr&id=SQLRF008
Mentor
Juan Pan, PMC of Apache ShardingSphere, panjuan@apache.org
Cassandra
Provide easy copypasta config formatting for nodetool get commands
Allow all nodetool commands which print out the state of the node or cluster to do so in a way that makes it easy to re-use or paste on other nodes or config files.
For example, the command getcompactionthroughput formats its output like this:
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput
Current compaction throughput: 64 MB/s
But with an --as-yaml option, it could do this instead:
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-yaml
compaction_throughput_mb_per_sec: 64
and with an --as-cli option, it could do this:
[jshook@cass4 bin]$ ./nodetool getcompactionthroughput --as-cli
./nodetool setcompactionthroughput 64
Any other nodetool standard options should simply be carried along to the --as-cli form, with the exception of -pw.
Any -pw options should be elided with a warning in comments, but -pwf options should be allowed. This would allow users using -pw to append a password at their discretion, but would allow -pwf to work as usual.
In the absence of either of the options above (--as-yaml or --as-cli) the formatting should not be changed to avoid breaking extant tool integrations.
Improve visibility into repair state
Most of the repair flow is fire and forget, where we send a message, or start a job with a remote component, and then wait for a response of some type, or for the failure detector to tell us a node has died. This leaves several cases where the repair can hang, and operators have to guess about it’s state, and the best course of action. It would help if the state of a given repair could be polled, and possibly cancelled. This is going to involve touching the validation, anti-compaction, and streaming code.
Per-node overrides for table settings
There is a few cases where it's convenient to set some table parameters on only one of a few nodes. For instance, it's useful for experimenting with settings like caching options, compaction, compression, read repair chance, gcGrace ... Another case is when you want to completely migrate to a new setting, but want to do that node-per-node (mainly useful when switching compaction strategy, see CASSANDRA-10898).
I'll note that we can already do some of this through JMX for some of the settings as we have methods like ColumnFamilyStoreMBean.setCompactionParameters(), but:
- parameters settings are initially set in CQL. Having to go to JMX for this sounds less consistent to me. The fact we have both a ColumnFamilyStoreMBean.setCompactionParameters() and a ColumnFamilyStoreMBean.setCompactionParametersJson() (as I assume the former one is inconvenient to use) is also proof to me than JMX ain't terribly appropriate.
- I think this can be potentially useful for almost all table settings, but we don't expose JMX methods for all settings, and it would be annoying to have to. The method suggested below wouldn't have to be updated every time we add a new settings (if done right).
- Changing options through JMX is not persistent across restarts. This may arguably be fine in some cases, but if you're trying to migrate your compaction strategy node per node, or want to experiment with a setting over a mediumish time period, it's mostly a pain.
So what I suggest would be add node overrides in the normal table setting (which would be part of the schema as any other setting). In other words, if you want to set LCS for only one specific node, you'd do:
ALTER TABLE foo WITH node_overrides = { '192.168.0.1' : { 'compaction' : { 'class' : 'LeveledCompactionStrategy' } }
}
I'll note that I already suggested that idea on CASSANDRA-10898, but as it's more generic than what that latter ticket is about, so creating its own ticket.
Expose application_name and application_version in virtual table system_views.clients
Recent java-driver's com.datastax.oss.driver.api.core.session.SessionBuilder respects properties ApplicationName and ApplicationVersion.
It would be helpful to exposed this information via virtual table system_views.clients and with nodetool clientstats.
Prevent and fail-fast any attempts to incremental repair cdc/mv tables
Running incremental repairs on CDC or MV tables breaks them.
Attempting to run incremental repair on such should fail-fast and be prevented, with a clear error message.
Global configuration parameter to reject repairs with anti-compaction
We have moved from Cassandra 2.1 to 3.0 and from an operational aspect, the Cassandra repair area changed significantly / got more complex. Beside incremental repairs not working reliably, also full repairs (-full command-line option) are running into anti-compaction code paths, splitting repaired / non-repaired data into separate SSTables, even with full repairs.
Casandra 4.x (with repair enhancements) is quite away for us (for production usage), thus we want to avoid anti-compactions with Cassandra 3.x at any cost. Especially for our on-premise installations at our customer sites, with less control over on how e.g. nodetool is used, we simply want to have a configuration parameter in e.g. cassandra.yaml, which we could use to reject any repair invocations that results in anti-compaction being active.
I know, such a flag still can be flipped then (by the customer), but as a first safety stage possibly sufficient enough to reject anti-compaction repairs, e.g. if someone executes nodetool repair ... the wrong way (accidentally).
Cleanup key ranges during compaction
Currently cleanup is considered an optional, manual operation that users are told to run to free disk space after a node was affected by topology changes. However, unmanaged key ranges could also end up on a node through other ways, e.g. manual added sstable files by an admin.
I'm also not sure unmanaged data is really that harmless and cleanup should really be optional, if you don't need to reclaim the disk space. When it comes to repairs, users are expected to purge a node after downtime in case it was not fully covered by a repair within gc_grace afterwards, in order to avoid re-introducing deleted data. But the same could happen with unmanaged data, e.g. after topology changes activate unmanaged ranges again or after restoring backups.
I'd therefor suggest to avoid rewriting key ranges no longer belonging to a node and older than gc_grace during compactions.
Maybe we could also introduce another CLEANUP_COMPACTION operation to find candidates based on SSTable.first/last in case we don't have pending regular or tombstone compactions.
Improve visibility into repair state
Most of the repair flow is fire and forget, where we send a message, or start a job with a remote component, and then wait for a response of some type, or for the failure detector to tell us a node has died. This leaves several cases where the repair can hang, and operators have to guess about it’s state, and the best course of action. It would help if the state of a given repair could be polled, and possibly cancelled. This is going to involve touching the validation, anti-compaction, and streaming code.
Per-node overrides for table settings
There is a few cases where it's convenient to set some table parameters on only one of a few nodes. For instance, it's useful for experimenting with settings like caching options, compaction, compression, read repair chance, gcGrace ... Another case is when you want to completely migrate to a new setting, but want to do that node-per-node (mainly useful when switching compaction strategy, see CASSANDRA-10898).
I'll note that we can already do some of this through JMX for some of the settings as we have methods like ColumnFamilyStoreMBean.setCompactionParameters(), but:
- parameters settings are initially set in CQL. Having to go to JMX for this sounds less consistent to me. The fact we have both a ColumnFamilyStoreMBean.setCompactionParameters() and a ColumnFamilyStoreMBean.setCompactionParametersJson() (as I assume the former one is inconvenient to use) is also proof to me than JMX ain't terribly appropriate.
- I think this can be potentially useful for almost all table settings, but we don't expose JMX methods for all settings, and it would be annoying to have to. The method suggested below wouldn't have to be updated every time we add a new settings (if done right).
- Changing options through JMX is not persistent across restarts. This may arguably be fine in some cases, but if you're trying to migrate your compaction strategy node per node, or want to experiment with a setting over a mediumish time period, it's mostly a pain.
So what I suggest would be add node overrides in the normal table setting (which would be part of the schema as any other setting). In other words, if you want to set LCS for only one specific node, you'd do:
ALTER TABLE foo WITH node_overrides = { '192.168.0.1' : { 'compaction' : { 'class' : 'LeveledCompactionStrategy' } }
}
I'll note that I already suggested that idea on CASSANDRA-10898, but as it's more generic than what that latter ticket is about, so creating its own ticket.
Add ability to disable schema changes, repairs, bootstraps, etc (during upgrades)
There are a lot of operations that aren't supposed to be run in a mixed version cluster: schema changes, repairs, topology changes, etc. However, it's easily possible for these operations to be accidentally run by a script, another user unaware of the upgrade, or an operator that's not aware of these rules.
We should make it easy to follow the rules by making it possible to prevent/disable all of these operations through nodetool commands. At the start of an upgrade, an operator can disable all of these until the upgrade has been completed.
Migrate use of maven-ant-tasks to resolver-ant-tasks
Cassandra resolves dependencies and generates maven pom files through the use of maven-ant-tasks. This is no longer a supported project.
The recommended upgrade is to resolver-ant-tasks. It follows similar APIs so shouldn't be too impactful a change.
The existing maven-ant-tasks has caused some headaches already with internal super poms referencing insecure http:// central maven repository URLs that are no longer supported.
We should also take the opportunity to
- define the "test" scope (classpath) for those dependencies only used for tests (currently we are packaging test dependencies into the release binary artefact),
- remove the jar files stored in the git repo under the "lib/" folder.
These two above points have to happen in tandem, as the jar files under lib/ are those that get bundled into the build/dist/lib/ and hence the binary artefact. That is, all jar files under lib/ are the project's "compile" scope, and all other dependencies defined in build.xml are either "provided" or "test" scope. These different scopes for dependencies are currently configured in different maven-ant-tasks poms. See https://github.com/apache/cassandra/commit/d43b9ce5092f8879a1a66afebab74d86e9e127fb#r45659668
Expose application_name and application_version in virtual table system_views.clients
Recent java-driver's com.datastax.oss.driver.api.core.session.SessionBuilder respects properties ApplicationName and ApplicationVersion.
It would be helpful to exposed this information via virtual table system_views.clients and with nodetool clientstats.
Add Plugin Support for CQLSH
Currently the Cassandra drivers offer a plugin authenticator architecture for the support of different authentication methods. This has been leveraged to provide support for LDAP, Kerberos, and Sigv4 authentication. Unfortunately, cqlsh, the included CLI tool, does not offer such support. Switching to a new enhanced authentication scheme thus means being cut off from using cqlsh in normal operation.
We should have a means of using the same plugins and authentication providers as the Python Cassandra driver.
Here's a link to an initial draft of CEP.
Add ability to ttl snapshots
It should be possible to add a TTL to snapshots, after which it automatically cleans itself up.
This will be useful together with the auto_snapshot option, where you want to keep an emergency snapshot in case of accidental drop or truncation but automatically remove it after a specified period when it's no longer useful. So in addition to allowing a user to specify a snapshot ttl on nodetool snapshot we should have a auto_snapshot_ttl option that allows a user to set a ttl for automatic snaphots on drop/truncate.
Allow table property defaults (e.g. compaction, compression) to be specified for a cluster/keyspace
During an IRC discussion in cassandra-dev it was proposed that we could have table property defaults stored on a Keyspace or globally within the cluster. For example, this would allow users to specify "All new tables on this cluster should default to LCS with SSTable size of 320MiB" or "all new tables in Keyspace XYZ should have Zstd commpression with a 8 KiB block size" or "default_time_to_live should default to 3 days" etc ... This way operators can choose the default that makes sense for their organization once (e.g. LCS if they are running on fast SSDs), rather than requiring developers creating the Keyspaces/Tables to make the decision on every creation (often without context of which choices are right).
A few implementation options were discussed including:
- A YAML option
- Schema provided at the Keyspace level that would be inherited by any tables automatically
- Schema provided at the Cluster level that would be inherited by any Keyspaces or Tables automatically
In IRC it appears that rough consensus was found in having global -> keyspace -> table defaults which would be stored in schema (no YAML configuration since this isn't node level really, it's a cluster level config).
Prevent and fail-fast any attempts to incremental repair cdc/mv tables
Running incremental repairs on CDC or MV tables breaks them.
Attempting to run incremental repair on such should fail-fast and be prevented, with a clear error message.
Add ability to disable schema changes, repairs, bootstraps, etc (during upgrades)
There are a lot of operations that aren't supposed to be run in a mixed version cluster: schema changes, repairs, topology changes, etc. However, it's easily possible for these operations to be accidentally run by a script, another user unaware of the upgrade, or an operator that's not aware of these rules.
We should make it easy to follow the rules by making it possible to prevent/disable all of these operations through nodetool commands. At the start of an upgrade, an operator can disable all of these until the upgrade has been completed
Global configuration parameter to reject repairs with anti-compaction
We have moved from Cassandra 2.1 to 3.0 and from an operational aspect, the Cassandra repair area changed significantly / got more complex. Beside incremental repairs not working reliably, also full repairs (-full command-line option) are running into anti-compaction code paths, splitting repaired / non-repaired data into separate SSTables, even with full repairs.
Casandra 4.x (with repair enhancements) is quite away for us (for production usage), thus we want to avoid anti-compactions with Cassandra 3.x at any cost. Especially for our on-premise installations at our customer sites, with less control over on how e.g. nodetool is used, we simply want to have a configuration parameter in e.g. cassandra.yaml, which we could use to reject any repair invocations that results in anti-compaction being active.
I know, such a flag still can be flipped then (by the customer), but as a first safety stage possibly sufficient enough to reject anti-compaction repairs, e.g. if someone executes nodetool repair ... the wrong way (accidentally).
Script to autogenerate cassandra.yaml
It would be useful to have a script that can ask the user a few questions and generate a recommended cassandra.yaml based on their answers. This will help solve issues like selecting num_tokens. It can also be integrated into OS specific packaging tools such as debconf[1]. Rather than just documenting on the website, it is best to provide a simple script to auto-generate configuration based on common use-cases.
Cleanup key ranges during compaction
Currently cleanup is considered an optional, manual operation that users are told to run to free disk space after a node was affected by topology changes. However, unmanaged key ranges could also end up on a node through other ways, e.g. manual added sstable files by an admin.
I'm also not sure unmanaged data is really that harmless and cleanup should really be optional, if you don't need to reclaim the disk space. When it comes to repairs, users are expected to purge a node after downtime in case it was not fully covered by a repair within gc_grace afterwards, in order to avoid re-introducing deleted data. But the same could happen with unmanaged data, e.g. after topology changes activate unmanaged ranges again or after restoring backups.
I'd therefor suggest to avoid rewriting key ranges no longer belonging to a node and older than gc_grace during compactions.
Maybe we could also introduce another CLEANUP_COMPACTION operation to find candidates based on SSTable.first/last in case we don't have pending regular or tombstone compactions.
Migrate use of maven-ant-tasks to resolver-ant-tasks
Cassandra resolves dependencies and generates maven pom files through the use of maven-ant-tasks. This is no longer a supported project.
The recommended upgrade is to resolver-ant-tasks. It follows similar APIs so shouldn't be too impactful a change.
The existing maven-ant-tasks has caused some headaches already with internal super poms referencing insecure http:// central maven repository URLs that are no longer supported.
We should also take the opportunity to
- define the "test" scope (classpath) for those dependencies only used for tests (currently we are packaging test dependencies into the release binary artefact),
- remove the jar files stored in the git repo under the "lib/" folder.
These two above points have to happen in tandem, as the jar files under lib/ are those that get bundled into the build/dist/lib/ and hence the binary artefact. That is, all jar files under lib/ are the project's "compile" scope, and all other dependencies defined in build.xml are either "provided" or "test" scope. These different scopes for dependencies are currently configured in different maven-ant-tasks poms. See https://github.com/apache/cassandra/commit/d43b9ce5092f8879a1a66afebab74d86e9e127fb#r45659668
Add ability to ttl snapshots
It should be possible to add a TTL to snapshots, after which it automatically cleans itself up.
This will be useful together with the auto_snapshot option, where you want to keep an emergency snapshot in case of accidental drop or truncation but automatically remove it after a specified period when it's no longer useful. So in addition to allowing a user to specify a snapshot ttl on nodetool snapshot we should have a auto_snapshot_ttl option that allows a user to set a ttl for automatic snaphots on drop/truncate.
...
Apache APISIX: support to fetch more useful information of client request
What's Apache APISIX?
Apache APISIX is a dynamic, real-time, high-performance API gateway, based on the Nginx library and etcd.
APISIX provides rich traffic management features such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more.
You can use Apache APISIX to handle traditional north-south traffic, as well as east-west traffic between services. It can also be used as a k8s ingress controller.
Background (route matching and run plugins)
When the client completes a request, there is a lot of useful information inside Apache APISIX.
Task
Needs a way to show it. It is convenient for callers to troubleshoot problems and understand the workflow of Apache APISIX.
The first version target can display:
1. Which route is matched.
2. Which plugins are loaded.
In subsequent versions, we will add more information that the caller cares about, such as:
- Whether the global plugin is executed
- Time consumption statistics
- The return value when the plugin is executed
Relevant Skills
1. Master Lua language
2. Have a basic understanding of API Gateway or Web server
Apache APISIX: support apply certificates from Let’s Encrypt or any other ACMEv2 service
Apache APISIX
Apache APISIX is a dynamic, real-time, high-performance API gateway, based on the Nginx library and etcd.
APISIX provides rich traffic management features such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more.
You can use Apache APISIX to handle traditional north-south traffic, as well as east-west traffic between services. It can also be used as a k8s ingress controller.
Background
The data plane of Apache APISIX supports dynamic loading of ssl certificates, but the control plane does not have the support of ACME.
Although users can use other tools to obtain ACME certificates, and then call the admin API to write them into Apache APISIX, this is not convenient for many users.
Task
In the Apache APISIX dashboard (https://github.com/apache/apisix-dashboard) project, add support for ACME, which can automatically obtain and update certificates
Relevant Skills
TypeScript
Golang
familiar with Apache APISIX's admin API
Mentor
Ming Wen, PMC of Apache APISIX, wenming@apache.org
Apache APISIX: Enhanced verification for APISIX ingress controller
Apache APISIX
Apache APISIX is a dynamic, real-time, high-performance API gateway, based on the Nginx library and etcd.
APISIX provides rich traffic management features such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more.
You can use Apache APISIX to handle traditional north-south traffic, as well as east-west traffic between services. It can also be used as a k8s ingress controller.
Background
We can use APISIX as kubernetes ingress.Use CRD (Custom Resource Definition) on kubernetes to define APISIX objects, such as route/service/upstream/plugins.
We have done basic structural verification of CRD, but we still need more verification. For example, plug-in schema verification, dependency verification between APISIX objects, rule conflict verification, etc. All these verifications need to be completed before CRD is applied.
Task
1. Implement a validating admission webhook.
2. Support plugins schema verification.
3. Support object dependency verification.
Relevant Skills
1. Golang
2. Be familiar with Apache APISIX's admin API
3. Be familiar with kubernetes
Mentor
Wei Jin, PMC of Apache APISIX, kvn@apache.org
Apache APISIX: Support Nacos in a native way
.org
Apache APISIX: improve the website
Apache APISIX
Apache APISIX is a dynamic, real-time, high-performance API gateway, based on the Nginx library and etcd, and we have a standalone website to let more people know about the Apache APISIX.
Background
The website of Apache APISIX is used for showing people what's Apache APISIX is, and it will include up to date docs to let developers searching guides more easily, and so on.
Task
In the website[1] and its repo[2], we are going to refactor the homepage, improve those docs which include apisix's docs and some like release guide.
Relevant Skills
TypeScript
React.js
Mentor
Zhiyuan, PMC of Apache APISIX, juzhiyuan@apache.org
[1] https://apisix.apache.org/
[2]
Apache APISIX is a dynamic, real-time, high-performance cloud-native API gateway, based on the Nginx library and etcd.
Page: https://apisix.apache.org
Github: https://github.com/apache/apisix
Background
To get the upstream information dynamically, APISIX need to be integrated with other service discovery systems. Currently we already support Eureka, and many people hope we can support Nacos too.
Nacos is a widely adopted service discovery system: https://nacos.io/en-us/index.html
Previously we try to support Nacos via DNS. Nacos provides a CoreDNS plugin to expose the information via DNS: https://github.com/nacos-group/nacos-coredns-plugin
However, this plugin seems to be unmaintained.
Therefore, it would be good if we can support Nacos natively via its API, which is expected to be maintained.
Task
Integrate Nacos with APISIX via Nacos's HTTP API.
Relevant Skills
1. Master Lua language and HTTP protocol
2. Have a basic understanding of APISIX / Nacos
Targets files
1. https://github.com/apache/apisix-website
Apache APISIX: Support Nacos in a native way
Apache APISIX is a dynamic, real-time, high-performance cloud-native API gateway, based on the Nginx library and etcd.
Page: https://apisix.apache.org
Github
References
1. Nacos Open API: https://nacosgithub.iocom/en-us/docs/open-api.html
Mentor
Zexuan Luo, committer of Apache APISIX
Background
To get the upstream information dynamically, APISIX need to be integrated with other service discovery systems. Currently we already support Eureka, and many people hope we can support Nacos too.
Nacos is a widely adopted service discovery system: https://nacos.io/en-us/index.html
Previously we try to support Nacos via DNS. Nacos provides a CoreDNS plugin to expose the information via DNS: https://github.com/nacos-group/nacos-coredns-plugin
However, this plugin seems to be unmaintained.
Therefore, it would be good if we can support Nacos natively via its API, which is expected to be maintained.
Task
Integrate Nacos with APISIX via Nacos's HTTP API.
Relevant Skills
1. Master Lua language and HTTP protocol
2. Have a basic understanding of APISIX / Nacos
Targets files
1. https://github.com/apache/apisix/tree/master/apisix/discovery
References
1. Nacos Open API: https://nacos.io/en-us/docs/open-api.html
Mentor
Zexuan Luo, committer of Apache APISIX
Apache APISIX: improve the website
Apache APISIX
Apache APISIX is a dynamic, real-time, high-performance API gateway, based on the Nginx library and etcd, and we have a standalone website to let more people know about the Apache APISIX.
Background
The website of Apache APISIX is used for showing people what's Apache APISIX is, and it will include up to date docs to let developers searching guides more easily, and so on.
Task
In the website[1] and its repo[2], we are going to refactor the homepage, improve those docs which include apisix's docs and some like release guide.
Relevant Skills
TypeScript
React.js
Mentor
Zhiyuan, PMC of Apache APISIX, juzhiyuan@apache.org
[1] https://apisix.apache.org/
[2]https://github.com/apache/apisix-website
Apache APISIX Dashboard: Enhancement plugin orchestration
The Apache APISIX Dashboard is designed to make it as easy as possible for users to operate Apache APISIX through a frontend interface.
The Dashboard is the control plane and performs all parameter checks; Apache APISIX mixes data and control planes and will evolve to a pure data plane.
This project includes manager-api, which will gradually replace admin-api in Apache APISIX.
Background
The plugin orchestration feature allows users to define the order of plugins to meet their scenarios. At present, we have implemented the plugin scheduling feature, but there are still many points to be optimized.
Task
- develop a new plugin, conditional judgment card style. Currently, both the conditional judgment card and the plug-in card are square shaped, which makes it difficult for users to distinguish them, so in this task the conditional judgment card needs to be changed to a diamond shape. As shown in the figure below.
2. Add arrows for connecting lines.The connection lines in the current plugin orchestration are not directional, we need to add arrows to the connection lines as shown in the figure below.
3. Limit plugin orchestration operations. We need to restrict the connection of each card to ensure the proper use of the plugin orchestration, and the situation shown in the arrow below is not allowed.
Relevant Skills
1. Basic use of HTML, CSS, and JavaScript.
2. Basic use of React Framework.
Mentor
Yi Sun, committer of Apache APISIX,sunyi@apache.org