Background

The default way of registering / downloading templates in CloudStack involves caching them on the secondary store and then during VM deployment, the template is copied to the primary store. However, from ACS version 4.11.1 onward, a feature was added for KVM hypervisor to enable direct download to primary store. This massively reduces the usage of secondary store and also quickens the entire VM deployment process, as there is no need to copy the template from secondary to primary store. 

Requirement

We would like to propose an idea to extend this feature of direct download of templates onto primary store for other hypervisors - namely, VMware and XenServer. This would gravely benefit end-users to efficiently use the secondary storage and save overall time of VM deployment on the respective hypervisors

Relevant Skills:

Java
MySQL
Vue.js

Difficulty:

175 hours (Only VMware)
350 hours (VMware & XenServer)

Potential Mentors:

Abhishek Kumar (abhishek.mrt22@gmail.com)
Pearl Dsilva (pearl1594@gmail.com)

Shutting down / Restarting Cloudstack is a necessary step in upgrades, system maintenance, etc. As of now, there is no way to safely shutdown or restart CloudStack. It is directly terminated via systemd. Since this is the case, any asyncronous job or background task is abrubptly terminated and can fail. As of now, CloudStack maintains a list of asynchronous jobs wihtin it's database along with their status.

This idea aims to provide a way to safely shutdown CloudStack. It involves two parts :

• Prevent new asynchronous jobs from being added to CloudStack when a safe shutdown is triggered
• Check the status of the async jobs and Shut down CloudStack when all the jobs have been completed

Provide the ability to safely shutdown CloudStack

Add API (and/or UI) support to :

• Trigger a safe shutdown
• (Optional) Support restarts
• (Optional) Support a forced shutdown when CloudStack will quit even if there are async jobs running

Duration

• Some Experience : 175 hours
• Newbie : 350 hours

Potential Mentors

References

https://

www.shapeblue.com/how-to-deploy-templates-without-using-secondary-storage-on-kvm/
https://www.shapeblue.com/cloudstack-feature-first-look-direct-download-agnostic-of-the-storage-provider/
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Bypass+Secondary+Storage+%28Direct+Download%29+on+KVM
https://www.youtube.com/watch?v=SwepUTfGiKc

Background

As of now the CloudStack Terraform Provider does not support managing CKS clusters

This proposal aims to add support to the CloudStack Terraform Provider to manage CKS clusters

This would involve supporting the following actions on CKS clusters :

• Create
• Stop / Start
• Scale
• Upgrade
• Delete

[Optional]
Support the following actions on the binary ISOs :

• Register
• Enable / Disable
• Delete

Duration

• 175 hours

Potential Mentors

• Harikrishna Patnala
• David Jumani

References

Terraform is an Infrastructure as Code (IaC) software that provides a consistent CLI workflow to manage resources in
many cloud services. Cloudstack Terraform provider integrates with Cloudstack to aid in managing and automating the deployment of resources in cloudstack. We have recently made the first release of CloudStack Terraform Provider v0.4.0 

https://github.com/apache/cloudstack

-terraform-provider

Requirement

Terraform defines a datasource as, "something that allows Terraform to use the information defined outside of Terraform, defined by another separate Terraform configuration, or modified by functions". Most resources offer data sources alongside their set of resource types. However, currently Cloudstack Terraform Provider only has one datasource for template. Hence, we propose an idea for students to get involved in enhancing the features of the Cloudstack Terraform Provider by adding support for datasources.

Image Added

If the students are enjoying the project, the scope can be extended to support adding further resources in Terraform such that the CloudStack Terraform Provider may become a de-facto tool for automating CloudStack deployments.

The current set of resources Cloudstack terraform provider supports are:
https://registry.terraform.io/providers/cloudstack/cloudstack/latest/docs , where as its counterpart Ansible boasts of a more evolved list of resources  https://docs.ansible.com/ansible/latest/collections/ngine_io/cloudstack/index.html  mainly zones, clusters, accounts, domains, etc. It would be great if students interested want to go a step ahead and help add support for these too.

Relevant Skills:

GoLang (basic)

Difficulty:

Medium

Potential Mentors:

Harikrishna Patnala
Pearl Dsilva

Example and references

https://registry.terraform.io/providers/cloudstack/cloudstack/latest/docs : check Resources and Data Sources section under CloudStack Provider
Depends on CloudStack Go SDK - https://github.com/apache/cloudstack-go

Github issue: https://github.com/apache/cloudstack/issues/6016

Background

Currently, Deploy Instances/Virtual Machines(VMs) in Cloudstack requires to specify some offerings, template and other settings through the API (check the API here: https://cloudstack.apache.org/api/apidocs-4.16/apis/deployVirtualMachine.html) or the 'Instance Deployment Wizard' in the UI.

Requirement

Provision to user/operator to quick deploy an instance using a VM definition/profile. The VM definition/profile would hold the details of the template, offerings (including any custom values - size, iops), ssh keypair, instance group, affinity group and other settings (boot type, dynamic scaling, userdata, keyboard language, etc) that are required, and the underlying definition/profile id can be used to launch an instance. At the minimum, the definition should hold all the mandatory details for deploying an instance. With this, only the VM definitions/profiles (and other important options, with the associated billing details) can be exposed to the users for VM deployment, instead of the offerings and other VM options.

Need to add new APIs (and/or UI) support for the VM definition/profile CRUD operations, and support for definition in the deployVirtualMachine API.

Relevant Skills

• Java, MySQL
• Vue.js (for UI)
• Some knowledge of Virtualization and CloudStack

Difficulty

Medium

Potential Mentors

• Suresh Kumar Anaparti
• David Jumani

Project Scope/Duration

Medium / 175 hours

References

Currently, defining an affinity or anti-affinity rule works only at hosts level. I would like to have more detail on the affinity group, extending it at different levels (cluster, pod, zone,..) and also within the same level, being able to add or remove resources from the group.

For hosts and storage pools, administrators can make use of host tags or storage tags to get a similar result. However, the extension of affinity/anti-affinity groups would make the administration easier.

Size of the project: Medium (~175hs)

Background

Over recent years, Edge computing has been gaining popularity as it defines a model that brings compute and storage closer to
where they are consumed by the end-user. By being closer to the end-user a better experience can be provided with reduction on overall latency, lower bandwidth requirements, lower TCO, more flexible hardware/software model, while also ensuring security and reliability. To align ACS with this evolving cloud computing model we would like to propose an idea of supporting Edge Zones in CloudStack, which
can be also looked upon as a lightweight zone, with minimal resources.

Requirement

Today, when a Zone is setup in CloudStack, it by default comes up with a secondary storage VM(SSVM) and a console proxy VM(CPVM). As part of this project, we would need to define a new zone type to decide the change in workflow required to ensure that a CPVM & SSVM isn't spawned up by default. Basic characteristics of an Edge zone include:

• no need for Secondary Storage
• no Secondary Storage VM
• no Console Proxy VM
• Local storage only as typically an edge device comprises of a single compute node (host)
• And supports L2 and Isolated networks.

A high-level view of an edge zone would look something like:

Image Added

Relevant Skills:

Java
MySQL
Vue.js (Basic)

Difficulty:

Medium

Project Duration:

175 hours

Potential Mentors:

Alex Mattioli
Nicolas Vazquez
Pearl Dsilva

As of now, when an admin encounters an issue or error in CloudStack, the maximum information they can immediately get is the API failure response which provides a reason for the failure. At times this might not be sufficinet to diagnose the error and would require the admin to investiage the CloudStack logs. This would require the admin or the sysadmin to log into the VM running CloudStack and either view or export the logs, and then dive into identifying the issue. This idea aims to eiliminate that step.

The goal of this is to provide admins the ability to view the logs directly in the UI. This would make diagnosing failures and RCAs much quicker.

Provide the ability display the logs in the UI

Add an API / WebSocket (and UI) support to :

• View the logs
• Live follow the logs (similar to 'tail -f')

Duration

• 175 hours

Potential Mentors

• David Jumani

References

• https://github.com/apache/cloudstack/issues/6011

Shutting down / Restarting Cloudstack is a necessary step in upgrades, system maintenance, etc. As of now, there is no way to safely shutdown or restart CloudStack. It is directly terminated via systemd. Since this is the case, any asyncronous job or background task is abrubptly terminated and can fail. As of now, CloudStack maintains a list of asynchronous jobs wihtin it's database along with their status.

This idea aims to provide a way to safely shutdown CloudStack. It involves two parts :

• Prevent new asynchronous jobs from being added to CloudStack when a safe shutdown is triggered
• Check the status of the async jobs and Shut down CloudStack when all the jobs have been completed

Provide the ability to safely shutdown CloudStack

Add API (and/or UI) support to :

• Trigger a safe shutdown
• (Optional) Support restarts
• (Optional) Support a forced shutdown when CloudStack will quit even if there are async jobs running

Duration

• Some Experience : 175 hours
• Newbie : 350 hours

Potential Mentors

• David Jumani

References

• https://github.com/apache/cloudstack/issues/6021

As of now the CloudStack Terraform Provider does not support managing CKS clusters

This proposal aims to add support to the CloudStack Terraform Provider to manage CKS clusters

This would involve supporting the following actions on CKS clusters :

• Create
• Stop / Start
• Scale
• Upgrade
• Delete

[Optional]
Support the following actions on the binary ISOs :

• Register
• Enable / Disable
• Delete

Duration

• 175 hours

Potential Mentors

• Harikrishna Patnala
• David Jumani

References

• https://github.com/apache/cloudstack/issues/6040

Background

Currently, Deploy Instances/Virtual Machines(VMs) in Cloudstack requires to specify some offerings, template and other settings through the API (check the API here: https://cloudstack.apache.org/api/apidocs-4.16/apis/deployVirtualMachine.html) or the 'Instance Deployment Wizard' in the UI.

Requirement

Provision to user/operator to quick deploy an instance using a VM definition/profile. The VM definition/profile would hold the details of the template, offerings (including any custom values - size, iops), ssh keypair, instance group, affinity group and other settings (boot type, dynamic scaling, userdata, keyboard language, etc) that are required, and the underlying definition/profile id can be used to launch an instance. At the minimum, the definition should hold all the mandatory details for deploying an instance. With this, only the VM definitions/profiles (and other important options, with the associated billing details) can be exposed to the users for VM deployment, instead of the offerings and other VM options.

Need to add new APIs (and/or UI) support for the VM definition/profile CRUD operations, and support for definition in the deployVirtualMachine API.

Relevant Skills

• Java, MySQL
• Vue.js (for UI)
• Some knowledge of Virtualization and CloudStack

Difficulty

Medium

Potential Mentors

• Suresh Kumar Anaparti
• David Jumani

Project Scope/Duration

Medium / 175 hours

References

• http://docs.cloudstack.apache.org/en/latest/adminguide/index.html#working-with-virtual-machines
• https://cloudstack.apache.org/api/apidocs-4.16/apis/deployVirtualMachine.html
• https://cloudstack.apache.org/api/apidocs-4.16/

Currently, defining an affinity or anti-affinity rule works only at hosts level. I would like to have more detail on the affinity group, extending it at different levels (cluster, pod, zone,..) and also within the same level, being able to add or remove resources from the group.

For hosts and storage pools, administrators can make use of host tags or storage tags to get a similar result. However, the extension of affinity/anti-affinity groups would make the administration easier.

Size of the project: Medium (~175hs)

An infrastructure outage can take out several or all VMs. In the aftermath it's not always possible to know which VMs were supposed to be ON or OFF, especially if HA is not enabled. People keep powered off VMs around all the time for many reasons.

I propose we add a feature where Cloudstack keeps track of the "last known state" of a VM and after an outage either enforce it (ie start the VM or leave it off) or at least show some information to the operator in the UI/API so they can do it themselves; perhaps make this behaviour configurable in the global settings.

Thanks

Background

CloudStack allows users/operators to perform various operations on the Virtual Machines (VMs). When multiple operations are performed on a VM at the same time, these operations are maintained and sync-ed using the sync queues. Any long running job (eg. volume snapshot) of a VM keeps other jobs in waiting/pending state, and only be picked once the active job is finished. Currently, it is not possible for an operator to list the pending jobs on a VM, cancel or re-prioritise any job if needed.

Requirement

Provision to admin/operator, to the list the pending jobs of a VM, cancel or re-prioritise a job if needed. Also, allow to clear all the pending jobs of a VM.

Add API (and/or UI) support to

• List the active jobs for a VM
• List all the pending jobs of a VM (in queue, by their order of execution)
• Re-prioritise a job from the pending jobs (if possible)
• Cancel any job from the pending jobs
• Clear all the pending jobs of a VM

Relevant Skills

• Java, MySQL
• Vue.js (for UI)
• Some knowledge of CloudStack and its Job framework

Difficulty

Medium

Potential Mentors

• Suresh Kumar Anaparti
• Any Developer from CS Community

Project Scope/Duration

Large / 350hrs (can be Medium / 175 hours - with reduced scope of API/UI work)

References

• http://docs.cloudstack.apache.org/en/latest/adminguide/index.html#working-with-virtual-machines
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/FS+-+VMSync+improvement
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/Snapshot+throttling+FS
• https://cloudstack.apache.org/api/apidocs-4.16/

Future Extensions

This can be extended for other resources (hosts, primary storage, network, etc).
[APIs should take resource type as a param for generic implementation]

This can be an interesting task for an engineer with domain knowledge on backend services in Java and some knowledge in Vue.js or relevant tech. Also, domain knowledge of OAuth authentication is desirable.

The main objectives of this task are:

• Create a new CloudStack authentication plugin: this plugin will allow authentication to third-party libraries such as Google, Facebook, Github, etc.
• Extend CloudStack configurations: allow administrators to enable/disable the plugin and configure the auth provider

More information about the task on: https://github.com/apache/cloudstack/issues/4834

Size: Medium

With regards to IP info reporting, Cloudstack relies entirely on it's DHCP data bases and so on. When this is not available (L2 networks etc) no IP information is shown for a given VM.

I propose we introduce a mechanism for "IP autodetection" and try to discover the IPs used inside the machines by means of querying the hypervisors. For example with KVM/libvirt we can simply do something like this:

            root@fedora35 ~]# virsh domifaddr win2k22 --source agent  
            Name MAC address Protocol Address 
            ------------------------------------------------------------------------------- 
            Ethernet 52:54:00:7b:23:6a ipv4 192.168.0.68/24 
            Loopback Pseudo-Interface 1 ipv6 ::1/128 - - ipv4 127.0.0.1/8 

The above command queries the qemu-guest-agent inside the Windows VM. The VM needs to have the qemu-guest-agent installed and running as well as the virtio serial drivers (easily done in this case with virtio-win-guest-tools.exe ) as well as a guest-agent socket channel defined in libvirt.

Once we have this information we could display it in the UI/API as "Autodetected VM IPs" or something like that.

I imagine it's very similar for VMWare and XCP-ng.

Thank you

The existing compaction strategies have a number of drawbacks that make all three unsuitable as a general use compaction strategy, for example STCS creates giant files that are hard to back up, mess with read performance and the page cache, and led to many of the early re-open bugs. LCS improved dramatically on this but also has various issues e.g. lack of performant full compaction or due to the strict leveling with e.g. bulk loading when writes exceed the rate we can do the L0 - L1 promotion.

In this talk I proposed a novel compaction strategy that aims to expose a single tunable that the user can control for the read amplification. Raise the min_threshold_levels and you tradeoff read/space performance for write performance. Since then a proof of concept patch has been published along with some rudimentary documentation but this is still not tracked in Jira.

The remaining work here is

1. Validate the algorithm is correct via test suites and performance testing stress testing and benchmarking with OSS tools (e.g. cassandra-stress, tlp-stress, or ndbench). When issues are found (there likely will be issues as the patch is a PoC), devise how to adjust the algorithm and implementation appropriately. Key metric of success is we can run Cassandra stably for more than 24 hours while applying sustained load, with minimal compaction load (and also compaction can keep up).

2. Do more in depth experiments measuring performance across a wide range of workloads (e.g. write heavy, read heavy, balanced, time series, register update, etc ...) and in comparison with LCS (leveled), STCS (size tiered), and TWCS (time window). Key metrics of success are establishing that as we tune max_read_per_read we should get more predictable read latency under low system load (ρ < 30%) while not degrading at high system load (ρ > 70%), and we should match LCS performance under low load while doing better at high load.

Once this is validated a Cassandra blog post reporting on the findings (positive or negative) may be advisable.

We should provide users a way to understand how their query will be executed and some information on the amount of work that will be performed.
Explain statements are the most common way to do that.
A CEP Draft has been open for that: (DRAFT) CEP-4: Explain. This draft propose to add support for EXPLAIN and EXPLAIN ANALYZE but I believe that we should split the work in 2 parts because a simple EXPLAIN would already provide relevant information.

To complete this work I believe that the following steps will be required:

• Rework and submit the CEP
• Add missing statistics
• Implements the logic behind the EXPLAIN statements

Apache Beam [1] is a unified and portable programming model for data processing jobs. The Beam model [2, 3, 4] has rich mechanisms to process endless streams of events.

Complex Event Processing [5] lets you match patterns of events in streams to detect important patterns in data and react to them.

Some examples of uses of CEP are fraud detection for example by detecting unusual behavior (patterns of activity), e.g. network intrusion, suspicious banking transactions, etc. Also trend detection is another interesting use case in the context of sensors and IoT.

The goal of this issue is to implement an efficient pattern matching library inspired by [6] and existing libraries like Apache Flink CEP [7] using the Apache Beam Java SDK and the Beam style guides [8]. Because of the time constraints of GSoC we will probably try to cover first simple patterns of the ‘a followed by b followed by c’ kind, and then if there is still time try to cover more advanced ones e.g. optional, atLeastOne, oneOrMore, etc.

[1] https://beam.apache.org/
[2] https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
[3] https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102
[4] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43864.pdf
[5] https://en.wikipedia.org/wiki/Complex_event_processing
[6] https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf
[7] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/cep.html
[8] https://beam.apache.org/contribute/ptransform-style-guide/

Ray (https://ray.io) is a framework to develop distributed applications. There is a push to develop several libraries to support vario7us forms for AI/ML analytics with Ray. There is an opportunity to develop a Beam runner for Ray.

https://docs.google.com/document/u/1/d/1vt78s48Q0aBhaUCHrVrTUsProJSP8-EBqDDRGTPEr0Y/edit

We have the Beam Pydoc set up, and some functions have examples written into their documentaztion, however we do not run the examples that we express in Pydoc.

This work item consists in improving the Pydoc for Apache Beam to run examples, adding some examples, and reformatting any existing examples / existing Pydoc that needs to be better expressed for Beam.

It would be desirable to develop a Beam Sink that supports all of the 'best practices' like throttling, auto-sharding, exactly-once capabilities, etc.

A design proposal is here: https://docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org^{Image Added} if you have questions, or comment here.

It would be desirable to develop a Beam Sink that supports all of the 'best practices' like throttling, auto-sharding, exactly-once capabilities, etc.

A design proposal is here: https://docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org^{Image Added} if you have questions, or comment here.

The goal for this project has changed: We now want to create a completely new Capability Matrix that is based on the ValidatesRunner tests that we run on the various Apache Beam runners.

We can use the test in ./test-infra/validates-runner/ to generate a JSON file that contains the capabilities supported by various runners and tested by each individual test.

----------------------------------------------------

Discussion: https://lists.apache.org/thread.html/8aff7d70c254356f2dae3109fb605e0b60763602225a877d3dadf8b7@%3Cdev.beam.apache.org%3E

Summarizing that discussion, we have a lot of issues/wishes. Some can be addressed as one-off and some need a unified reorganization of the runner comparison.

Basic corrections:

• Remove rows that impossible to not support (ParDo)
• Remove rows where "support" doesn't really make sense (Composite transforms)
• Deduplicate rows are actually the same model feature (all non-merging windowing / all merging windowing)
• Clearly separate rows that represent optimizations (Combine)
• Correct rows in the wrong place (Timers are actually a "what...?" row)
• Separate or remove rows have not been designed ([Meta]Data driven triggers, retractions)
• Rename rows with names that appear no where else (Timestamp control, which is called a TimestampCombiner in Java)
• Switch to a more distinct color scheme for full/partial support (currently just solid/faded colors)
• Switch to something clearer than "~" for partial support, versus ✘ and ✓ for none and full.
• Correct Gearpump support for merging windows (see BEAM-2759)
• Correct Spark support for non-merging and merging windows (see BEAM-2499)

Minor rewrites:

• Lump all the basic stuff (ParDo, GroupByKey, Read, Window) into one row
• Make sections as users see them, like "ParDo" / "side Inputs" not "What?" / "side inputs"
• Add rows for non-model things, like portability framework support, metrics backends, etc

Bigger rewrites:

• Add versioning to the comparison, as in BEAM-166
• Find a way to fit in a plain English summary of runner's support in Beam. It should come first, as it is what new users need before getting to details.
• Find a way to describe production readiness of runners and/or testimonials of who is using it in production.
• Have a place to compare non-model differences between runners

Changes requiring engineering efforts:

• Gather and add quantitative runner metrics, perhaps Nexmark results for mid-level, smaller benchmarks for measuring aspects of specific features, and larger end-to-end benchmarks to get an idea how it might actually perform on a use case
• Tighter coupling of the matrix portion of the comparison with tags on ValidatesRunner tests

If you care to address some aspect of this, please reach out and/or just file a subtask and address it.

In Stream processing, we have many methods, starting from the primitive checkpoint-and-replay to a more fancy version of reconfiguration and reinitiation of stream workloads. We aim to find a way to find the most effective and efficient way of reconfiguring stream workloads. Sub-issues are to be created later on.

An infrastructure outage can take out several or all VMs. In the aftermath it's not always possible to know which VMs were supposed to be ON or OFF, especially if HA is not enabled. People keep powered off VMs around all the time for many reasons.

I propose we add a feature where Cloudstack keeps track of the "last known state" of a VM and after an outage either enforce it (ie start the VM or leave it off) or at least show some information to the operator in the UI/API so they can do it themselves; perhaps make this behaviour configurable in the global settings.

Thanks

Nemo has a policy layer that allows powerful optimization with configurable runtime modules. In terms of caching, it is possible to identify frequently used data and decide to cache them in-memory ahead of execution, without user annotation.

Implementation would need:

• On policy layer, build compile-time pass that identify reused data and mark them as cached
• On runtime, design and implement caching mechanism that manages per-executor cached data and discard them when these are no longer used.

Currently, Nemo doesn't have a spill mechanism. This makes executors prone to memory problems such as OOM(Out Of Memory) or GC when task data is large. For example, handling skewed shuffle data in Nemo results in OOM and executor failure, as all data has to be handled in-memory.

We need to spill in-memory data to secondary storage when there are not enough memory in executor.

In-memory caching and spilling are essential features in in-memory big data processing frameworks, and Nemo needs one.

• Identify and persist frequently used data and unpersist it when its usage ended
• Spill in-memory data to disk upon memory pressure

Background

CloudStack allows users/operators to perform various operations on the Virtual Machines (VMs). When multiple operations are performed on a VM at the same time, these operations are maintained and sync-ed using the sync queues. Any long running job (eg. volume snapshot) of a VM keeps other jobs in waiting/pending state, and only be picked once the active job is finished. Currently, it is not possible for an operator to list the pending jobs on a VM, cancel or re-prioritise any job if needed.

Requirement

Provision to admin/operator, to the list the pending jobs of a VM, cancel or re-prioritise a job if needed. Also, allow to clear all the pending jobs of a VM.

Add API (and/or UI) support to

• List the active jobs for a VM
• List all the pending jobs of a VM (in queue, by their order of execution)
• Re-prioritise a job from the pending jobs (if possible)
• Cancel any job from the pending jobs
• Clear all the pending jobs of a VM

Relevant Skills

• Java, MySQL
• Vue.js (for UI)
• Some knowledge of CloudStack and its Job framework

Difficulty

Medium

Potential Mentors

• Suresh Kumar Anaparti
• Any Developer from CS Community

Project Scope/Duration

Large / 350hrs (can be Medium / 175 hours - with reduced scope of API/UI work)

References

• http://docs.cloudstack.apache.org/en/latest/adminguide/index.html#working-with-virtual-machines
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/FS+-+VMSync+improvement
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/Snapshot+throttling+FS
• https://cloudstack.apache.org/api/apidocs-4.16/

Future Extensions

The load of streaming jobs usually fluctuate according to the input rate or operations (e.g., window). Supporting the automatic scaling could reduce the operational cost of running streaming applications, while minimizing the performance degradation that can be caused by the bursty loads. 

We can harness the cloud resources such as VMs and serverless frameworks to acquire computing resources on demand. To realize the automatic scaling, the following features should be implemented.

1) state migration: scaling jobs require moving tasks (or partitioning a task to multiple ones). In this situation, the internal state of the task should be serialized/deserialized. 

2) input/output rerouting: if a task is moved to a new worker, the input and output of the task should be redirected. 

3) dynamic Executor or Task creation/deletion: Executor}}s or {{Task can be dynamically created or deleted. 

4) scaling policy: a scaling policy that decides when and how to scale out/in should be implemented. 

This can be an interesting task for an engineer with domain knowledge on backend services in Java and some knowledge in Vue.js or relevant tech. Also, domain knowledge of OAuth authentication is desirable.

The main objectives of this task are:

• Create a new CloudStack authentication plugin: this plugin will allow authentication to third-party libraries such as Google, Facebook, Github, etc.
• Extend CloudStack configurations: allow administrators to enable/disable the plugin and configure the auth provider

More information about the task on: https://github.com/apache/cloudstack/issues/4834

Size: Medium

Mifos X was built to be cloud ready from the ground up. One of the most popular deployment environments for MifosX has been on Amazon EC2, however due to country specific regulation, many implementors are forced to seek alternative models that can scale as effectively. The aim of this project is two-fold:

• Propose a scalable deployment model for Mifos on Google Cloud. Your application should highlight a starting point with some details of your planned deployment architecture, as Mentors would be not giving you step-by-step instructions in this project, just "nudge" you along; you would be expected to learn about how to deploy Mifos yourself and by autonomously using the documentation available and help from the public mailing list and IRC channel, and figure out the details of the Cloud deployment.

• Propose how the above proposed model could be contributed to Mifos in the form of e.g. ready-to-run "configurations" etc. allowing ANYONE to deploy THE LATEST VERSION of Mifos in the Cloud themselves, and then implement this approach in practice. (Contrast this with a "one-off exercise", e.g. taking the current Mifos X WAR file, and UI, and manually making some changes to it, and then manually deploying that to some Cloud PaaS - this would not be sufficient for this project's expectations.)

• Implement a Continuous Deployment "Devops" EXAMPLE instance of this scalable blueprint using the latest nightly Mifos build artifacts.

• Publish a high level whitepaper of the same, which can be used as a reference for local implementors, who would additionally take care of provisioning their own hardware. This documentation should be ideally high-level, and what it described much be automated; only providing lengthy step-by-step manual instructions would not be sufficient for this project's expectations.

To prepare for this project, applying contibutors must demonstrate at least that they have already successfully locally built and ran a Mifos X REST back-end server and UI, populated the database etc. as well as provided a simple pull request proposing some minimal deployment related improvement.

Note that we now believe that a Platform as a Service (PaaS) is a more suitable foundation for this project than a raw Cloud Infrastructure as a Service (IaaS) platform (such as Openstack, offered by public cloud provider such as e.g. Rackspace; or Azure, or raw Amazon EC2). This is because a PaaS, such OpenShift, already come with relevant features such as built-in, managed, supported and monitored HTTP load balancing (e.g. OpenShift comes with HAProxy).

The MariaDB (MySQL) database used by Apache Fineract/Mifos does not offer clustering. We believe that this would not be required, and that proper configuration of the already existing cache facility (incl. distributed cache invalidation) available in Mifos X will add more value at signficantly less operational complexity.

You may need to develop some minor "adjustments" for Mifos X to work well in a PaaS. For example, writeable directories may be limited, and configuration changes may be needed to pick up allowed data directories from an environment variable configuration (but consider multi node distribution in this cluster setup!). Also a cloud PaaS like OpenShift may not support "always running" instances, and scheduled jobs may have to be configured to be kicked off via an explicit HTTP "wake up" request from a cron job.

As our product is core banking platform and our clients are financial institutions, we strive hard to make our code base as secure as possible. However, due to ever increasing security threats and vulnerabilities, it is the need of hour that we analyze our code base in depth for security vulnerabilities. During pull request merge process, we have a process in place wherein we do peer code review,QA and integration tests. This practice has been very effective and our community is already reaping the benefits of such a strong code review process. However, we should test our code against the standard vulnerabilities which have been identified by reputed organisations like Mitre to gain more confidence. It has become a critical part of independent and partner-led deployments

We can make use of opensource tools like Jlint, Findbugs , SonarQube or frameworks like  Total output Integration Framework (TOIF) - used by companies dedicated to produce military grade secure systems. As our environments become more containerized we can also utilize tools like: Anchore, Snyk.io, and Docker Bench for Security

It would be worthwhile, if we can dedicate one GSOC project for this analysis and fixing of critical vulnerabilities and actual bugs. The student would be responsible to analyse the findings, generate reports, identify if it is really a bug and then submit a fix after consultation from the community. Of course, the student needs to demonstrate some basic understanding of security vulnerabilities( like buffer overflow etc) and should have some academic level of experience working with static analysis tools.

Prioritization of Focus would be on:

• Vulnerabilities, Hotspots, Bugs, and Code Smells in that order.

We should provide users a way to understand how their query will be executed and some information on the amount of work that will be performed.
Explain statements are the most common way to do that.
A CEP Draft has been open for that: (DRAFT) CEP-4: Explain. This draft propose to add support for EXPLAIN and EXPLAIN ANALYZE but I believe that we should split the work in 2 parts because a simple EXPLAIN would already provide relevant information.

To complete this work I believe that the following steps will be required:

• Rework and submit the CEP
• Add missing statistics
• Implements the logic behind the EXPLAIN statements

Discussion: https://lists.apache.org/thread.html/8aff7d70c254356f2dae3109fb605e0b60763602225a877d3dadf8b7@%3Cdev.beam.apache.org%3E

Summarizing that discussion, we have a lot of issues/wishes. Some can be addressed as one-off and some need a unified reorganization of the runner comparison.

Basic corrections:

• Remove rows that impossible to not support (ParDo)
• Remove rows where "support" doesn't really make sense (Composite transforms)
• Deduplicate rows are actually the same model feature (all non-merging windowing / all merging windowing)
• Clearly separate rows that represent optimizations (Combine)
• Correct rows in the wrong place (Timers are actually a "what...?" row)
• Separate or remove rows have not been designed ([Meta]Data driven triggers, retractions)
• Rename rows with names that appear no where else (Timestamp control, which is called a TimestampCombiner in Java)
• Switch to a more distinct color scheme for full/partial support (currently just solid/faded colors)
• Switch to something clearer than "~" for partial support, versus ✘ and ✓ for none and full.
• Correct Gearpump support for merging windows (see BEAM-2759)
• Correct Spark support for non-merging and merging windows (see BEAM-2499)

Minor rewrites:

• Lump all the basic stuff (ParDo, GroupByKey, Read, Window) into one row
• Make sections as users see them, like "ParDo" / "side Inputs" not "What?" / "side inputs"
• Add rows for non-model things, like portability framework support, metrics backends, etc

Bigger rewrites:

• Add versioning to the comparison, as in BEAM-166
• Find a way to fit in a plain English summary of runner's support in Beam. It should come first, as it is what new users need before getting to details.
• Find a way to describe production readiness of runners and/or testimonials of who is using it in production.
• Have a place to compare non-model differences between runners

Changes requiring engineering efforts:

• Gather and add quantitative runner metrics, perhaps Nexmark results for mid-level, smaller benchmarks for measuring aspects of specific features, and larger end-to-end benchmarks to get an idea how it might actually perform on a use case
• Tighter coupling of the matrix portion of the comparison with tags on ValidatesRunner tests

If you care to address some aspect of this, please reach out and/or just file a subtask and address it.

As our product is core banking platform and our clients are financial institutions, we strive hard to make our code base as secure as possible. However, due to ever increasing security threats and vulnerabilities, it is the need of hour that we analyze our code base in depth for security vulnerabilities. During pull request merge process, we have a process in place wherein we do peer code review,QA and integration tests. This practice has been very effective and our community is already reaping the benefits of such a strong code review process. However, we should test our code against the standard vulnerabilities which have been identified by reputed organisations like Mitre to gain more confidence. It has become a critical part of independent and partner-led deployments

We can make use of opensource tools like Jlint, Findbugs , SonarQube or frameworks like  Total output Integration Framework (TOIF) - used by companies dedicated to produce military grade secure systems. As our environments become more containerized we can also utilize tools like: Anchore, Snyk.io, and Docker Bench for Security

It would be worthwhile, if we can dedicate one GSOC project for this analysis. The student would be responsible to analyse the findings, generate reports, identify if it is really a bug and then submit a fix after consultation from the community. Of course, the student needs to demonstrate some basic understanding of security vulnerabilities( like buffer overflow etc) and should have some academic level of experience working with static analysis tools.

The increasing need for fast and reliable access to financial services has prompted the expansion of Apache Fineract from a single complex financial platform to a platform constituted of multiple micro-services that interact and scale to meet up with this increased need - Apache Fineract CN. Apache Fineract CN is a digital financial application platform built to render financial services to consumers in a fast, reliable and scalable manner. Deploying this platform such that consumers get the latest features with no reduction impact requires an optimized release cycle in a CI/CD (continuous integration and continuous Deployment) environment.

In view of that, last year Courage began this work by implementing the needed scripts to containerize and deploy the Fineract CN services using Docker, Docker compose and Kubernetes. For the Google Summer of Code 2020, you are required to complete this work by performing the following task:

• Improve Docker-compose deployment configuration to deploy on a swarm node

• Implement new Fineract service to generate RSA keys and complete the provisioning process.

• Improve provisioner and migration script to work with both a swarm cluster and a single machine running multiple compose services.

• Build and publish the Fineract images on Docker hub.

• Link Docker Hub to Github service repositories via an Automation Server pipeline.

• Publish the built Fineract CN services libraries to a Maven Artifactory so developers will not have to manually clone and publish these services by themselves.

N.B: 

• I would like to hear the applicants own ideas.

• The task for the completion of this project may change depending on input from the community, the mentors and the applicant.

A new reference user interface on Fineract CN for staff of financial institutions such as digital, challenger, and neo-banks that focused on individual accounts is needed for multiple reasons:

1. The current fims-web-app reference UI on top of Fineract CN is incomplete, unpolished and doesn't serve as a good representation of capabilities of Fineract CN.

1. As more financial inclusion providers focus on individual lending and savings products and more digital banks/neo-banks and fintechs that don't have group or center-based operations explore Mifos and Fineract CN, we'd need to have a reference UI that is more in line with those requirements. We don't want prospective users to come and see the microfinance-centric UI and immediately think that the platform might not be useful for them.

Intern will work on the following tasks:

• Upgrade dependencies to latest versions

• Improve overall user experience and look and feel

• implement the front-end UI screens for the Fineract CN web UI for the following functionalities and use case:

• • Account Details

• • Notifications

• • Transaction Details

• • Account Opening

• • Accounting

• • Reporting

• More Use cases to be listed.

We have the Beam Pydoc set up, and some functions have examples written into their documentaztion, however we do not run the examples that we express in Pydoc.

This work item consists in improving the Pydoc for Apache Beam to run examples, adding some examples, and reformatting any existing examples / existing Pydoc that needs to be better expressed for Beam.

The goal of this project is to expand unit testing coveragea cross the Finerat platform. Currently most of our automated testing is only through integration tests which take a long time to run and aren’t consistent. Cucumber is being implemented as the unit test framework and this project would focus on converting existing integration tests to unit test and writing new unit tests.

Goals are to increase testing coverage of core modules, reduce run-time at build of completing tests, and implementing some automated reporting to show testing coverage.

The student will be working on implementing the following things:

1. Collaborate with mentor to implement Cucumber framework

1. Collaborate with mentor to implement test containers

1. Refine test data set and scripts

1. Convert high priority integration tests to unit tests

1. Write unit tests for key functional modules

1. Implement reporting to show test coverage.

It would be desirable to develop a Beam Sink that supports all of the 'best practices' like throttling, auto-sharding, exactly-once capabilities, etc.

A design proposal is here: https://docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org^{Image Removed} if you have questions, or comment here.

It would be desirable to develop a Beam Sink that supports all of the 'best practices' like throttling, auto-sharding, exactly-once capabilities, etc.

A design proposal is here: https://docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org^{Image Removed} if you have questions, or comment here.

Just as we have a mobile field operations app on Apache Fineract 1.x, we have recently built out on top of the brand new Apache Fineract CN micro-services architecture, an initial version of a mobile field operations app with an MVP architecture and material design. Given the flexibily of the new architecture and its ability to support different methodologies - MFIs, credit unions, cooperatives, savings groups, agent banking, etc - this mobile app will have different flavors and workflows and functionalities. 

In 2021, our Google Summer of Code intern, Varun Jain worked on additional functionality in the Fineract CN mobile app. In 2022, the student will work on the following tasks:

• Continuing migration to Kotlin

• Incorporating more external transaction flows through Payment Hub EE

• Improving notifications generation in the app.

• Enhancing the customer onboarding and loan origination user experience I

• Refining the UI and look and feel of app.

• More robust survey and data capture features.

• Integration with external ID systems for biometric verification

• Integration with voucher generation.

• Improve GIS features like location tracking, dropping of pin into the app

• Improve offline mode via Couchbase support

• Write Unit Test, Integration Test and UI tests

In Stream processing, we have many methods, starting from the primitive checkpoint-and-replay to a more fancy version of reconfiguration and reinitiation of stream workloads. We aim to find a way to find the most effective and efficient way of reconfiguring stream workloads. Sub-issues are to be created later on.

Nemo has a policy layer that allows powerful optimization with configurable runtime modules. In terms of caching, it is possible to identify frequently used data and decide to cache them in-memory ahead of execution, without user annotation.

Implementation would need:

• On policy layer, build compile-time pass that identify reused data and mark them as cached
• On runtime, design and implement caching mechanism that manages per-executor cached data and discard them when these are no longer used.

Just as we have a client-facing mobile banking app for our generation 2 Apache Fineract 1.0 platform, we need to provide a reference mobile banking app on top of the Apache Fineract CN architecture which allows a client to securely authenticate against the microservices architecture and interact with his/her accounts. 

A major focal area for the 2022 GSOC is to integrate with the Open Banking API layer built on top of the WS02 API Gateway which provides a secure authentication and integration layer for first party applications as currently the app only is consuming a mock layer of data. Additional use cases would include better support for transactions via external payment systems, improving the workflow for sign-up and account creation and implementing new UI designs. 

• Integrate with Fineract CN via Open Banking API layer on WS02 API Gateway

• Map APIs to Open Banking APIs

• Improve workflow for self-guided sign-up, account creation, and initial authentication. 

• Implement new UI designs developed during GCI and documented at https://docs.google.com/presentation/d/1yFR19vGlKW-amxzGms8TgPzd1jWkrALPFcaC85EyYpw/edit?usp=sharing

• Integration with external payment systems via Mojaloop and GSMA mobile money API. 

Currently, Nemo doesn't have a spill mechanism. This makes executors prone to memory problems such as OOM(Out Of Memory) or GC when task data is large. For example, handling skewed shuffle data in Nemo results in OOM and executor failure, as all data has to be handled in-memory.

We need to spill in-memory data to secondary storage when there are not enough memory in executor.

In-memory caching and spilling are essential features in in-memory big data processing frameworks, and Nemo needs one.

• Identify and persist frequently used data and unpersist it when its usage ended
• Spill in-memory data to disk upon memory pressure

Lombok could help us to not only reduce a large amount of code, but also to fix a couple of inconsistencies in the code base:

• getters/setters with non-standard characters (e. g. underscores)

• getters/setters with typos

The layered architecture of Fineract requires mapping between REST DTO classes and internal entity classes. The current code base contains various strategies to achieve this:

• private functions

• static functions

• mapping classes

All of these approaches are very manual (and error prone) and difficult to maintain. Mapstruct can help here:

• throw errors at compile time (missing new attributes, type changes etc.)

• one common concept (easier to understand)

• reduce manually maintained code and replace mostly generated code

Challenges:

• maintain immutability (especially in DTO classes)

• should we fluent builder pattern?

• backwards compatibility

• these improvements cannot be introduced as one pull request, but have to be split up at least at the “module” level (clients, loans, accounts etc.). This would result in approximately 30 pull requests; if we split up Lombok and Mapstruct then it would be 30 PRs each (=60); we would need this fine grained approach to make a transition as painless as possible

• some classes are maybe beyond repair (e. g. Loan.java with 6000 lines of code, the smaller part getters/setters and a long list of utility/business logic functions)

Dubbo provides implementations of almost all mainstream languages from Java, Golang, Javascript, C# to Python, etc.In this project, we want to build a basic Rust implementation for Dubbo.to Python, etc.

In this project, we want to build a basic Rust implementation for Dubbo.

Description

Please read the Observasibility proposal here first to know about the ultimate goal behind this issue.

If you are interested in this project and the objective described in the proposal, please leave comments on the corresponding Github issue below so we can further exchange information on the tasks that need to be done.

https://github.com/apache/dubbo-go/issues/1807

Please read the detailed proposal of Dubbo Sidecar Proxyless Mesh or Thin SDK here first to know about the ultimate goal behind this issue.

The details of this project will be posted on the following GitHub issue, please keep posted there.

https://github.com/apache/dubbo/issues/98859884

Please read read the detailed proposal of Dubbo Proxyless Mesh here Sidecar Mesh or Thin SDK here first to know about the ultimate goal behind this issue.

The details of this project will be posted on the following GitHub issue, please keep posted there.

https://github.com/apache/dubbo-go/issues/98841809

Description

Please read the Observasibility proposal here Please read the detailed proposal of Dubbo Proxyless Mesh here first to know about the ultimate goal behind this issue.

If you are interested in this project and the objective described in the proposal, please leave comments on the corresponding Github issue below so we can further exchange information on the tasks that need to be doneThe details of this project will be posted on the following GitHub issue, please keep posted there.

https://github.com/apache/dubbo-go/issues/18071808

The details of this project will be posted on the following GitHub issue, please keep posted there.

https://github.com/apache/dubbo-rust/issues/2

The details of this project will be posted on the following GitHub issue, please keep posted there.

https://github.com/apache/dubbo-gorust/issues/18081

We will utilize ngx-graph library simialar to interactive query plan viewer (ASTERIXDB-2863) in order to display an interactive query plan that supports DAGs.

Features:

• Colored nodes (by operator)
• Zoom out to fit whole plan
• Zoom and drag through the plan
• Traverse the nodes or jump to nodes in a Depth First Search (DFS) fashion
• Detail number of locations for execution 
• Detailed mode (contains more information per node)
  • Search using string match
• Clear all selections and reset the interactive plan
   

Complete all transports in MFT

• Currently SCP, S3 is known to work
• Others need effort to optimize, test, and declare readiness
• Develop a complete a fully functional MFT Command-line interface
• Have a feature-complete Python SDK
• A minimum implementation will be prvoided, students need to complete it and test it.Clear all selections and reset the interactive plan