Background

Currently, Deploy Instances/Virtual Machines(VMs) in Cloudstack requires to specify some offerings, template and other settings through the API (check the API here: https://cloudstack.apache.org/api/apidocs-4.16/apis/deployVirtualMachine.html) or the 'Instance Deployment Wizard' in the UI.

Requirement

Provision to user/operator to quick deploy an instance using a VM definition/profile. The VM definition/profile would hold the details of the template, offerings (including any custom values - size, iops), ssh keypair, instance group, affinity group and other settings (boot type, dynamic scaling, userdata, keyboard language, etc) that are required, and the underlying definition/profile id can be used to launch an instance. At the minimum, the definition should hold all the mandatory details for deploying an instance. With this, only the VM definitions/profiles (and other important options, with the associated billing details) can be exposed to the users for VM deployment, instead of the offerings and other VM options.

Need to add new APIs (and/or UI) support for the VM definition/profile CRUD operations, and support for definition in the deployVirtualMachine API.

Relevant Skills

• Java, MySQL
• Vue.js (for UI)
• Some knowledge of Virtualization and CloudStack

Difficulty

Medium

Potential Mentors

• Suresh Kumar Anaparti
• David Jumani

Project Scope/Duration

Medium / 175 hours

References

• http://docs.cloudstack.apache.org/en/latest/adminguide/index.html#working-with-virtual-machines
• https://cloudstack.apache.org/api/apidocs-4.16/apis/deployVirtualMachine.html
• https://cloudstack.apache.org/api/apidocs-4.16/

Background

CloudStack allows users/operators to perform various operations on the Virtual Machines (VMs). When multiple operations are performed on a VM at the same time, these operations are maintained and sync-ed using the sync queues. Any long running job (eg. volume snapshot) of a VM keeps other jobs in waiting/pending state, and only be picked once the active job is finished. Currently, it is not possible for an operator to list the pending jobs on a VM, cancel or re-prioritise any job if needed.

Requirement

Provision to admin/operator, to the list the pending jobs of a VM, cancel or re-prioritise a job if needed. Also, allow to clear all the pending jobs of a VM.

Add API (and/or UI) support to

• List the active jobs for a VM
• List all the pending jobs of a VM (in queue, by their order of execution)
• Re-prioritise a job from the pending jobs (if possible)
• Cancel any job from the pending jobs
• Clear all the pending jobs of a VM

Relevant Skills

• Java, MySQL
• Vue.js (for UI)
• Some knowledge of CloudStack and its Job framework

Difficulty

Medium

Potential Mentors

• Suresh Kumar Anaparti
• Any Developer from CS Community

Project Scope/Duration

Large / 350hrs (can be Medium / 175 hours - with reduced scope of API/UI work)

References

• http://docs.cloudstack.apache.org/en/latest/adminguide/index.html#working-with-virtual-machines
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/FS+-+VMSync+improvement
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/Snapshot+throttling+FS
• https://cloudstack.apache.org/api/apidocs-4.16/

Future Extensions

This can be extended for other resources (hosts, primary storage, network, etc).
[APIs should take resource type as a param for generic implementation]

The existing compaction strategies have a number of drawbacks that make all three unsuitable as a general use compaction strategy, for example STCS creates giant files that are hard to back up, mess with read performance and the page cache, and led to many of the early re-open bugs. LCS improved dramatically on this but also has various issues e.g. lack of performant full compaction or due to the strict leveling with e.g. bulk loading when writes exceed the rate we can do the L0 - L1 promotion.

In this talk I proposed a novel compaction strategy that aims to expose a single tunable that the user can control for the read amplification

The existing compaction strategies have a number of drawbacks that make all three unsuitable as a general use compaction strategy, for example STCS creates giant files that are hard to back up, mess with read performance and the page cache, and led to many of the early re-open bugs. LCS improved dramatically on this but also has various issues e.g. lack of performant full compaction or due to the strict leveling with e.g. bulk loading when writes exceed the rate we can do the L0 - L1 promotion.

In this talk I proposed a novel compaction strategy that aims to expose a single tunable that the user can control for the read amplification. Raise the min_threshold_levels and you tradeoff read/space performance for write performance. Since then a proof of concept patch has been published along with some rudimentary documentation but this is still not tracked in Jira.

The remaining work here is

1. Validate the algorithm is correct via test suites and performance testing stress testing and benchmarking with OSS tools (e.g. cassandra-stress, tlp-stress, or ndbench). When issues are found (there likely will be issues as the patch is a PoC), devise how to adjust the algorithm and implementation appropriately. Key metric of success is we can run Cassandra stably for more than 24 hours while applying sustained load, with minimal compaction load (and also compaction can keep up).

2. Do more in depth experiments measuring performance across a wide range of workloads (e.g. write heavy, read heavy, balanced, time series, register update, etc ...) and in comparison with LCS (leveled), STCS (size tiered), and TWCS (time window). Key metrics of success are establishing that as we tune max_read_per_read we should get more predictable read latency under low system load (ρ < 30%) while not degrading at high system load (ρ > 70%), and we should match LCS performance under low load while doing better at high load.

Once this is validated a Cassandra blog post reporting on the findings (positive or negative) may be advisable.

Apache Beam [1] is a unified and portable programming model for data processing jobs. The Beam model [2, 3, 4] has rich mechanisms to process endless streams of events.

Complex Event Processing [5] lets you match patterns of events in streams to detect important patterns in data and react to them.

Some examples of uses of CEP are fraud detection for example by detecting unusual behavior (patterns of activity), e.g. network intrusion, suspicious banking transactions, etc. Also trend detection is another interesting use case in the context of sensors and IoT.

The goal of this issue is to implement an efficient pattern matching library inspired by [6] and existing libraries like Apache Flink CEP [7] using the Apache Beam Java SDK and the Beam style guides [8]. Because of the time constraints of GSoC we will probably try to cover first simple patterns of the ‘a followed by b followed by c’ kind, and then if there is still time try to cover more advanced ones e.g. optional, atLeastOne, oneOrMore, etc.

[1] https://beam.apache.org/
[2] https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
[3] https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102
[4] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43864.pdf
[5] https://en.wikipedia.org/wiki/Complex_event_processing
[6] https://wwwpeople.cs.oreillyumass.comedu/~yanlei/ideaspublications/the-world-beyond-batch-streaming-101
[3sase-sigmod08.pdf
[7] https://wwwci.oreillyapache.comorg/projects/ideasflink/theflink-world-beyond-batch-streaming-102
[4docs-stable/dev/libs/cep.html
[8] https://staticbeam.googleusercontentapache.com/media/research.google.com/en//pubs/archive/43864.pdf
[5] https://en.wikipedia.org/wiki/Complex_event_processing
[6] https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf
[7] https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/cep.html
[8] https://beam.apache.org/contribute/ptransform-style-guide/org/contribute/ptransform-style-guide/

Ray (https://ray.io) is a framework to develop distributed applications. There is a push to develop several libraries to support vario7us forms for AI/ML analytics with Ray. There is an opportunity to develop a Beam runner for Ray.

https://docs.google.com/document/u/1/d/1vt78s48Q0aBhaUCHrVrTUsProJSP8-EBqDDRGTPEr0Y/edit

We have the Beam Pydoc set up, and some functions have examples written into their documentaztion, however we do not run the examples that we express in Pydoc.

This work item consists in improving the Pydoc for Apache Beam to run examples, adding some examples, and reformatting any existing examples / existing Pydoc that needs to be better expressed for Beam.

It would be desirable to develop a Beam Sink that supports all of the 'best practices' like throttling, auto-sharding, exactly-once capabilities, etc.

A design proposal is here:

Ray (https://ray.io) is a framework to develop distributed applications. There is a push to develop several libraries to support vario7us forms for AI/ML analytics with Ray. There is an opportunity to develop a Beam runner for Ray.

https://docs.google.com/document/u/1/d/1vt78s48Q0aBhaUCHrVrTUsProJSP8-EBqDDRGTPEr0Y/edit

docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org^{Image Added} if you have questions, or comment here

We have the Beam Pydoc set up, and some functions have examples written into their documentaztion, however we do not run the examples that we express in Pydoc.

This work item consists in improving the Pydoc for Apache Beam to run examples, adding some examples, and reformatting any existing examples / existing Pydoc that needs to be better expressed for Beam.

It would be desirable to develop a Beam Sink that supports all of the 'best practices' like throttling, auto-sharding, exactly-once capabilities, etc.

A design proposal is here: https://docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org^{Image Removed} if you have questions, or comment here.

, exactly-once capabilities, etc.

A design proposal is here: https://docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org^{Image Added} if you have questions, or comment here.

In Stream processing, we have many methods, starting from the primitive checkpoint-and-replay to a more fancy version of reconfiguration and reinitiation of stream workloads. We aim to find a way to find the most effective and efficient way of reconfiguring stream workloads. Sub-issues are to be created later on.

It would be desirable to develop a Beam Sink that supports all of the 'best practices' like throttling, auto-sharding, exactly-once capabilities, etc.

A design proposal is here: https://docs.google.com/document/d/1UIWv6wnD86GYAkeqbVWCG3mx4dTZ9WstUUThPWQmcFM/edit#heading=h.smc16ifdre2

A prototype for the API and parts of implementation is here: https://github.com/apache/beam/pull/16763

Contact Pablo Estrada on dev@beam.apache.org^{Image Removed} if you have questions, or comment here.

Nemo has a policy layer that allows powerful optimization with configurable runtime modules. In terms of caching, it is possible to identify frequently used data and decide to cache them in-memory ahead of execution, without user annotation.

Implementation would need:

• On policy layer, build compile-time pass that identify reused data and mark them as cached
• On runtime, design and implement caching mechanism that manages per-executor cached data and discard them when these are no longer used.

Currently, Nemo doesn't have a spill mechanism. This makes executors prone to memory problems such as OOM(Out Of Memory) or GC when task data is large. For example, handling skewed shuffle data in Nemo results in OOM and executor failure, as all data has to be handled in-memory.

We need to spill in-memory data to secondary storage when there are not enough memory in executorIn Stream processing, we have many methods, starting from the primitive checkpoint-and-replay to a more fancy version of reconfiguration and reinitiation of stream workloads. We aim to find a way to find the most effective and efficient way of reconfiguring stream workloads. Sub-issues are to be created later on.

In-memory caching and spilling are essential features in in-memory big data processing frameworks, and Nemo needs one.

• Identify and persist frequently used data and unpersist it when its usage ended
• Spill in-memory data to disk upon memory pressure

Nemo has a policy layer that allows powerful optimization with configurable runtime modules. In terms of caching, it is possible to identify frequently used data and decide to cache them in-memory ahead of execution, without user annotation.

Implementation would need:

• On policy layer, build compile-time pass that identify reused data and mark them as cached
• On runtime, design and implement caching mechanism that manages per-executor cached data and discard them when these are no longer used.

Currently, Nemo doesn't have a spill mechanism. This makes executors prone to memory problems such as OOM(Out Of Memory) or GC when task data is large. For example, handling skewed shuffle data in Nemo results in OOM and executor failure, as all data has to be handled in-memory.

We need to spill in-memory data to secondary storage when there are not enough memory in executor.

The load of streaming jobs usually fluctuate according to the input rate or operations (e.g., window). Supporting the automatic scaling could reduce the operational cost of running streaming applications, while minimizing the performance degradation that can be caused by the bursty loads. 

We can harness the cloud resources such as VMs and serverless frameworks to acquire computing resources on demand. To realize the automatic scaling, the following features should be implemented.

1) state migration: scaling jobs require moving tasks (or partitioning a task to multiple ones). In this situation, the internal state of the task should be serialized/deserialized. 

2) input/output rerouting: if a task is moved to a new worker, the input and output of the task should be redirected. 

3) dynamic Executor or Task creation/deletion: Executor}}s or {{Task can be dynamically created or deleted. 

4) scaling policy: a scaling policy that decides when and how to scale out/in should be implemented. 

In-memory caching and spilling are essential features in in-memory big data processing frameworks, and Nemo needs one.

• Identify and persist frequently used data and unpersist it when its usage ended
• Spill in-memory data to disk upon memory pressure