Apache Airavata

NOTE: Don't update the Title of the page

GSoC PROJECT TITLE:

Design and implementation of lifecycle management of the CyberShuttle agents

Mentors: Suresh Marru, Dimuthu Wannipurage


Original Proposal Summary:

Apache Airavata MFT is a distributed high-performance, multi-protocol data transfer engine to orchestrate data movement and operations across most cloud and on-premises storages. Agent is the core of this distributed system which does the heavy lifting of transfer jobs. Right now, agents have to be manually deployed and configured to start the transfer. This is a problem for researchers and people who need to transfer data without having much technical skills or time. The proposed solution is to automate the process of deploying and lifecycle management of the agents when a transfer job is submitted from an edge machine. Upon requirements evaluation, along with my mentor I came up with an updated proposal including the cybershuttle agent to MFT agent.

Updated Proposal Summary:

Updated proposal includes creating a desktop application and cybershuttle server to communicate with each other. This will guide cybershuttle agent and MFT agent from cybershuttle server to perform execution of jobs and data transfer, this would also maintain the lifecycle of cybershuttle agent and mft agents.


Architecture of the Solution:

The architecture of this solution contains below major components with technologies used:

  • Desktop Application: Where a user could login to check their available code bases and agents to deploy. One can launch and stop the agents from this interface with a single click.
    • Technologies: javascript, html, css, electronJS, Vuejs, gRPC, protobuf, pinia, vuetify
  • Server: It plays the central role in the architecture right from authenticating users, providing available data, setting the jobs to be executed/removed in queue, create auth keys for orchestration agents etc.
    • Technologies: java, springboot, gRPC, protobuf, maven, consul sdk
  • Consul: Consul comes into play in service discovery and distributed KV store which is used for all communications between server and orchestration agent
  • Orchestration Agent: It is responsible for orchestrating the life cycle of agents in the system. I connects server/consul to local system and also executes docker images, java runnables, other executable files using Nomad (workload orchestration service)
    • Technologies: java, springboot, maven, consul sdk, nomad sdk
  • Nomad: It is the workhorse which starts and stops all jobs which is controlled by orchestration agent.

Workflow:

The total workflow of finished product is as follows:

  • User gets logged in to desktop to access all the app and agents available to deploy which are retrieved from server.
  • With a single click they will be able to launch app or stop app.
  • On executing launch app, server puts the executable in secure consul KV store path
  • Orchestration agent which is live, waits for any changes in the above KV path. Once it finds a executable, it will trigger the Nomad to start job and upon successful start it updates job as processing.
  • Server reads the processing path and updates user that app is running
  • Similarly user will be able to stop the previously launched apps which works in same work flow as launching an app except that the Nomad will stop the job now instead

Results:

The project is successfully tested with automation of all the workflow of agent management from launching to stopping which is always a click away. This could potentially provide time savings for the user in the magnitudes of hours to days depending on the complexity of the app/agent/code setup, prerequisites, ability to run etc.

Deliverables with respective Pull Requests:

Desktop application:

Cybershuttle app server:

Orchestration Agent:

Communication threads:

Most of the communication with mentors was over slack, video conference and personal chat. Though a few milestones in the process are documented over community threads as follows:

Future development opportunities:

  • We can use the desktop app code and build a website which would decouple the placement of orchestration agent and user interface making the solution highly scalable
  • Complete the login/authentications of the server side of cybershuttle
  • Implement datastore at server end to store user related properties, nomad jsons etc
  • Add features to deploy agents at desired VMs and not just local
  • Automate the process of generating json files used by Nomad server to execute jobs
  • Polish the frontend of the desktop/web app to accommodate multiple views and streamline the visuals

Thank you for going through the project. My information is as below.

BIOGRAPHICAL INFORMATION:

Name: Praneeth Kumar Chityala

Email: praneethchityala44@gmail.com, pkchitya@iu.edu

Education :

August 2021 - May 2023

Indiana University Bloomington
MS in Computer Science - 3.97/4

August 2012 - May 2016

National Institute of Technology, Warangal (NITW)

B.Tech

Programming Languages: Java, Python, Javascript, SQL, HTML5

Technologies/Frameworks: Spring, SpringBoot, GCP, AWS, Microservices, Docker, Nomad, Consul, Kubernetes, React, Vuejs, electonJS, gRPC, MongoDB, PostgreSQL, MySQL, Heroku, Jenkins, Maven, JUnit, Mockito, Git, YourKit, Log4j

  • No labels

For an overview of Apache Airavata visit the Website - https://airavata.apache.org/