Airavata Jupyter Platform Services

Background :-

There is a need to develop a UI application integrated with Apache Custos as the authentication service to host jupyter notebooks platform on Jetstream that can cater to handle multiple concurrent users at the same time with proper resource isolation, persist user states, and integrated to allow for uploading files from the local and add to the existing services of Apache Airavata as a scientific gateway.

Design/Description of work :-

Platform architecture - Airavata based infrastructure with functionality similar to colab.

1. UI :- Develop a UI and Integrate the jupyter notebook with cloud service(Jetstream) with a network filesystem(explored Dynamic Partitioning file system.) and along with authentication with Apache Custos. UI will authenticate users with the help of Custos services. The UI will have a login credential form and on successful authentication, a new user session will be created and user will be redirected to the Jupyter Notebook hosted on Kubernetes Jetstream.

2. Jupyter Hub :- JupyterHub is the best way to serve Jupyter notebook for multiple users. It is a multi-user Hub that spawns, manages, and proxies multiple instances of the single-user Jupyter notebook server. JupyterHub allows users to interact with a computing environment through a webpage. As most devices have access to a web browser, JupyterHub makes it is easy to provide and standardize the computing environment for a group of people.

1. Kubernetes Cluster :- For now, we will go for a simple Master Worker nodes setup for a Kubernetes cluster. We need to host the Jupyter Hub containers/pods on our Kubernetes cluster. Each user will be signed onto a container which will be hosting a jupyter notebooks on one of its ports and this will be load balanced with the help of the Kubernetes load balancer service. The UI will be using the K8s cluster through a load balancer which will itself have a number of PODS. Each POD will have a container, which can scale users using namespaces.

2. Dynamic Volume Provisioning :- Now, to persist the user data, the user work directories will be hosted on the network file system. On a successful connection, the users session pod will be connected to an appropriate volume with the help of Dynamic Volume Provisioning.

Results for the Apache community :-

Scalable colab based application allowing the use of jupyter notebook services in an isolated fashion based on the user authentication based on Custos on Jetstream cloud.

Deliverables and Timelines :-