Introduction

I introduced a few new components.

1) Scheduler

The scheduler is a new component it includes many critical features.

...

Each queue will receive activation messages from the Kafka for a given action and send them to the ContainerProxy in response to requests from it.

2) ETCD

ETCD is a distributed reliable key-value store. etcd is mostly used for transactional support and information sharing among components.

3) Akka-grpc

Akka-grpc is introduced to replace Kafka based execution path.

In this version, I could not fully rule out it due to heavy dependencies. My final objective is to exclude it at least from the critical path.

4) Akka-remote

Akka-cluster is used for schedulers to communicate with each other.

...

The entire system is comprised of three flows, 1. queue creation flow, 2. container creation flow, 3. activation flow.

1. Queue Creation Flow

① Once a new request comes, there is no queue for the action, a PoolBalancer starts queue creation flow by randomly choosing a scheduler and send a CreateQueue request via akka-remote.

...

⑤ A DataManagementService receives the data it stores it in ETCD. Since PoolBalancers are watching ETCD events for queue endpoints. So once a queue is created, it does not try to create a queue for the given action anymore and send activation messages to the scheduler which has the queue.

2. Container Creation Flow

① When a new queue is created, it immediately sends a container creation request to the ContainerManager because there is no container yet.

...

⑦ At the same time, once a new container is created, it stores its data to ETCD. This data is used while scheduling containers.

3. Activation Flow

① While the queue and the containers are being created, activation flows the system like above. PoolBalancer just sends the request to the scheduler topic in Kafka. The MessageConsumer who subscribes to the topic receives the request and deserializes it. The message is sent to the leader queue. If there is no queue for the given activation message, it will forward it to the proper scheduler by fetching the queue endpoint information from ETCD.

...