Introduction

I introduced a few new components.

1) Scheduler

Scheduler is a new components it include many critical features.

It is in charge of two major features, 1. queueing and routing activations 2. decide whether to add more containers based on the loads.

It has sub component `Queue`. The role of the queue is similar to the Kafka topic. A dedicated queue is created for each action.

Each queue will receive activation messages from the Kafka for a given action and send them to the ContainerProxy in respond to requests from it.

2) etcd

etcd is a distributed reliable key-value store. etcd is mostly used for transactional support and information sharing among components.

3) Akka-grpc

Akka-grpc is introduced to replace Kafka based execution path.

In this version, I could not fully rule out it due to heavy dependencies. My final objective is to exclude it at least from the critical path.

It is also being used to send queue creation request to the scheduler.

4) Akka-cluster

Akka-cluster is used for schedulers to communicate with each other.

Akka-grpc is required to define a grpc message, there are some advantages in akka-cluster when it is being used for simple inter-cluster communication.

Entire system is comprised of three flows, 1. queue creation flow, 2. container creation flow, 3. activation flow.

1. Queue Creation Flow

① Once a new request comes, there is no queue for the action, a PoolBalancer starts queue creation flow by randomly choosing a scheduler and calling QueueCreationAPI(Akka-grpc).

② If a scheduler receives QueueCreation request, it checks whether there is already queues for the action. If there are already three queues(1 leader + 2 followers), it responds with Success. If there are lesser than 3 queues, it first tries ETCD transaction, and if succeeded, write the information in ETCD to describe queue creation is in progress.

③ During this step, other schedulers can also receives the queue creation requests then the transaction will be failed and they will respond with Success because queue creation is already in progress.

④ A scheduler who achieves the transaction, select the three schedulers with the least number of queues and send queue creation request to QueueManagers in each scheduler via Akka-cluster(remoting). This implies all schedulers should be clustered using Akka-cluster.

⑤ Once a QueueManager receives the request, it will create a queue and send a Start message to it and wait for the Ack.

⑥ Each created queue tries to become a leader and only one of them becomes a leader. The leader writes the endpoint of the scheduler in ETCD for activation clients(ContainerProxy), and responds with Ack[Leader|Follower] to QueueManager. All followers watch the leader key and wait for the leadership change to become a leader in case of the leader failure.

⑦ Once a QueueManager receives Ack from the leader queue, it creates a MessageConsumer with the ActorRef of the queue then the MessageConsumer fetches activation messages from Kafka and send them to the leader queue.

In the future, PoolBalancer would send activation requests to the scheduler directly.

2. Container Creation Flow

① When a new queue is created, it immediately sends a container creation request to the ContainerManager because there is no container yet.

② ContainerManager selects one of healthy invokers and sends a ContainerCreation request via invokerN Kafka topic.

③ Invoker receives the ContainerCreation Message via Kafka.

④ Invoker create a ContainerProxy for the give action.

⑤ When container creation is finished or failed, the invoker sends the ack(Success | Rescheduling) to the ContainerScheduler via creationAck topic. The newly created ContainerProxy accesses to ETCD and figures out the endpoint of the leader queue and initialize the Akka-grpc client to fetch the activation from the leader queue.

⑥ Once the ContainerManager receives the ack message. It finishes the ContainerCreation in the normal case, and if it receives Rescheduling ack, it chooses another invoker and send ContainerCreation to it.

3. Activation Flow

① While the queue and the containers are being created, activation flows the system like above. PoolBalancer just sends the request to the action topic in Kafka.

② The MessageConsumer who subscribes the topic receives the request and deserialize it.

③ The message is sent to the leader queue.

④ ContainerProxy requests activation messages via  FetchActivation API(Akka-grpc).

⑤ This request is delegated to the QueueManager. The QueueManager finds the queue for the action, forwards the ActivationRequest and receives the ActivationResponse. Finally the ActivationResponse is passed to the ContainerProxy.

⑥ Once an activation is over, the ContainerProxy sends the results to PoolBalancer via completedM Kafka topic. After then, ContainerProxy autonomously calls the fetchActivation API again whenever the invocation finished so that we can maximize the performance of it and take advantage of back-pressure.