Basics

The scheduler is a new core component that acts as an activation queue and activation scheduler.

It constructs and associates many other components such as QueueManager, MemoryQueueFactory, gRPC services, and so on.

Just like the other core components such as controllers or invokers, it is a main entry point for the Scala application and it initializes akka-http service which serves akka-grpc requests.

It has two akka-grpc interfaces, one is QueueService and the other is ActivationService.

QueueService interface is used by controllers to create queues for each action. ActivationService interface is used by invokers to fetch activations from queues.

Health states of schedulers are managed by an etcd using a lease. When a scheduler is failed, it cannot keep the lease alive and the lease will be expired.

Upon expiration of the lease, health data for the scheduler will be automatically removed from the etcd.

So other components can safely assume they send requests to healthy schedulers only.

Since queues are dynamically created and removed, the number of queues in each scheduler varies.

It is important to create a new queue to the scheduler with the least number of queues for load balancing.

So each scheduler will periodically publish its state(the number of queues) to an etcd and a new queue will be created in the proper scheduler based on this information.

References

• ETCD Lease: https://github.com/etcd-io/etcd/blob/master/Documentation/dev-guide/interacting_v3.md#grant-leases