Myriad Use Cases

The main uses cases for Myriad include:

Infrastructure Consolidation

Increasingly organizations are developing microservice-oriented applications using tools like Docker, and deploying them using schedulers like Mesos.  At the same time, these organizations are processing large amounts of data using tools like Hadoop scheduled by YARN.  

Deploying parallel infrastructures can be costly and inefficient. Both front-end web applications and analytics workloads tend to be bursty, having periods of heavy utilization followed by periods of light utilization.  This forces IT departments to size each cluster based on peak utilization, letting resources go underutilized much of the time.  Worse, these parallel infrastructures force users to constantly deal with moving data back and forth between clusters.  

With Myriad, these organizations can deploy, manage, and monitor a single cluster that supports both Docker-based microservices deployed via Mesos frameworks like Marathon, as well as YARN-based processing applications like MapReduce and Spark.  All applications are fully isolated using Linux containers, ensuring that analytics workloads don’t interfere with operational applications or vise versa.  With fine-grained scaling, analytics workloads can consume large amounts of available resources when they need them, releasing them back to the shared pool when they are not.  In addition, distributed, shared data services can be provisioned on the shared cluster, eliminating data movement between applications and analytics.

Hadoop Multi-Tenancy

As organizations become more reliant on data processing technologies like Hadoop, it is common to encounter “cluster-sprawl” situations, where several different clusters are deployed to support different business groups or different lifecycle stages, such as dev, test, and prod, each running a different version.  Each new cluster requires new servers to be purchased and maintained, and large amounts of data to be copied over to support the new use case.  

Using Myriad, these organizations can save money and increase agility by provisioning multiple logical Hadoop clusters on a single physical Mesos cluster, with either shared or dedicated data services.  Each logical cluster can be tailored to the end user, with a custom configuration and security policy, running it’s own version, with either static or dynamic resources allocated to it.

In a multi-tenant environment, this model means that a shared pool of resources can be shared among many data processing frameworks, with each capable of allocating additional resources when needed, and release them when not.  The top-level Mesos scheduler will ensure fairness in the case that multiple frameworks are competing for resources.

In case of a version migration, this model means that two logical Hadoop clusters of different versions can be deployed side by side on top of the same shared data.  Users can migrate workloads from the old version to new gradually, adding resources to the new cluster and taking them away from the old.  After all workloads are moved over they can decommission the old cluster.