Page History

The main uses cases for Myriad include:

* Containerized enterprise environments
* Web-scale infrastructure consolidation
* Hadoop service provider

Containerized Enterprise Environment

This use case is for environments that are either running separate clusters for Mesos and YARN or are statically partitioning their servers to run both simultaneously.

Key characteristics include:

• Diverse Docker-ized applications on Mesos. 
• Hadoop jobs or applications run on YARN. 
• Workload prioritization by YARN FairScheduler. 
• File access is granted based on the user that submitted a YARN application. 
• Dynamic on-demand allocation of resources to all user application running across both frameworks which ensures full utilization of resources without administrators being a critical path in application provisioning. 
• Fair allocation of resource between Myriad and other frameworks. 
• Isolation where applications such as YARN AMs or Mesos frameworks are isolated from other applications in terms of cpu, memory, and disk space. 
• Shared single data layer. 
• Security where a user running a job is only allowed access to that data and only authorized user can provision Myriad-RM instances. 
• Multiple YARN clusters on the same infrastructure, for example, to host dev, test, and production clusters. Typically tens or hundreds of nodes. 
• Marathon implementation for integration with the PaaS frameworks on Mesos. 
• Spark and Storm application compatibility with both YARN and Mesos. 
• Monitoring through Marathon health checks, Ganglia, and Job History interfaces such as MapReduce and Spark HistoryServer.

Web-scale Infrastructure Consolidation

This use case is similar to the containerized enterprise environment.

Key characteristics include:

• Customized engineering of the data center infrastructure where early stage software may be
  extended, maintained, and integrated with custom tools.
• Very large cluster need to be supported. Typically, thousands of nodes.
• Small number of applications where the scale for each application is very large.
  For example, hundreds of servers performing web front-end tasks.
• Single tenancy supporting a single product.
• Minimal security where security is often enforced by IP firewalls rather than at the application level.

Hadoop Service Provider

This use case is for multi-tenancy environments where each tenant is a different company. Thus, isolation and security is extremely critical. Because of application diversity, is is not likely that non-Hadoop tasks will be run on Mesos. In this case, Mesos is only used to negotiate with per-tenant YARN clusters for resource allocation.

Key characteristics include:

Infrastructure Consolidation

Increasingly organizations are developing microservice-oriented applications using tools like Docker, and deploying them using schedulers like Mesos.  At the same time, these organizations are processing large amounts of data using tools like Hadoop scheduled by YARN.  

Deploying parallel infrastructures can be costly and inefficient. Both front-end web applications and analytics workloads tend to be bursty, having periods of heavy utilization followed by periods of light utilization.  This forces IT departments to size each cluster based on peak utilization, letting resources go underutilized much of the time.  Worse, these parallel infrastructures force users to constantly deal with moving data back and forth between clusters.  

With Myriad, these organizations can deploy, manage, and monitor a single cluster that supports both Docker-based microservices deployed via Mesos frameworks like Marathon, as well as YARN-based processing applications like MapReduce and Spark.  All applications are fully isolated using Linux containers, ensuring that analytics workloads don’t interfere with operational applications or vise versa.  With fine-grained scaling, analytics workloads can consume large amounts of available resources when they need them, releasing them back to the shared pool when they are not.  In addition, distributed, shared data services can be provisioned on the shared cluster, eliminating data movement between applications and analytics.

Hadoop Multi-Tenancy

As organizations become more reliant on data processing technologies like Hadoop, it is common to encounter “cluster-sprawl” situations, where several different clusters are deployed to support different business groups or different lifecycle stages, such as dev, test, and prod, each running a different version.  Each new cluster requires new servers to be purchased and maintained, and large amounts of data to be copied over to support the new use case.  

Using Myriad, these organizations can save money and increase agility by provisioning multiple logical Hadoop clusters on a single physical Mesos cluster, with either shared or dedicated data services.  Each logical cluster can be tailored to the end user, with a custom configuration and security policy, running it’s own version, with either static or dynamic resources allocated to it.

In a multi-tenant environment, this model means that a shared pool of resources can be shared among many data processing frameworks, with each capable of allocating additional resources when needed, and release them when not.  The top-level Mesos scheduler will ensure fairness in the case that multiple frameworks are competing for resources.

In case of a version migration, this model means that two logical Hadoop clusters of different versions can be deployed side by side on top of the same shared data.  Users can migrate workloads from the old version to new gradually, adding resources to the new cluster and taking them away from the old.  After all workloads are moved over they can decommission the old cluster.

...