Page History

...

Infrastructure Consolidation

Increasingly, organizations are developing microservice-oriented applications using tools like Docker , and deploying them using schedulers like Mesos.  At At the same time, these organizations are processing large amounts of data using tools like Hadoop scheduled by YARN.  

Deploying parallel infrastructures can be costly and inefficient. Both front-end web applications and analytics workloads tend to be bursty, having periods of heavy utilization followed by periods of light utilization.  This This forces IT departments to size each cluster based on peak utilization, letting resources go underutilized under-utilized much of the time.  WorseWorse, these parallel infrastructures force users to constantly deal with moving data back and forth between clusters.  

With Myriad, these organizations can deploy, manage, and monitor a single cluster that supports both Docker-based microservices deployed via Mesos frameworks like Marathon , as well as and YARN-based processing applications like MapReduce and Spark.  All All applications are fully isolated using Linux containers, ensuring that analytics workloads don’t interfere with operational applications or vise vice versa.  With With fine-grained scaling, analytics workloads can consume large amounts of available resources when they need them, releasing them back to the shared pool when they are not.  In In addition, distributed, shared data services can be provisioned on the shared cluster, eliminating data movement between applications and analytics.

...

As organizations become more reliant on data processing technologies like Hadoop, it is common to encounter “cluster-sprawl” situations, where several different clusters are deployed to support different business groups or different lifecycle stages , (such as devdevelopment, test, and prod, each production) where each group or lifecycle stage is running a different version.  Each Each new cluster requires new servers to be purchased and maintained, and large amounts of data to be copied over to support the new use casecases.  

Using Myriad, these organizations can save money and increase agility by provisioning multiple logical Hadoop clusters on a single physical Mesos cluster , with either shared or dedicated data services.  Each Each logical cluster can be tailored to the end user, with a custom configuration and security policy, while running it’s own a specific version, and with either static or dynamic resources allocated to it.

In a multi-tenant environment, this model means that a shared pool of resources can be shared among many data processing frameworks, with each capable of allocating additional resources when needed , and release releasing them when not.  The The top-level Mesos scheduler will ensure ensures fairness in the case that multiple frameworks are competing for resources.

In case of a version migration (for example, upgrading only one of two Hadoop clusters), this model means that two logical Hadoop clusters of different versions can be deployed side by side on top of the same shared data.  Users Users can migrate workloads from the old version versions to new versions gradually, adding add resources to the new cluster and taking them away , and take resources away from the old cluster.  After After all workloads are moved over they can decommission , the old cluster can be decommissioned.