Design Doc - Mesos Modules

Adding a first class primitive, abstraction and process for dynamic library writing and loading can make it easier to extend inner workings of Mesos. Making it possible to have dynamic loadable resource allocators, isolators, containerizes, authenticators and much more.
This could be a powerful feature, as we get even more extensible and flexible ways of setting up Mesos - but also for isolating dependencies and complexity in external libraries and to ease experimentation with new features.
For example, imagine a loadable allocators which contains a VM (Lua, Python, ...) which makes it possible to try out new allocator algorithms without forcing those dependencies into the project.

Definitions

The API from the point of view of a module writer

#include <mesos/module.hpp> // module system API
#include <calculator.hpp> // module role type declaration is in here

MESOS_MODULE_LIBRARY() // declares the module library (and its versions)

class ExampleCalculator: public Calculator
{
public:
  // An example function that the module implements.
  virtual int compute(char a, long b)

MESOS_MODULE(Calculator, example) // declares the module

Additional compatibility checks

MESOS_IS_MODULE_COMPATIBLE(example)

Here, instead of simply returning true, the module could also perform its own checks, and return false under certain conditions. In the latter case the module will not be admitted, no matter what the results of any other checks by Mesos indicate. The module’s own checks are open-ended. In particular, they may include queries about other loaded libraries and modules and their respective versions. We will later provide an API for that.

The API from the point of view of a Mesos implementor

Only modules with pre-declared roles and binding sites can be loaded by Mesos. There is no means of dynamic discovery or binding of extra roles. To declare a loadable module, a Mesos developer needs to specify an abstract class with at least one virtual method. Here is an example that matches the module code above.

class Calculator {
public:
  Calculator() {}
  virtual ~Calculator() {}

  virtual int compute(char a, long b) = 0;
};

That’s all. To employ a specific instantiation of this module role/type, you can write something like this.

#include <module/module_manager.hpp>

Try<Calculator*> module = mesos::ModuleManager::loadModule<Calculator>("example");
if (module.isError())

}

Calculator* calculator = module.get();

And then you can already use the module.

int n = calculator->compute(‘A’, 10);

Module Libraries and Module Versioning

• Modules can be used by all constituents of Mesos, in particular slaves as well as masters. Different sets of modules may or may not apply to either. Each Mesos version of a module client program defines its own finite set of module bindings, i.e. module injection points.
• Modules come packed in dynamic libraries (dynlibs).
• An installed module is a C++ object created by a call into a module implementation in a dynlib. So far there is no well-defined set of features that work or don’t work. However, we use a dynamic_cast with RTTI to verify compatibility of a module with binding into Mesos. At a minimum, this restricts Module types to abstract classes with at least one virtual function. In case RTTI turns out not to be water-proof, we will eventually have to enumerate features that do work. This still seems less tedious than breaking our very simple API up into a much longer series of C-only constructs.
• A compiler version and flag check may become necessary.
• If really needed, an hourglass interface (http://cppcon2014.sched.org/event/e659d8c088904f7a1540524b196afbe9#.VBiSd0sjMeE,https://github.com/CppCon/CppCon2014/raw/master/Presentations/Hourglass%20Interfaces%20for%20C%2B%2B%20APIs%20-%20Stefanus%20Du%20Toit%20-%20CppCon%202014/Hourglass%20Interfaces%20for%20C%2B%2B%20APIs%20-%20Stefanus%20Du%20Toit%20-%20CppCon%202014.pdf) can be added later, with an extra Mesos Module System version.
• Module installation has these phases: 1) Dynlib loading, 2) verification including dependency checking, 3) instantiation, 4) binding (assignment to an l-value that gets used somehow).
• All modules are named in one command line flag, which gets parsed early. After all dynlibs have been loaded, all verification is run. Instantiation happens later, in various places in Mesos, wherever modules are involved. This is driven by other command line flags which then reference identifiers given by the module flag. At first we only need to support a very simple naming scheme where the module name is used directly and it is expected that there are never conflicting module names. Example:
  slave --modules=”/root/path1:module1,path2:module2” --allocator=”module1” --auth=”module2”
• There can be multiple modules in a given dynlib. This allows shared implementation and data elements and potential packaging convenience.
• Each dynlib must indicate what version of MMS it is built for. Then future MMSs can determine whether to use a given pre-existing dynlib or not. Conversely, an older Mesos/MMS can determine that a dynlib relies on a later MMS version. This also serves as a handshake between Mesos and any arbitrary dynlib to ensure it is dealing with a Mesos module dynlib at all.
• Once it is thus established that Mesos is dealing with a proper module dynlib with a compatible version, the dynlib is trusted to behave cooperatively and non-maliciously.
• Each dynlib indicates what minimum Mesos version it is compatible with.
• Each module indicates its “role”, e.g. Isolator, Allocator, Authenticator.
• There can be multiple modules for the same role in the same dynlib, especially isolator modules.
• User code does not need to instantiate and bind all modules in a dynlib. It can cherry-pick.

Limitations and Simplifications

• If you build a dynlib against a certain Mesos version, it will not be allowed to be used with older Mesos versions. So for simplicity we bake the respective Mesos version into the dynlibs.
• For each Role, a corresponding mesos version is kept in a table. If the role changes in a non-compatible way, its version must be bumped by the responsible Mesos developer to match the current mesos version.
• To use the module API, The module writer only faces a header file. Nothing needs to be linked in for this purpose. However, to implement the specific payload features of any given module, it may have to reference any number of other parts of Mesos, including having to link against those.
• We consider corralling all Mesos features that are used by modules into a facade layer. Thus backward compatibility could be maintained with more ease, albeit at the cost of erecting the facade.

Things that we may want to support in the future and that should not be impeded by the first MMS implementation

• Modules should be able to express and check interdependencies and mutual compatibility.
• We do not check module versions beyond their type for now. In addition we may want to give them a version number that gets bumped even if the type remains the same, if the protocol for using the module changed.
• Somehow organize automatic module deletion. So far it is up to client code in Mesos to explicitly call the destructor.