Move membership code to a separate gradle sub-project

To be Reviewed By:

Authors:

Status: Draft | Discussion | Active | Dropped | Superseded

Superseded by: N/A

Related: N/A

Problem

Geode has its own custom membership system, which is responsible for discovering other members of the system, and for detecting and removing failed members. We want to be able to exhaustively test that the membership system is working correctly in all circumstances, However, this membership system is not cleanly separated from the rest of the geode code. This makes it difficult and expensive to test, because we have to create full geode members and connect them over a network in order to test any interaction between the membership system of two geode members.

By isolating the membership code such that it does not depend on the rest of the geode, we can test the membership system by itself. This will allow us to write much faster and more exhaustive tests of how multiple members interact.

Creating a well defined internal API for the membership system and hiding the internals from the rest of geode will also make it easier for geode developers to reason about what the membership system is and is not doing while writing and testing other components of geode.

Anti-Goals

• It is not a goal to give the membership system a public API for use by geode users. The membership system will have an internal  API and be usable in isolation, but we don't intend to advertise it as a publicly available component by itself.
• It is not a goal to create an SPI that allows different membership systems to be swapped in.
• It is not a goal to change the membership algorithms or wire protocol in any way.

Solution

Describe your solution and how it’s going to solve the problem. This is likely the largest section of your proposal and might even include some high-level diagrams if you are proposing code changes. While all important aspects need to be covered, also keep in mind that shorter documents are more likely to be read.

We will create a new gradle subproject called geode-membership. All of the code in the org.apache.geode.distributed.internal.membership.gms and related unit tests will be moved out of geode-core and into geode-membership. geode-core will have a dependency on geode-membership, and interact with membership through its API, defined below.

geode-membership itself depends on serialization. Therefore we will also create a new gradle subproject for serialization, called geode-serialization. The full dependency tree will look something like below.

Breaking circular dependencies

The membership code currently has a number of dependencies on other classes within geode core. We will break these dependencies in essentially three different ways

1. Move the dependencies to a separate subproject - serialization is moving to separate subproject so that membership can depend directly on that
2. Inject dependencies at runtime - dependencies such as gemfire stats can be injected into the membership system by providing interfaces such as GMSStatsListener (see below) that geode core must provide when creating the membership system.
3. Wrap the membership APIs - some functionality is membership is tied to the rest of geode-core - for example DistributedMember is a public class that contains membership information as well as other, non membership related information. In geode-core, we can wrap the GMSMember in an appropriate class for use in the rest of geode-core.

Classes used to wrap the membership APIs or inject dependencies into membership will be put in the org.apache.geode.distributed.internal.membership.adapter package.

TODO

geode-membership API

The geode-membership subproject will provide the following API to the rest of the system. We will enforce that other parts of the system can only interact with this API.

/** 
* Creates the membership system, given the provided configuration
*/

interface MembershipManagerFactory {
  static MembershipManagerFactory newMembershipManagerFactory()
  setConfig(Config)
  setAuthenticator(Authenticator)
  setMembershipListener(MembershipListener)
  setMessageHandler(MessageHandler)
  setStatsListener(StatsListener)
  setMemberFactory(MemberIDFactory)
  create() : MembershipManager
}

/**
* Core class of a running membership system.
*/
interface MembershipManager {
  getMembershipView(): MembershipView
  getMessenger(): Messenger
  getLocalMember(): MemberID
  contactedBy(MemberID)
  isShunned(MemberID)
  close()
  isClosed()
}

/**
* API for sending messages to other members, using memberships messaging system
* Membership currently allows geode-core to send messages over it's UDP messaging system. 
* This interface provides that functionality.
*/
interface Messenger {
  send(Message)
  getMessageState(): Object
  waitForMessageState(Object)
}


/** 
* Interface used to create a MemberID from a set of properties
* This API is provided solely to allow geode-core to add additional
* properties to the MemberID that are not relevant to membership
* For example DurableClientAttributes. Membership will provide
* A default MemberID which just has the membership relevant attributes

interface MemberIDFactory {
  MemberID create(host, port, uuid, ...)
}

/**
* A single member of the system. In practice, this will
* be implemented by InternalDistributedMember
*/
interface MemberID {
}
/**
* Provides the current members of the system
*/
interface MembershipView {
  List<MemberID> getMembers()
}

/**
* Receives notifications about changes to membership
*/
interface GMSMembershipListener {
  memberJoined(GMSMember)
  memberDeparted(GMSMember)
  forceDisconnected()
}

/**
* Receives all messages sent from other members
*/
interface MessageHandler {
  processMessage(Message)
}

/**
* Interface used by membership to authenticate other members
*/
interface Authenticator {
}

/**
* Interface to notify statistics systems about membership changes
*/
interface StatsListener {
  memberJoined()
  memberDeparted()
}

/**
* Primitive configuration options for membership
* - timeouts, etc.
*/
interface Config {
  getJoinTimeout()
  ...
}

TODO

geode-serialization API (DRAFT)

The geode-serialization subproject will provide the following API to the rest of the system. We will enforce that other parts of the system can only interact with this API.

Internally Geode uses DataSerializableFixedID as the interface for most messages and data objects.  The membership system uses this exclusively.  We will separate the serialization framework for this interface and leave the rest (PDX, DataSerializable, etc.) in geode-core.

DataSerializableFixedID currently has the same signature for its toData and fromData methods but this will be changed to take another parameter, a serialization context.  The context provides access to serializers and information about the source/destination (currently the peer's version).

/** writes and reads DataSerializableFixedID instances.  Supports reading/writing of nulls */
interface DSFIDSerializer {
  void writeDSFID(DataSerializableFixedID, DataOutput output) throws IOException;
  DataSerializableFixedID readDSFID(DataInput input) throws IOException, ClassNotFoundException;
}

/** provides context for toData and fromData serialization methods */
interface SerializationContext {
  DSFIDSerializer getDSFIDSerializer();
  SerializerPlugin getDataSerializer();  // provides InternalDataSerializer readObject/writeObject
  SerializationVersion getSerializationVersion();
}

/** new superclass of Geode's Version class holding an ordinal */
class SerializationVersion implements Comparable<SerializationVersion> {
  SerializationVersion getCurrentVersion()
  short ordinal();
}

/** a new superclass of HeapDataOutputStream that separates the methods used with Buffers from
    a bunch of other methods concerning other functionality such as Streamables, PDX, etc. */
class BufferDataOutputStream {
  // methods moved from HeapDataOutputStream unchanged
}

/** DataSerializableFixedID is modified to have more parameters in toData/fromData methods */
class DataSerializableFixedID {
  public SerializationVersion[] getSerializationVersions();
  public int getDSFID();
  public void toData(DataOutput output, SerializationContext context) throws IOException;
  public void fromData(DataInput input, SerializationContext context) throws ClassNotFoundException, IOException;
}

A collection of other classes will be moved, virtually unchanged, into the new sub-project to fill out the API:

• ByteArrayDataInput
• DataSerializableFixedID will continue to hold all of Geode's DSFID constants until we find a better home for them.
• DSCODE - basic serialization constants
• DscodeHelper
• DSFIDNotFoundException
• SerializationVersions - interface for backward-compatible serialization
• ThreadLocalByteArrayCache
• VersionedDataInputStream
• VersionedDataOutputStream
• VersionedDataStream

The new DSFID serializer sub-project will by and large not be composed of static methods as is InternalDataSerializer and its superclass DataSerializer.  InternalDataSerializer will hold an instance of the DSFIDSerializer and will register DSFID codes and constructors with it.  Instantiation will insert a SerializerPlugin into the DSFIDSerializer to provide users of the serializer the ability to write PDX, DataSerializables and other interfaces supported by Geode but not (directly) by the DSFIDSerializer.

Changes and Additions to Public Interfaces

This proposal does not add or remove any public API.

Performance Impact

The intention of this proposal is to not change the performance of geode significantly.

Backwards Compatibility and Upgrade Path

These changes will maintain backwards compatibility with rolling upgrades. No messages will change as a result of this proposal (VERIFY THIS).

This change does introduce a new geode-membership maven artifact. Users building a classpath with maven/gradle/etc. or using gfsh will not be impacted. However, if someone is manually launching a process with a hardcoded list of geode jars, they will need to add the new geode-membership jar.

Prior Art

There have been multiple proposals to make geode more modular  - Geode Modularization Proposal (work in progress), Proposal for Geode Modularization. This proposal can be considered an incremental step in that direction.

As an alternative, we could isolate the membership code without creating a new gradle subproject and geode-membership jar file. However without enforcing what the membership code can and cannot depend on, it is likely that the membership code will not stay isolated as different developers work on the code.

FAQ

Q: Why not introduce a separate messenging module, and leave things like Messenger out of geode-membership?

A: We may eventually try to split messenging out of membership, but that is not part of the current proposal. The reason membership includes messaging in this proposal is that the product currently uses the messaging system bootstrapped by membership to send arbitrary messages. Membership requires a lower level messaging system (jgroups) but it uses that to provide a higher level messaging system to the rest of the system. The higher level messaging provided by membership includes additional features such as cluster wide encryption and ensuring that we only process messages from current members.
Answers to questions you’ve commonly been asked after requesting comments for this proposal.

Errata

What are minor adjustments that had to be made to the proposal since it was approved?