Move membership code to a separate gradle sub-project

Authors: Dan Smith, Bruce SchuchardtJens Deppe, Patrick Johnson

Status: Development Discussion

Superseded by: N/A

Related: N/A

Problem

Geode has its own custom membership system, which is responsible for discovering other members of the system and for detecting and removing failed members. We want to be able to exhaustively test that the membership system is working correctly in all circumstances. However this membership system is not cleanly separated from the rest of the geode code. This makes it difficult and expensive to test, because we have to create full geode members and connect them over a network in order to test any interaction between the membership system of two geode members.

By isolating the membership code such that it does not depend on the rest of the geode, we can test the membership system by itself. This will allow us to write much faster and more exhaustive tests of how multiple members interact.

Creating a well defined internal API for the membership system and hiding the internals from the rest of geode will also make it easier for geode developers to reason about what the membership system is and is not doing while writing and testing other components of geode.

Anti-Goals

currently has a hard dependency on Spring libraries as a result of gfsh being developed against the spring-shell library.

We would like to extract gfsh and related cli code into a separate module such that Geode can be run without gfsh as part of the core if so desired. This decoupling will have various benefits:

• Removal of Spring dependencies from Geode core.
• Clearer understanding of which components rely on Spring.
• Clear separation of various generations of admin frameworks. As the newer Cluster Configuration Management functionality reaches maturity, users will be able to easily chose which functionality they wish to include.

Anti-Goals

• It is not a goal to provide a standalone gfsh utility
• It is not a goal to give the membership system a public API for use by geode users. The membership system will have an internal  API and be usable in isolation, but we don't intend to advertise it as a publicly available component by itself.
• It is not a goal to create an SPI that allows different membership systems to be swapped in.
• It is not a goal to change the membership algorithms or wire protocol in any way.

Solution

We will create a new gradle subproject called geode-membership. All gfsh-cli. Ideally, all of the code in the org.apache.geode.distributedmanagement.internal.membership.gms cli package and related unit tests, will be moved out of geode-core and into geode-membershipgfsh-cli. geode-gfsh-core cli will have a dependency on geode-membership, and interact with membership through its API, defined below.geode-membership itself depends on serialization. Therefore we will also create a new gradle subproject for serialization, called geode-serialization. The full dependency tree will look something like below.

@startuml
"geode-core" --> geode-membership
"geode-core" --> geode-serialization
"geode-membership" --> geode-serialization
@enduml

...

core.

JMX management functionality will be unaffected except for the ability to execute gfsh commands via the MemberMXBean.processCommand method. 

Changes and Additions to Interfaces

This proposal does not add or remove any public API.

A new CacheService will be added called CommandProcessor which will enable the MemberMXBean.processCommand functionality.

The current mechanism to detect commands will not be changed, however the functionality will also be moved into the new module. Currently this involves scanning the classpath for classes implementing the org.springframework.shell.core.CommandMarker interface. Any user-created gfsh commands will need to depend on the new geode-gfsh-cli module.

Performance Impact

There is no expected performance impact

1. Move the dependencies to a separate subproject - serialization is moving to separate subproject so that membership can depend directly on that
2. Inject dependencies at runtime - dependencies such as gemfire stats can be injected into the membership system by providing interfaces such as MembershipStatistics (see below) that geode core must provide when creating the membership system.

Classes used to wrap the membership APIs or inject dependencies into membership will be put in the org.apache.geode.distributed.internal.membership.adapter package.

Delineating Interfaces

Not only does this RFC specify inter-subproject dependency constraints (between geode-core, geode-membership, geode-serialization), it goes further. Classes outside geode-membership may only see geode-membership interfaces defined in the (TBD) membership API package (see below). Other interfaces and classes defined by membership will be inaccessible outside of membership.

Gradle Subprojects First—Java Modules Someday

This RFC describes creation of (2) new Gradle subprojects (geode-membership, geode-serialization). It also specifies the the allowed (and disallowed) inter-subproject dependencies. Our intention is to lay the ground for eventual use of Java modules to enforce these dependency constraints.

geode-membership API

The geode-membership subproject will provide the following API to the rest of the system. We will enforce that other parts of the system can only interact with this API.

@startuml
interface MembershipBuilder {
 + static newMembershipBuilder()
----
 + set...()

}

interface Membership {
Core class of a running membership system.
+ getMembershipView(): MembershipView
}
interface Messenger {
+ send(Message)

}
interface MembershipView {
  + getMembers(): List<MemberIdentifier>
}
interface MemberIdentifier {
A single member of the system
}
interface MembershipListener {


}
interface MessageHandler {

}

interface Authenticator {
}


interface MembershipStatistics {

}
interface Config {

}
interface MembershipView {
}

MembershipBuilder --> Membership : creates
MembershipBuilder *-- Authenticator
MembershipBuilder *-- MembershipStatistics
MembershipBuilder *-- MessageHandler
MembershipBuilder *-- Config
MembershipBuilder *-- MembershipListener
Membership *-- Messenger
Membership *-- MembershipView
MembershipView *-- MemberIdentifier
@enduml

/** 
* Creates the membership system, given the provided configuration
*/

interface MembershipBuilder {
  static MembershipBuilder newMembershipBuilder()
  MembershipBuilder setConfig(Config)
  MembershipBuilder setAuthenticator(Authenticator)
  MembershipBuilder setMembershipListener(MembershipListener)
  MembershipBuilder setMessageHandler(MessageHandler)
  MembershipBuilder setMembershipStatistics(MembershipStatistics)
  MembershipBuilder setMemberIDFactory(MemberIdentifierFactory)
  create() : Membership
}

/**
* Core class of a running membership system.
*/
interface Membership {
  getMembershipView(): MembershipView
  getMessenger(): Messenger
  getLocalMember(): MemberIdentifier
  contactedBy(MemberIdentifier)
  isShunned(MemberIdentifier)
  close()
  isClosed()
}

/**
* API for sending messages to other members, using memberships messaging system
* Membership currently allows geode-core to send messages over it's UDP messaging system. 
* This interface provides that functionality.
*/
interface Messenger {
  send(Message)
  /** Get an object the represents what messages have been sent to to the given
   * MemberIdentifier. This is used in waitForMessengerState to wait for these messages
   * to arrive on the remote side           
   */  
  getMessengerState(MemberIdentifer): MessengerState
  waitForMessengerState(MessengerState)
}

/**
 * A Message that will be sent to the returned recipients.
 */
interface Message {
  Set<MemberIdentifier> getRecipients()
}


/**
* An object which represents the state of the sending side of a communication
* channel at a point in time. This object can be passed to the waitForMessengerState
* method on the receiving side to ensure all messages have been received on
* the receiving side
*/
interface MessengerState extends DataSerializableFixedID {
}/** 
* Interface used to create a MemberIdentifier from a set of properties
* This API is provided solely to allow geode-core to add additional
* properties to the MemberIdentifier that are not relevant to membership
* For example DurableClientAttributes. Membership will provide
* A default MemberIdentifier which just has the membership relevant attributes
*/

interface MemberIdentifierFactory {
  MemberIdentifier create(memberData)
}

/**
* A single member of the system. In practice, this will
* be implemented by InternalDistributedMember
*/
interface MemberIdentifier {
}

/**
* Provides the current members of the system
*/
interface MembershipView {
  List<MemberIdentifier> getMembers()
}

/**
* Receives notifications about changes to membership
*/
interface MembershipListener {
  memberJoined(MemberIdentifier)
  memberDeparted(MemberIdentifier)
  memberCrashed(MemberIdentifier)
  forceDisconnected(String reason)
}

/**
* Receives all messages sent from other members
*/
interface MessageHandler {
  processMessage(Message)
}

/**
* Interface used by membership to authenticate other members
*/
interface Authenticator {
/**
   * Authenticate peer member
   *
   * @return null if authentication succeed (including no authenticator case), otherwise, return
   *         failure message
   */
  String authenticate(MemberIdentifier member, Properties credentials);

  /**
   * Get credential object for the given GemFire distributed member.
   */
  Properties getCredentials(MemberIdentifier member);
}

/**
* Interface to notify statistics systems about membership changes
*/
interface MembershipStatistics {
  memberJoined()
  memberDeparted()
  ...
}

/**
* Primitive configuration options for membership
* - timeouts, etc.
*/
interface Config {
  getJoinTimeout()
  ...
}

geode-serialization API

The geode-serialization subproject will provide the following API to the rest of the system. We will enforce that other parts of the system can only interact with this API.

Internally Geode uses DataSerializableFixedID as the interface for most messages and data objects.  The membership system uses this exclusively.  We will separate the serialization framework for this interface and leave the rest (PDX, DataSerializable, etc.) in geode-core.

DataSerializableFixedID currently has the same signature for its toData and fromData methods but this will be changed to take another parameter, a serialization context.  The context provides access to serializers and information about the source/destination (currently the peer's version).

/** Use DSFIDSerializerFactory to create a serializer */
class DSFIDSerializerFactory {
  /** set an ObjectSerializer that should be used when invoking
   *  toData/fromData methods on objects.  This will typically
   *  defer serialization of DataSerializableFixedID objects to
   *  the DSFIDSerializer
   */
  DSFIDSerializerFactory setObjectSerializer(ObjectSerializer serializer);
  DSFIDSerializerFactory setObjectDeserializer(ObjectDeserializer deserializer);
  DSFIDSerializer create();
}

/** writes and reads DataSerializableFixedID instances.  Supports reading/writing of nulls */
interface DSFIDSerializer {
  /** retrieve the ObjectSerializer.  If none was given to the factory this will be
   *  a default serializer that can only handle DataSerializableFixedID objects */
  ObjectSerializer getObjectSerializer();
  ObjectDeserializer getObjectDeserializer();

  /** register a class with the serializer.  It must have a no-arg constructor */
  void registerDSFID(int dsfid, Class dsfidClass);

  SerializationContext createSerializationContext(DataOutput dataOutput);
  DeserializationContext createDeserializationContext(DataInput dataInput);

  /** write a DataSerializableFixedID to the given DataOutput.  This method should be used
   *  by extensions to DSFIDSerializer, like InternalDataSerializer, that want to handle
   *  more types in ObjectSerializer but defer DSFID serialization to the
   *  default implementation */
  void writeDSFID(DataSerializableFixedID o, int dsfid, DataOutput out) throws IOException;

  /** read a DataSerializableFixedID from the given DataInput.  This method uses constructors
   *  registered via registerDSFID() and should be used by extensions to DSFIDSerializer,
   *  like InternalDataSerializer, that want to handle more types in ObjectDeserializer but
   *  defer DSFID serialization to the default implementation */
  Object create(int dsfid, DataInput in) throws IOException, ClassNotFoundException;

  void invokeToData(Object ds, DataOutput out) throws IOException;
  void invokeFromData(Object ds, DataInput in) throws IOException, ClassNotFoundException;

  int readDSFIDHeader(DataInput dis) throws IOException;
  void writeDSFIDHeader(int dsfid, DataOutput out) throws IOException;
}

 /** provides context for toData serialization methods */
  interface SerializationContext {
    ObjectSerializer getSerializer();
    Version getSerializationVersion();
  }

  /** provides context for toData serialization methods */
  interface DeserializationContext {
    ObjectDeserializer getDeserializer();
    Version getSerializationVersion();
  }

  interface ObjectSerializer {
    public void writeObject(Object obj, DataOutput output) throws IOException;
    void invokeToData(Object ds, DataOutput out) throws IOException;
    ...
  }

  interface ObjectDeserializer {
    public Object readObject(DataInput input) throws IOException, ClassNotFoundException;
    void invokeFromData(Object ds, DataInput in) throws IOException, ClassNotFoundException;
    ...
  }

/** a new superclass of HeapDataOutputStream that separates the methods used with Buffers from
    a bunch of other methods concerning other functionality such as Streamables, PDX, etc. */
class BufferDataOutputStream implements DataOutputStream {
  // methods moved from HeapDataOutputStream unchanged
}

/** DataSerializableFixedID is modified to have more parameters in toData/fromData methods */
interface DataSerializableFixedID {
  public Version[] getSerializationVersions();
  public int getDSFID();
  public void toData(DataOutput output, SerializationContext context) throws IOException;
  public void fromData(DataInput input, DeserializationContext context) throws ClassNotFoundException, IOException;
}

A collection of other classes will be moved, virtually unchanged, into the new sub-project to fill out the API:

• ByteArrayDataInput
• DataSerializableFixedID will continue to hold all of Geode's DSFID constants until we find a better home for them.
• DSCODE - basic serialization constants
• DscodeHelper
• DSFIDNotFoundException
• SerializationVersions - interface for backward-compatible serialization
• ThreadLocalByteArrayCache
• VersionedDataInputStream
• VersionedDataOutputStream
• VersionedDataStream
• Version - Geode's serialization version class

Use DSFIDSerializerFactory to create a new serializer.  Set an ObjectSerializer/ObjectDeserializer if you want to provide serialization of other types of objects in readObject/writeObject.  Use DSFIDSerializer.registerClass() to register your classes so that the serializer knows how to instantiate them.

The new DSFID serializer sub-project will by and large not be composed of static methods as is InternalDataSerializer and its superclass DataSerializer.  InternalDataSerializer will hold an instance of the DSFIDSerializer and will register DSFID codes and constructors with it.  Instantiation will insert an ObjectSerializer and ObjectDeserializer into the DSFIDSerializer to provide users the ability to write PDX, DataSerializables and other serialization types supported by Geode but not (directly) by the DSFIDSerializer.  InternalDataSerializer and DataSerializer will defer serialization of DSFIDs to the DSFIDSerializer instance.  The geode-core DSFIDFactory will register all geode-core DataSerializableFixedID classes with the instance of the serializer held by InternalDataSerializer, as will other Geode subprojects like geode-wan.

While serialization of primitives and Strings can be performed using DataInput and DataOutput there are a number of static serialization methods currently in geode-core that will have to be shifted into the DSFIDSerializer (writeString/readString, writeHashmap/readHashmap, etc) to maintain compatibility with past releases.  Those methods shifted from DataSerializer will remain available in that class since it is a public API.   The intent is to shift these methods further into the ObjectSerializer/ObjectDeserializer APIs and stop using static serialization methods.

Since Version is being repackaged any rolling-upgrade test code that refers to it will no longer work when running with an older version of Geode.  We will provide methods in the distributed test framework VersionManager to make this less painful.  We will replace the use of Version.CURRENT_ORDINAL with VersionManager.getInstance().getCurrentVersionOrdinal() in these tests.  Other methods will be added to VersionManager as needed.

Changes and Additions to Public Interfaces

This proposal does not add or remove any public API.

Performance Impact

The intention of this proposal is to not change the performance of geode significantly.

Backwards Compatibility and Upgrade Path

These changes will maintain backwards compatibility with rolling upgrades. No messages will change as a result of this proposal.

This change does will introduce a new optional geode-membership and geodegfsh-serialization cli maven artifactsartifact. Users building a classpath with maven/gradle/etc. or using gfsh will not be impacted. However, if someone is manually launching a process with a hardcoded list of geode jars, they will need to add the new geode-membership  and geode-serialization jarswill need to explicitly include this artifact if they wish to use gfsh commands.

Prior Art

There have been multiple proposals to make geode more modular  - Geode Modularization Proposal (work in progress), Proposal for Geode Modularization. This proposal can be considered an incremental step in that direction.

...

.

...

FAQ

Q: Why not introduce a separate messaging module, and leave things like Messenger out of geode-membershipcreate a standalone gfsh utility?

A: We may eventually try to split messaging out of membership, but that is not part of the current proposal. The reason membership includes messaging in this proposal is that the product currently uses the messaging system bootstrapped by membership to send arbitrary messages. Membership requires a lower level messaging system (jgroups) but it uses that to provide a higher level messaging system to the rest of the system. The higher level messaging provided by membership includes additional features such as cluster wide encryption and ensuring that we only process messages from current membersA proof-of-concept, using a Spring Boot jar approach, for such a thing has been created. It requires bundling all of Geode dependencies, including war files, into a single jar resulting in a ~120MB artifact. This proposal does not preclude ever doing that, but it is not a current goal.

Errata

What are minor adjustments that had to be made to the proposal since it was approved?