...

• We implemented our own non-standard encryption and key exchange system on top of jgroups UDP messaging in order to ensure that all P2P messages are encrypted. See Secure UDP Communication in Geode. This adds complexity to the configuration by requiring a users to also set a separate UDP encryption property, and it also adds the risk of security and functional issues with our implementation. We recently have been discussing discussed deprecating this property on the mailing list because of it's functional issues - See this thread.
• JGroups does not support rolling upgrades which makes it more difficult to upgrade JGroups.

...

• Set us up for removing JGroups as dependency in future versions by moving to a protocol that does not require JGroups
• Support rolling upgrades from the old JGroups protocol to the new TCP-based protocol
• Use the existing SSL settings to control how membership messages are encrypted
• Be as reliable in the face of networking failures as the previous protocol

Anti-Goals

...

• It is not a goal to replace the existing and separate peer-to-peer TCP messaging system that is used for cache operations. The new messaging system is initially targeted only at replacing our use of jgroups UDP for membership messages.

Design

All of the messaging related to membership is handled by the JGroupsMessenger class, which implements the Messenger interface. We will create a new implementation of Messenger that uses TCP sockets, rather than JGroups and UDP sockets.

...

In order to accomplish this a member will actually need to be listening for connections on both protocols when it initially starts up. We will create a VersionAwareMessenger that contains both a JGroupsMessenger and a NettyMessenger. It can install handlers in both of them, and decide which Messenger to use when sending a message based on the version of the recipient. If a member receives a view that contains no old members that don't support the old protocol it could shut down the JGroups-based MessengerJGroupsMessenger.

There is an issue here with the need to listen on two separate ports and distribute both ports to other members. Currently, we distribute the jgroups UDP server port as part of the InternalDistributedMember. Those member ids InternalDistributedMembers are sent as part of view messages , which allow and find coordinator requests to the locator. That allows all members to discover the listening port of other members (see GMSJoinLeave message sequence diagrams). We need to distribute the new tcp port field as well. There are a couple options we are evaluating:

...

Changes and Additions to Public Interfaces

Because we are eventually getting rid of udp messaging, the following udp messaging related gemfire properties will be deprecated:

• udp-fragment-size

• udp-recv-buffer-size

• udp-send-buffer-size

• security-udp-dhalgo

• mcast-address

• mcast-ttl
• mcast-flow-control
• mcast-port
• mcast-recv-buffer-size
• mcast-send-buffer-size
• disable-tcp

...

One option to address concerns with using custom encryption protocol would be to continue using jgroups, but remove our old security-udp-dhalgo property in favor of using DTLS. However, this approach is complicated by the fact the the JDK does not have DTLS support built in to Java 8, which would require us to backport and maintain an implementation. In addition , this would not help us fix the problem of being tied to an outdated jgroups stack that cannot be upgraded.

...

One the face of it, just using our existing TCP sockets for membership messages as well seems attractive. However , this code as it is written is dependent on the membership system to bootstrap itself. For example , it will not even send messages until the member has completely joined and some startup messages are exchanged. It would take some refactoring to be able to send membership messages over this system in order to bootstrap membership. The existing P2P socket code is also fairly old, and we feel a better path forward is to move towards a new system based on netty that is less entangled with the rest of the system.

FAQ

Answers to questions you’ve commonly been asked after requesting comments for this proposal.

Errata

What are minor adjustments that had to be made to the proposal since it was approved?

...