Network protocol design is hard to get exactly right, and networking/serialization code has high standards of correctness – all the other code in a project depends on the right data getting sent across the wire. If you're trying to communicate between programs written in multiple languages, that code has to be written (and tested, and maintained) separately in each language.
RPC frameworks offer a nice solution to this problem: write the interface you want and your message definition in some Interface Definition Language (IDL), and then compile it to working (and hopefully idiomatic) code in a multitude of languages.
However, there are drawbacks:
- We don't control the implementation, which means it is hard to optimize for performance, and harder to add new features later.
- many of the RPC frameworks don't support push notifications from the server to client (implemented by some as "streaming").
- Generally, IDLs require static type definition. This means that we would have to define remote procedure calls as only taking one type. The only really workable approach would be to take a byte array serialized with some other data serializer and pass that through the RPC. This reduces the usefulness of an RPC framework somewhat, as it can't handle everything. However, having function stubs and having the transport taken care of is still quite useful, especially as the code works in multiple languages.
"Server-Client push" in the table below refers to the ability of the server to push messages back over the same communication channel the client uses to connect to it. BERT, for example, can take an address to deliver a message to when some remote call is done, but it has to be able to connect to that server. Many Geode users use firewalls, which would cause issues connecting to the client. Also, it's more adapted to the actor model than to a client-server model, as we use.
RPC Framework | Supported Languages | Serialization | Transport Layer | Server-Client push? | Dynamic type definition? | Throughput (msg/s) | Comments |
---|---|---|---|---|---|---|---|
Avro | at least Java, Python, Ruby, C#, C++ | Binary or JSON | transport-agnostic | No | Yes | ||
BERT | scroll to the bottom on the website to see. NO native Java support (though there is Scala). | Erlang | transport-agnostic or BERP (custom) | No | No | ||
gRPC | 10 (incl C++,Java,C#,Go) | Protobuf | Http2, Java uses Netty | Yes | No | ||
Apache Thrift | quite a few (incl C++,Java,C#, (unofficial) Go) | Custom | Pluggable: TTransport interface. | No | No |
Testing Setup
All perf tests were run on a MacPro 2.5Ghz i7, 16Gb RAM.
Two sets of performance tests were run:
- 1x server with 1 client doing 10mil blocking put messages
- 1x server with 2 clients doing 10mil blocking put messages each
Each test was run with the same message structure and key/value size.
Key Size: 64bytes
Value Size: 512bytes
The put message definition for Avro, Protobuf or Cap n Proto can be found HERE. GRPC uses Protobuf for message and service definition.