S3g Grpc Persistent Connections Feature Merge (HDDS-4440)

Checklist

1. stable builds/intermittent test failures

There are no intermittent test failures specific to the HDDS-4440-s3-performance feature branch.  On each commit during development, it was ensured that all CI workflow tests passed prior to the commit merge.  To keep the master branch stable, full CI workflow runs are run multiple times on the latest commit prior to the final merge.

2. documentation

Documented in Apache Ozone Design docs as HDDS-4440 Proposed persistent OM connection for S3 gateway.

• hadoop-hdds/docs/content/design/s3-performance.md

3. design, attached the docs

Design found in Jira HDDS-4440 and supporting related Jiras HDDS-5881, HDDS-5630.  ASF feature branch slack channel is, #ozone-s3g-grpc.

4. s3 compatibility

This feature tries to provide 100% S3 compatibility when ozone.om.enable.filesystem.paths=false.  This feature branch provides an enhancement to s3 gateway behavior for handling and relaying s3 errors to the client. 

Whereas the existing gateway implementation returns an ambiguous 500 http response code with INTERNAL_ERROR in the event of client connection failures, this feature returns standard S3 errors for most errors. 

This holds true except in the event the initial persistent connection cannot be established between the s3 gateway and the Ozone Manager.  In this case the 500 http response is returned to the caller.

5. docker-compose / acceptance tests

Added enabling ozone manager Grpc server service to each docker-config for the development clusters: ozone, ozonesecure, ozone-ha and ozonesecure-ha.

To test the s3 gateway performance persistent connection Grpc feature with docker-compose / acceptance tests.   Add the following configuration key settings to the docker-compose.yaml : om container - OZONE-SITE.XML_ozone.om.s3.grpc.server_enabled: "true" & s3g container - OZONE-SITE.XML_ozone.om.transport.class : "org.apache.hadoop.ozone.om.protocolPB.GrpcOmTransportFactory".

Then run acceptance tests, test.sh, for development cluster configured including ozone, ozonesecure, ozone-ha and ozonesecure-ha.

Also, with development cluster configured for s3g Grpc can load test the S3 gateway using the endpoint, localhost:9878.  Load testers used include freon and warp.

6. support of containers / Kubernetes:

NA. Deployment model for OzoneManager remains as earlier.

7. coverage/code quality: 

Sonar master branch code coverage.

Sonar HDDS-4440-s3-performance feature branch code coverage.

Code coverage is nearly unchanged from master to feature branch, from 76.5% to 75.7%.  In addition the feature branch has no Duplications nor Security vulnerabilities.  Further, the feature branch has a better Maintainability number than the master, 26 vs 207 (lower better).

8. build time

Recent Master branch build time: https://github.com/apache/ozone/actions/runs/2064597445

Current HDDS-4440-s3-performance feature branch build time: https://github.com/apache/ozone/actions/runs/1264276504.

Building on a local machine with ubuntu linux six-core i5 Coffee Lake and 64Gb Ram ($ mvn clean install -DskipTests):

9. possible incompatible changes: 

The S3g Grpc Persistent Connections feature is enabled through two s3g Grpc specific configuration keys.  One configuration key is to enable the Grpc server service on the Ozone Manager, OM, and the other is to enable the Grpc client on the S3 gateway, s3g.  By default the S3 gateway Grpc client is off and communication between the s3g and OM is though the existing Hadoop RPC.

To enable this feature set,

1. ozone.om.s3.grpc.server_enabled set to true in ozone-site.xml. (enable service on om)
   
2. ozone.om.transport.class set to org.apache.hadoop.ozone.om.protocolPB.GrpcOmTransportFactory in ozone-site.xml. (enable Grpc on s3g client)

With these two configuration keys disabled, the s3 gateway <-> ozone manager channel operates in legacy mode with the existing Hadoop RPC.  This can used in the upgrade period to turn off the feature when the feature is unstable and operate in legacy mode (Hadoop RPC communication).

10. third party dependencies/license changes:

For the s3-performance Grpc feature, network transport related jars are added to support native encryption on the wire, TLS:

11. performance

We compare the performance of the s3 gateway using the Grpc persistent connection with TLS to the existing Hadoop RPC, hRPC connections with encryption on the wire for metadata requests.  We find that in load testing the s3 performance feature branch with Grpc and encryption on the wire outperforms the existing hRPC connection both encrypted and in plaintext.  This is particularly evident in the comparison of Grpc with TLS to encrypted wire Hadoop RPC where the increase is greater than 2X.

Load test used: minio Warp S3 benchmarking tool.  

{$ ./warp stat --host=<host> --duration=1s –bucket bucket1 --concurrent=64 --noclear --obj.size=1KiB --access-key=$AWS_ACCESS_KEY --secret-key=$AWS_SECRET_ACCESS_KEY}

12. security considerations

This feature branch supports Grpc encryption channel communication between the S3 gateway and Ozone Manager through TLS.  Encryption on the wire for the Grpc channel is configured by the ozone-site.xml key,

1. hdds.grpc.tls.enabled set to true

  A new security model is introduced for S3 gateway persistent connections and was implemented in supporting Jira master branch patch, HDDS-5881.  This branch uses the same security model for s3 user authentication on a per request basis.