THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Documented in Apache Ozone Design docs as HDDS-4440 Proposed persistent OM connection for S3 gateway.
3. design, attached the docs
Design found in Jira HDDS-4440 and supporting related Jiras HDDS-5881, HDDS-5630. ASF feature branch slack channel is, #ozone-s3g-grpc.
4. s3 compatibility
This feature tries to provide 100% S3 compatibility when ozone.om.enable.filesystem.paths=false. This feature branch provides an enhancement to s3 gateway behavior for handling and relaying s3 errors to the client.
...
| Master branch build time: | 06:52 22 min |
|---|---|
| Feature branch HDDS-4440-s3-performance build time: | 06:16 min |
...
10. third party dependencies/license changes:
For the s3-performance Grpc feature, network transport related jars are added to support native encryption on the wire, TLS:
Added to License.txt |
|---|
| + io.netty:netty-tcnative-boringssl-static |
| + io.netty:netty-tcnative |
11. performance
We compare the performance of the s3 gateway using the Grpc persistent connection with TLS to the existing Hadoop RPC, hRPC connections with encryption on the wire for metadata requests. We find that in load testing the s3 performance feature branch with Grpc and encryption on the wire outperforms the existing hRPC connection both encrypted and in plaintext. This is particularly evident in the comparison of Grpc with TLS to encrypted wire Hadoop RPC where the increase is greater than 2X.
| s3g Transport Type | Description | Load Test Performance for Metadata throughput, Objects / sec (objs/sec) | |
|---|---|---|---|
| 1 | Grpc TLS (feature branch) | s3g ↔ ozone manager connection over Grpc with encryption on the wire, TLS. Persistent connection. | 9026.12 |
| 2 | hRPC plaintext (current) | s3g ↔ ozone manager connection over Hadoop Rpc plaintext. Persistent connection (HDDS-5881). | 6508.85 |
| 3 | hRPC encrypted wire (current) | s3g↔ ozone manager connection over Hadoop Rpc with encryption on the wire (privacy configuration). Persistent connection (HDDS-5881). | 3989.35 |
Load test used: minio Warp S3 benchmarking tool.
{$ ./warp stat --host=<host> --duration=1m –bucket bucket1 --concurrent=64 --noclear --obj.size=1KiB --access-key=$AWS_ACCESS_KEY --secret-key=$AWS_SECRET_ACCESS_KEY}
Test cluster consists of native ozone deployment, bare-metal. Om-SCM on one node, s3 gateway on separate node.
12. security considerations
This feature branch supports Grpc encryption channel communication between the S3 gateway and Ozone Manager through TLS. Encryption on the wire for the Grpc channel is configured by the ozone-site.xml key,
hdds.grpc.tls.enabledset totrue
A new security model is introduced for S3 gateway persistent connections and was implemented in supporting Jira master branch patch, HDDS-5881. This branch uses the same security model for s3 user authentication on a per request basis.