THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Design found in Jira HDDS-4440 and supporting related Jiras HDDS-5881, HDDS-5630. ASF feature branch slack channel is, #ozone-s3g-grpc.
4. s3 compatibility
This feature tries to provide 100% S3 compatibility when ozone.om.enable.filesystem.paths=false. This feature branch provides an enhancement to s3 gateway behavior for handling and relaying s3 errors to the client.
...
For the s3-performance Grpc feature, network transport related jars are added to support native encryption on the wire, TLS:
Added to License.txt |
|---|
| + io.netty:netty-tcnative-boringssl-static |
| + io.netty:netty-tcnative |
11. performance
We compare the performance of the s3 gateway using the Grpc persistent connection with TLS to the existing Hadoop RPC, hRPC connections with encryption on the wire for metadata requests. We find that in load testing the s3 performance feature branch with Grpc and encryption on the wire outperforms the existing hRPC connection both encrypted and in plaintext. This is particularly evident in the comparison of Grpc with TLS to encrypted wire Hadoop RPC where the increase is greater than 2X.
| s3g Transport Type | Description | Load Test Performance for Metadata throughput, Objects / sec (objs/sec) | |
|---|---|---|---|
| 1 | Grpc TLS (feature branch) | s3g ↔ ozone manager connection over Grpc with encryption on the wire, TLS. Persistent connection. | 9026.12 |
| 2 | hRPC plaintext (current) | s3g ↔ ozone manager connection over Hadoop Rpc plaintext. Persistent connection (HDDS-5881). | 6508.85 |
| 3 | hRPC encrypted wire (current) | s3g↔ ozone manager connection over Hadoop Rpc with encryption on the wire (privacy configuration). Persistent connection (HDDS-5881). | 3989.35 |
Load test used: minio Warp S3 benchmarking tool.
...
12. security considerations
A new security model is introduced for S3 gateway persistent connections and was implemented in supporting Jira master branch patch, HDDS-5881. This branch uses the same security model for s3 user authentication on a per request basis.