THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Checklist
1. stable builds/intermittent test failures
There are no intermittent test failures specific to the HDDS-4440-s3-performance feature branch. On each commit during development, it was ensured that all CI workflow tests passed prior to the commit merge. To keep the master branch stable, full CI workflow runs are run multiple times on the latest commit prior to the final merge.
2. documentation
Documented in Apache Ozone Design docs as HDDS-4440 Proposed persistent OM connection for S3 gateway.
3. design, attached the docs
Design found in Jira HDDS-4440 and supporting related Jiras HDDS-5881, HDDS-5630.
4. s3 compatibility
This feature tries to provide 100% S3 compatibility when ozone.om.enable.filesystem.paths=false. This feature branch provides an enhancement to s3 gateway behavior for handling and relaying s3 errors to the client.
Whereas the existing gateway implementation returns an ambiguous 500 http response code with INTERNAL_ERROR in the event of client connection failures, this feature returns standard S3 errors for most errors.
This holds true except in the event the initial persistent connection cannot be established between the s3 gateway and the Ozone Manager. In this case the 500 http response is returned to the caller.
5. docker-compose / acceptance tests
Added enabling ozone manager Grpc server service to each docker-config for the development clusters: ozone, ozonesecure, ozone-ha and ozonesecure-ha.
To test the s3 gateway performance persistent connection Grpc feature with docker-compose / acceptance tests. Add the following configuration key settings to the docker-compose.yaml : om container - OZONE-SITE.XML_ozone.om.s3.grpc.server_enabled: "true" & s3g container - OZONE-SITE.XML_ozone.om.transport.class : "org.apache.hadoop.ozone.om.protocolPB.GrpcOmTransportFactory".
Then run acceptance tests, test.sh, for development cluster configured including ozone, ozonesecure, ozone-ha and ozonesecure-ha.
Also, with development cluster configured for s3g Grpc can load test the S3 gateway using the endpoint, localhost:9878. Load testers used include freon and warp.