The easiest way to develop an S3 gateway endpoint is to run the gateway locally and start a cluster with docker.
To build Ozone and start a cluster, you can follow the Ozone Contributor Guide
mvn install -DskipTests -DskipShade -Pdist,hdds -Dmaven.javadoc.skip=true -Danimal.sniffer.skip=true -Dmaven.test.skip=true -Denforcer.skip=true -am -pl :hadoop-ozone-dist cd hadoop-ozone/dist/target/ozone-0.3.0-SNAPSHOT/compose/ozone docker-compose up -d |
Now you can check the SCM/OM web ui:
Now you can start S3 gateway from your IDE
Note: This section is based on Linux experiences. OSX usage could be different.
a.) You need a log4j.properties as this is handled usually by the IDE
b.) You need to configure ozone
c.) you need to have proper DNS to access the cluster
log4j.properties:
Use a simple file:
log4j.rootLogger=INFO,stdout
log4j.threshold=ALL
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n |
And activate it with VM option:
-Dlog4j.configuration=/home/elek/log4j.properties |
b.) ozone-site.xml
You can create an ozone-site.xml and add it to the classpath, but usually it's enough to configure the om.address:
-Dozone.om.address=ozoneManager |
Note: This is a program argument not a VM option.
c.) DNS
You need to access the datanode/ozoneManager with the docker names. You have multiple options. The first is just find the ip addresses of the containers (with docker inspect) and modify your host file.
The second option is to use a local DNS resolver:
docker run -d --name dns -p 53:53 -p 53:53/udp --network ozones3_default andyshinn/dnsmasq:2.76 -k -d |
Note 1: You need to change the DNS server with adding 127.0.0.0 to you resolv.conf
Note2: You need to adjust the value of the network argument. This should be the network for the docker-compose setup (ususally the directory name + _default)
To check the functionality of the existing AWS api you can use aws cli or s3cmd.
To make it easier to check the original response I use mitm proxy
Install the mitmproxy as defined on the product page
Start it in proxy mode`
mitmproxy -p 1212 |
Set up HTTP proxy in the command line
export HTTP_PROXY=http://localhost:1212 export HTTPS_PROXY=http://localhost:1212 |
Now you can use the aws cmd with the custom endpoint
aws s3 --endpoint-url http://localhost:9878/vol1 cp docker-config s3://bucket/qwe/dir1/dir1/file2 |