THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This document collects a few points to check before the merge, to make it easier to find possible problems in advance.
Merge
Please do NOT use github squash or rebase feature the merge feature branches. The records of multiple month of developments (Github PR and Jira discussions) are linked to the original commit ids. To keep them we should use simple git merge to merge branches.
Checklist
This section collects generic questions which can be checked for each feature branch. Some of them are obvious for some branches (for example: decommissioning feature didn't change the number of the components or the s3 interface) but it's good to go through of them for each of the merges. Answering the questions may also help the community to test the branch, therefore it can be useful to include similar information in the merge mail. If you have any new idea about what should be checked, please add it to the list.stable
1. stable builds/intermittent test failures
: To keep the master branch stable as much as possible, it's recommended to run full CI build multiple times on the latest commit, proposed to be merged.
How to check: The best to run 3-5 times, and they should be passed (any failure should be analyzed and collect previous failures on master, if it's unrelated to the branch)
acceptance tests: For bigger features we create acceptance test suite to the compose/ smoketest/ folders. It can help to keep the stability of the new feature AND can help to use an example for configuring/using the new feature in addition to the documentation.
.
- It can be done by creating a PR for the branch merge (please don't do a squash merge), or
- checking the last builds of the branch: github.com/apache/ozone/actions?query=BRANCH_NAME
2. documentation
Adding documentation: adding at least a basic level documentation before the merge may significant help the community to test the new feature. As writing initial documentation is just a few hours' effort, it's best to write it earlier.
Please add the documentation to the official documentation site instead of the wiki page. Wiki documentation is only for developers, the project itself should be documented in the `hadoop-hdds/docs`. Especially as this project is versioned together the source code: we can have different documentation for each release.
How to check:
| Code Block |
|---|
git diff $(git merge-base origin/master origin/$BRANCH_NAME) origin/$BRANCH_NAME -- hadoop-hdds/docs |
And
| Code Block |
|---|
cd hadoop-hdds/docs
hugo serve |
3. design, attached the docs
:Some internal design details are documented in the design docs. Ozone doc contains a section with the list of all the designs (for example: https://ci-hadoop.apache.org/view/Hadoop%20Ozone/job/ozone-doc-master/lastSuccessfulBuild/artifact/hadoop-hdds/docs/public/design.html). To make the original design doc easily accessible it's good to add a simple link (or full text) to the hadoop-hdds/docs/content/design. There we can also add additional notes (for example if some parts of the design doc is already out-dated it can be noted).
How to check:
Note: here we check the design subfolder
| Code Block |
|---|
git diff $(git merge-base origin/master origin/$BRANCH_NAME) origin/$BRANCH_NAME -- hadoop-hdds/docs/content/design |
| Code Block |
|---|
cd hadoop-hdds/docs
hugo serve |
4. s3 compatibility
:Today we try to provide 100% S3 compatibility by mimicking the original AWS S3 when ozone.om.enable.filesystem.paths=false (default). When it's enabled full s3 guarantees couldn't be provided as better Hadoop compatibility is favored.
How to check:
- This is not something which can be checked automatically. Please explain how s3 behavior changed by the branch.
5. docker-compose / acceptance tests
Ozone can be easily started by the docker-compose based scripts. We should make it easy to test new features with the help of docker-compose clusters. And if it can be started manually, adjusting smoketests (robot framework scripts) can make this testing permanent.
6. support of containers /
...
Kubernetes:
Today we have a full test suite / example deployments under the Kubernetes subdirectory. Most of the time it works out of the box, but in case of complex, new features it may be required to slightly update them.
How to check:
- hadoop-ozone/dist/src/main/k8s supposed to be updated
- Usually it's required only for new components or the configurations are changed.
7. coverage/code quality:
Coverage and code quality data is collected by Sonar. It's good to have the information of how these numbers are changing before the merge.
How to check:
Sonar has report for each branch. To URL template is https://sonarcloud.io/dashboard?branch=$BRANCH_NAME&id=hadoop-ozone
8. build time
: Another critical part is the build time. As more and more Apache project uses github actions the available resource will be less and less. We should keep track of the used build hours, and we should avoid significant increase of the build time unless it's fully necessary.
How to check:
- Choose the latest green build from the branch and from the mater (url template: github.com/apache/ozone/actions?query=BRANCH_NAME)
- Compare the execution time of the two builds
9. possible incompatible changes
...
:
Ozone 1.2.0 introduced a non-rolling upgrade framework to support upgrades and downgrades even when backwards incompatible features are present. Backwards incompatible features should be added to this framework so that they are not used until the Ozone upgrade is finalized, after which downgrading is not possible. Refer to the non-rolling upgrade user guide, design documents, and developer primer for more information.
Client cross compatibility should also be maintained as much as possible, with sensible error messages provided when this is not possible. An old client should be able to talk to the new Ozone instance, and a new client should be able to talk to the old Ozone instance.
10. third party dependencies/licence changes:
It's good to know which are the newly introduced third party dependencies, which can be easily checked by comparing the content of the distribution tar files. LICENSE (and NOTICE) should be updated for each new thirdparty third-party dependencies, and new jar files should be added only if they are necessary/used.
Merge
...
How to check:
The easiest check is building Ozone both with the new branch and master and compare the files on it.
| Code Block |
|---|
git checkout origin/master
mvn clean install -DskipTests
cd hadoop-ozone/dist/target/ozone-.../
find -type f | sort > /tmp/master
git checkout origin/$BRANCH_NAME
mvn clean install -DskipTests
cd hadoop-ozone/dist/target/ozone-.../
find -type f | sort > /tmp/$BRANCH_NAME
git diff /tmp/$BRANCH_NAME /tmp/master |
All new jar files should have new entries in LICENCE / NOTICE files.
11. performance
It's important to know how the base performance is changed by the branch. If there are any performance implications it is very useful to share the results from basic Freon testing to be sure that we don't introduce new bottlenecks.
12. security considerations
For bigger features we create acceptance test suite to the compose/ smoketest/ folders. It can help to keep the stability of the new feature AND can help to use an example for configuring/using the new feature in addition to the documentation.
Checklist template
Here the structure of the checklist is repeated (using h2. titles) to make it easier to copy-paste all the sections together:
| Code Block |
|---|
-------- COPY FROM HERE ----- |