Page History

Table of Contents

Introduction

This release process document is based on Apache Beam Release Guide and Apache Flink Release Guide

...

1. Community agrees to release
2. Community selects a Release Manager

Prepare for the release 

As a release manager, you should create a private Slack channel, named `hudi-<version>_release_work` (e.g. hudi-0_12_0_release_work) in Apache Hudi Slack for coordination. Invite all committers to the channel.

Before your first release, you should perform one-time configuration steps. This will set up your security keys for signing the release and access to various release repositories.

...

• Create a branch in your repo (<user>/hudi). 
• cherry pick commits from master that needs to be part of this release. (git cherry-pick commit-hash). You need to manually resolve the conflicts. For eg, a file might have been moved to a diff class in master where as in your release branch, it could be in older place. You need to take a call where to place it. Similar things like file addition, file deletion, etc. 
• Ensure both compilation and tests are good. 
• Update the release version by running "mvn versions:set -DnewVersion=${RELEASE}-rc${RC_NUM}", with "RELEASE" as the version and "RC_NUM" as the RC number.  Make sure the version changes are intended.  Then git commit the changes.
• Ensure both compilation and tests are good. 
• I assume you will I assume you will have apache/hudi as upstream. If not add it as upstream
• Once the branch is ready with all commits, go ahead and push your branch to upstream. 
• Go to apache/hudi repo locally and pull this branch. Here after you can work on this branch and push to origin when need be. 
• Do not forget to set the env variables from above section. 

...

1. 1. git checkout ${RELEASE_BRANCH}

   2. Run mvn version to set the proper rc number in all artifacts

      1. mvn versions:set -DnewVersion=${RELEASE_VERSION}-rc${RC_NUM}

   3. Run Unit tests  and ensure they succeed

      1. mvn test -DskipITs=true

   4. Run Integration Tests and ensure they succeed

      1. mvn verify -DskipUTs=true

   5. Commit and push this change to RELEASE branch

      1. git commit -am "Bumping release candidate number ${RC_NUM}" 

         There will be some backup files created which needs to be removed. You could do "git clean -fd" before doing the commit.

      2. git push origin ${RELEASE_BRANCH}
         

         If you already have a remote tag with same name as your branch, you can try below command.

         git push origin refs/heads/${RELEASE_BRANCH}

         "refs/heads/" refers to a branch.

         "refs/tags/" refers to tag.

   6. Generate Source Release: This will create the tarball under hudi/src_release directory

      1. git checkout ${RELEASE_BRANCH}

      2. cd scripts && ./release/create_source_release.sh
         

         If you have multiple gpg keys(gpg --list-keys), then the signing command will take in the first key most likely. You will release this when it asks for a passphrase in a pop up. When asked for passphrase, ensure the intended key is the one asked for. 

         Command used in script:

         gpg --armor --detach-sig ${RELEASE_DIR}/hudi-${RELEASE_VERSION}.src.tgz

         To use a specific key: update as follows: // replace FINGERPRINT

         gpg --local-user [FINGERPRINT] --armor --detach-sig ${RELEASE_DIR}/hudi-${RELEASE_VERSION}.src.tgz

      3. Verify Source release is signed and buildable
         1. cd hudi/src_release
         2. gpg --verify hudi-${RELEASE_VERSION}-rc${RC_NUM}.src.tgz.asc hudi-${RELEASE_VERSION}-rc${RC_NUM}.src.tgz
         3. tar -zxvf hudi-${RELEASE_VERSION}-rc${RC_NUM}.src.tgz && cd hudi-${RELEASE_VERSION}-rc${RC_NUM} && mvn clean package -DskipTests -Pintegration-tests
         4. If they pass, delete the repository we got from the tar-ball
            1. cd ../ && rm -rf hudi-${RELEASE_VERSION}-rc${RC_NUM}
   7. Create tag 

      1. git tag -s release-${RELEASE_VERSION}-rc${RC_NUM} -m "${RELEASE_VERSION}"
         

         If you run into some issues, and if want to re-run the same candidate again from start, ensure you delete existing tags before retrying again.

         // to remove local

         git tag -d release-${RELEASE_VERSION}-rc${RC_NUM}

         // to remove remote
         git push --delete origin release-${RELEASE_VERSION}-rc${RC_NUM}

      2. if apache repo is origin.

         1. git push origin release-${RELEASE_VERSION}-rc${RC_NUM} 

            If a branch with the same name already exists in origin, this command might fail as below.

            error: src refspec release-0.5.3 matches more than one

            error: failed to push some refs to 'https://github.com/apache/hudi.git'

            In such a case, try below command

            git push origin refs/tags/release-${RELEASE_VERSION}-rc${RC_NUM}

   8. Stage source releases on dist.apache.org
      1. If you have not already, check out the Hudi section of the dev repository on dist.apache.org via Subversion. In a fresh directory
      2. if you would not checkout, please try svn checkout https://dist.apache.org/repos/dist/dev/hudi again.

         1. svn checkout https://dist.apache.org/repos/dist/dev/hudi --depth=immediates

      3. Make a directory for the new release:

         1.  mkdir hudi/hudi-

      Deploy maven artifacts and verify
      1. This will deploy jar artifacts to the Apache Nexus Repository, which is the staging area for deploying jars to Maven Central. 
      2. Review all staged artifacts (https://repository.apache.org/). They should contain all relevant parts for each module, including pom.xml, jar, test jar, source, test source, javadoc, etc. Carefully review any new artifacts.

      3. git checkout ${RELEASE_BRANCH}
         

      4. ./scripts/release/deploy_staging_jars.sh 2>&1 | tee -a "/tmp/

         1. ${RELEASE_VERSION}-

         $

         1. rc${RC_NUM}

         .deploy.log"
         1. make sure your IP is not changing while uploading, otherwise it creates a different staging repo
         2. Use a VPN if you can't prevent your IP from switching
         3. after uploading, inspect the log to make sure all maven tasks said "BUILD SUCCESS"
      5. Review all staged artifacts by logging into Apache Nexus and clicking on "Staging Repositories" link on left pane. Then find a "open" entry for apachehudi
      6. Ensure it contains all 3 (2.11, 2.12 with spark2 and 2.12 with spark3) artifacts, mainly hudi-spark-bundle-2.11/2.12, hudi-spark3-bundle-2.12, hudi-spark-2.11/2.12, hudi-spark2-2.11/2.12, hudi-spark3-2.12, hudi-utilities-bundle_2.11/2.12 and hudi-utilities_2.11/2.12.
         1. With 0.10.1, we had 4 bundles. spark2 with scala11, spark2 with scala12, spark3.0.x bundles and spark3.1.x bundles. Ensure each spark bundle reflects the version correctly. hudi-spark3.1.2-bundle_2.12-0.10.1.jar and hudi-spark3.0.3-bundle_2.12-0.10.1.jar are the respective bundle names for spark3 bundles. 
      7. Once you have ensured everything is good and validation of step 7 succeeds, you can close the staging repo. Until you close, you can re-run deploying to staging multiple times. But once closed, it will create a new staging repo. So ensure you close this, so that the next RC (if need be) is on a new repo. So, once everything is good, close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Hudi, version ${RELEASE_VERSION}, release candidate ${RC_NUM}”.
      9. Copy Hudi source distributions, hashes, and GPG signature: 

         1. mv <hudi-dir>/src_release/* hudi/hudi-${RELEASE_VERSION}-rc${RC_NUM}

      10. Add and commit all the files. 

          1. cd hudi 

          2. svn add hudi-${RELEASE_VERSION}-rc${RC_NUM}

          3. svn commit

      11. Verify that files are present
      12. Run Verification Script to ensure the source release is sane

          1. For RC: cd scripts && ./release/validate_staged_release.sh --release=${RELEASE_VERSION} --rc_num=${RC_NUM} --verbose

          2. For finalized release in dev: cd scripts && ./release/validate_staged_release.sh --release=${RELEASE_VERSION} --verbose
   9. Deploy maven artifacts and verify
      1. This will deploy jar artifacts to the Apache Nexus Repository, which is the staging area for deploying jars to Maven Central. 
      2. Review all staged artifacts (https://repository.apache.org/). They should contain all relevant parts for each module, including pom.xml, jar, test jar, source, test source, javadoc, etc. Carefully review any new artifacts.

      3. git checkout ${RELEASE_BRANCH}
         

      4. ./scripts/release/deploy_staging_jars.sh 2>&1 | tee -a "/tmp/${RELEASE_VERSION}-${RC_NUM}.deploy.log"
         1. when prompted for the passphrase, if you have multiple gpg keys in your keyring, make sure that you enter the right passphase corresponding to the same key (FINGERPRINT) as used while generating source release in step f.ii.

            1. If the prompt is not for the same key (by default the maven-gpg-plugin will pick up the first key in your keyring so that could be different), then add the following option to your ~/.gnupg/gpg.conf file

               Code Block
               default-key <FINGERPRINT_OF_KEY_USED_FOR_SOURCE_RELEASE>

               
               

         2. make sure your IP is not changing while uploading, otherwise it creates a different staging repo
         3. Use a VPN if you can't prevent your IP from switching
         4. after uploading, inspect the log to make sure all maven tasks said "BUILD SUCCESS"
      5. Review all staged artifacts by logging into Apache Nexus and clicking on "Staging Repositories" link on left pane. Then find a "open" entry for apachehudi
      6. Ensure it contains all 3 (2.11, 2.12 with spark2 and 2.12 with spark3) artifacts, mainly hudi-spark-bundle-2.11/2.12, hudi-spark3-bundle-2.12, hudi-spark-2.11/2.12, hudi-spark2-2.11/2.12, hudi-spark3-2.12, hudi-utilities-bundle_2.11/2.12 and hudi-utilities_2.11/2.12.
         1. With 0.10.1, we had 4 bundles. spark2 with scala11, spark2 with scala12, spark3.0.x bundles and spark3.1.x bundles. Ensure each spark bundle reflects the version correctly. hudi-spark3.1.2-bundle_2.12-0.10.1.jar and hudi-spark3.0.3-bundle_2.12-0.10.1.jar are the respective bundle names for spark3 bundles. 
      7. Once you have ensured everything is good and validation of step 7 succeeds, you can close the staging repo. Until you close, you can re-run deploying to staging multiple times. But once closed, it will create a new staging repo. So ensure you close this, so that the next RC (if need be) is on a new repo. So, once everything is good, close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Hudi, version ${RELEASE_VERSION}, release candidate ${RC_NUM}”.
      8. After closing, run the script to validate the staged bundles again:

         1. ./scripts/release/validate_staged_bundles.sh orgapachehudi-<stage_repo_number> ${RELEASE_VERSION}-rc${RC_NUM} 2>&1 | tee -a /tmp/validate_staged_bundles_output.txt

      Stage source releases on dist.apache.org
      1. If you have not already, check out the Hudi section of the dev repository on dist.apache.org via Subversion. In a fresh directory
      2. if you would not checkout, please try svn checkout https://dist.apache.org/repos/dist/dev/hudi again.

         1. svn checkout https://dist.apache.org/repos/dist/dev/hudi --depth=immediates

      3. Make a directory for the new release:

         1.  mkdir hudi/hudi-${RELEASE_VERSION}-rc${RC_NUM}

      4. Copy Hudi source distributions, hashes, and GPG signature: 

         1. mv <hudi-dir>/src_release/* hudi/hudi-${RELEASE_VERSION}-rc${RC_NUM}

      5. Add and commit all the files. 

         1. cd hudi 

         2. svn add hudi-${RELEASE_VERSION}-rc${RC_NUM}

         3. svn commit

      6. Verify that files are present
      7. Run Verification Script to ensure the source release is sane
         1. cd scripts && ./release/validate_staged_release.sh --release=${RELEASE_VERSION} --rc_num=${RC_NUM} --verbose

Checklist to proceed to the next step

...

1. change the version from ${RELEASE_VERSION}-rc${RC_NUM} to ${RELEASE_VERSION} against release branch, use command `mvn versions:set -DnewVersion=${RELEASE_VERSION}`, e.g. change 0.5.1-rc1 to 0.5.1.
2. Commit and push the version change to release branch.
   1. git commit -am "[MINOR] Update release version to reflect published version  ${RELEASE_VERSION}"
   2. git push origin release-${RELEASE_VERSION}
3. Repeat the steps from Generate Source Release(f) to Stage source releases on dist.apache.org(i). Including staging jars with the release version and uploading source release. Note that make sure remove the -rc${RC_NUM} suffix when repeat the above steps. and please also verify the steps.  Ensure git tag is also done without  -rc${RC_NUM}
4. One more step is to deploy source code to release dist. https://dist.apache.org/repos/dist/release/hudi. Only PMC will have access to this repo. So, if you are not a PMC, do get help from somone who is. 
   1. svn checkout https://dist.apache.org/repos/dist/release/hudi --depth=immediates, if you would not checkout, please try svn checkout https://dist.apache.org/repos/dist/release/hudi again.
   2. Make a directory for the new release:
      1.  mkdir hudi/${RELEASE_VERSION}
   3. Copy Hudi source distributions, hashes, and GPG signature: 
      1. mv <hudi-dir>/src_release/* hudi/${RELEASE_VERSION}
   4. Add and commit all the files. 
      1. cd hudi 
      2. svn add ${RELEASE_VERSION}
      3. svn commit
   5. Verify that files are present
5. Use the Apache Nexus repository to release the staged binary artifacts to the Maven Central repository. In the Staging Repositories section, find the relevant release candidate orgapachehudi-XXX entry and click Release. Drop all other release candidates that are not being released.  It can take up to 24 hours for the new release to show up in Maven Central repository. 
6. In Jira, go to Releases → <Release Version> and ensure that all Jiras for the release are 'Closed' state, if not transition all 'Resolved' jiras to 'Closed'.
7. Finalize the Release in Jira by providing the release date. 
8. Update DOAP file in the root of the project via sending a PR like this one.
9. Create a new Github release, off the release version tag, you pushed before

...

Steps to cut doc version and update website.

1. Follow the   instructions for cutting a doc for this new release.
   1. For a minor release, make sure any docs updates regarding the new features for the next major release, not in the minor release, are excluded.  This involves reverting the changes or manually removing the relevant docs in a separate branch.
2. Build the site locally  and ensure the new doc version is available as intended.
3. Update site using instructions
4. There are few adhoc fixes that needs to be taken care apart from above steps. Adding examples of commits from 0.10.1 that you can follow. 
   1. https://github.com/apache/hudi/pull/4703/commits/1e7c9af976b075c9ed5a780da1cdb57f019a41d3
   2. https://github.com/apache/hudi/pull/4703/commits/b474ec266fe2243f8146ba7a112045bbc8b0ddc8
   3. https://github.com/apache/hudi/commit/549fa7a51b30b162dcd6fc70b42cf1779de1900b
   4. https://github.com/apache/hudi/commit/de3405855c23aeb449113de197591c186396a4c2

...

...