Releasing Apache Geode

This article contains all the steps required to release Apache Geode. 

Software dependencies

1. git →  https://git-scm.com/downloads
2. docker → https://docs.docker.com/docker-for-mac/install/ 
3. svn → shipped with MacOS [install with command line tools → https://stackoverflow.com/questions/9329243/xcode-install-command-line-tools]
4. gpg tools → https://gpgtools.org
5. web browser → https://support.google.com/chrome/answer/95346?co=GENIE.Platform%3DDesktop&hl=en
6. java JDK → https://docs.oracle.com/javase/8/docs/technotes/guides/install/install_overview.html
7. text editor : vi → should be pre-installed
8. Homebrew → https://brew.sh/

Permission and keys:

Creating the release branch:

1. Check for the most recent develop SHA that passes all the tests and contains all the features / bug fixes that need to go into the release at https://concourse.apachegeode-ci.info/beta/teams/main/pipelines/develop/jobs/UpdatePassingRef.
2. Ensure in JIRA, that all tickets with {version} as the fix version are either closed or resolved. If not inform the the assignees to either resolve/close it or if not fixed then increment the fix version in the ticket

3. Create the release branch.

   Create release branch

   git checkout -b release/{version}

4. gradle.properties

   1. 1. remove -SNAPSHOT in releaseType 
5. docker/Dockerfile
   1. 1. Update GEODE_GPG with your GPG fingerprint.
      2. Update GEODE_VERSION with the release version (eg. 1.8.0)

6. Update the version number in all the expected pom files. Doing it manually takes a lot of time. If you choose to do it I have listed the files below. Otherwise please use the script below to do it.

   Updating the expected-pom files

   - build geode after the version is updated in gradle.properties. NOTE: The build will fail but will have the correct expected-pom.xml files

   gradle build

   ./gradlew clean build -Dskip.tests=true

   - Update dependency versions using gradle task updateExpectedPom.

   gradle uEP

   ./gradlew uEP

   - Publish the release branch to origin

   Publish the release branch

   git add .
   git commit -a -m "Upgraded version number for releasing {version}"
   pit push origin release/{version}

7. Send email to dev@geode.apache.org informing the creation of the release branch and requesting feedback.

   Hello Geode Dev Community,
   
   
   We have created a new release branch for Apache Geode {version} - "release/{version}"
   
   Please do review and raise any concern with the release branch.
   If concerns are raised, we will start with the voting for the release candidate soon.
   
   
   Regards
   {Release Manager}

   
   

8. Setting up concourse release pipeline

   a. You can deploy a release pipeline by deploying a meta pipeline. Run

   Hello Geode Dev Community,
   $ cd ci/pipelines/meta
   $ ./deploy_meta.sh
   
   # You might need fly in your PATH 
   # fly target named `concourse.apachegeode-ci.info`
   $ fly -t concourse.apachegeode-ci.info login --concourse-url https://concourse.apachegeode-ci.info/

   b. Send a email to dev@geode.apache.org requesting creation of a CI pipeline

   Hello Geode Dev Community,
   
   
   As we have created a release branch for Apache Geode {version} - "release/{version}"
   please create a CI pipeline for running tests on this branch.
   
   
   Regards
   {Release Manager}

Release Doc preparation:

1. Start preparing the release docs at Release Notes. Use information from all the JIRAs that were closed or resolved. 

2. Also write up a short paragraph on what was added to the release. This will need to go with the final announce email. Sample:

   Geode 1.7.0 contains a number of improvements and bug fixes. It includes performance improvements in OQL order-by and distinct queries in client/server when security is enabled. New GFSH commands were added to get/set cluster config and to destroy gateway receivers. A new post processor was added to the new client protocol. Pulse now supports legacy SSL options. Auto-reconnecting members no more reuse old addresses and IDs. Duplicated or member-specific receivers are removed from cluster config during rolling upgrades. Users are encouraged to upgrade to the latest release.

Prepare develop branch for the next release

1. gradle.properties
   1. 1. update versionNumber to the next release version (for example: if you created release branch for 1.8.0 update version number to 1.9.0 in the below files in develop branch)
      2. ensure that releaseType has -SNAPSHOT in it.
2. geode-core/src/main/java/org/apache/geode/internal/Version.java
   

   1. 1. Add the new ordinal

         New ordinal

         //Add 5 to the previous ordinal
         private static final byte GEODE_190_ORDINAL = 100;

      2. Add the new version 

         Adding the new version

         public static final Version GEODE_190 =
             new Version("GEODE", "1.9.0", (byte) 1, (byte) 9, (byte) 0, (byte) 0, GEODE_190_ORDINAL);

      3. Set the current version to the new version

         Set current to new version

         public static final Version CURRENT = GEODE_190;

         
         
         

      4. Update the highest version

         Set current to new version

         public static final int HIGHEST_VERSION = 105;

3.  geode-core/src/main/java/org/apache/geode/internal/cache/tier/sockets/CommandInitializer.java

   1. 1. Add the next version to addCommands

         CommandInitializer

         ALL_COMMANDS.put(Version.GEODE_190, commands);

         
         

4. Update the version in geode-book/config.yml and geode-book/redirects.rb on develop.

5. Update the expected-pom.xml files

   Updating the expected-pom files

   - build geode after the version is updated in gradle.properties. NOTE:The build will fail but will have the correct pom-default.xml files

   gradle build

   ./gradlew clean build -Dskip.tests=true

   - Update dependency versions using gradle task updateExpectedPom.

   gradle uEP

   ./gradlew uEP

   - Publish the develop branch to origin

   Publish the release branch

   git add .
   git commit -a -m "Upgraded version number for releasing 1.x.x"
   pit push origin develop

Creating the release candidate:

1. Checkout the release branch.
   
2. If you are using GPG Tools >= 2.1, export your secret key to a file to be used by Gradle. You can export it with `gpg --export-secret-keys >~/.gnupg/secring.gpg`
            
3. Log in to repository.apache.org and select staging repositories. Find "orgapachegeode-####" and clink on "Drop"
   

4. Clean and build the artifacts. Run this command on a machine using GUI as you will be prompted to enter PGP and ASF LDAP password (Apache password). You will see messages like "Could not find metadata org.apache.geode:geode-cq/maven-metadata.xml" on your terminal. Ignore them.

   git clean -fdx && ./gradlew clean build publish -Paskpass -Psigning.keyId=last_8_characters_of_your_gpg_fingerprint -Psigning.secretKeyRingFile=/Users/yourusername/.gnupg/secring.gpg
   mavenUsername=your_apache_ldap_username

5. Copy release artifacts to ASF dist repo. DO NOT DELETE THE FOLDERS CREATED IN THIS STEP - THEY ARE NEEDED FOR THE FINAL RELEASE.

   #create a new empty directory
   
   
   svn checkout https://dist.apache.org/repos/dist --depth empty
   svn update --set-depth infinity --parents dist/dev/geode
   svn update --set-depth infinity --parents dist/release/geode
    
   cd dist/dev/geode
   
   
   # copy the KEYS file from the apache-geode release branch folder in your system.
   cp ${GEODE_SRC}/KEYS .
   svn commit -m "Updating Apache Geode KEYS file"
    
   mkdir {version}.RC#
   cp ${GEODE_SRC}/geode-assembly/build/distributions/* {version}.RC#
    
   svn add {version}.RC#
   svn commit -m "Releasing Apache Geode {version}.RC# distribution"

6. Verify that all the artifacts have been uploaded to the nexus repository by logging into repository.apache.org and then click on close. Example:
   

7. In the release branch folder tag the commit. All Release Candidates start with the number 1 eg. RC1. Be careful as rel/ folder is protected, once we push the release tag to server, it cannot be modified. Any other change will have to be moved to the next RC number.

   git config user.email "email_address_associated_with_your_gpg_key"
   git tag -s rel/v{version}.RC# -m "Release candidate # for 1.0.0"
   git push origin rel/v{version}.RC#

8. Now checkout the repo for geode-examples and create a release branch.

   Create release branch

   git checkout develop
   git checkout -b release/{version}

9. Update the version number in gradle.properties of geode-examples

   version = 1.7.0
   geodeVersion = 1.7.0

10. Commit the version changes and push the release branch of geode-examples

    git add -p
    git commit
    git push origin HEAD

11. check that gfsh is not on your path already

    $ gfsh
    -bash: gfsh: command not found

12. In the geode-examples folder, build and run it with the apache-geode version that was uploaded to the ASF dist repo.

    # first build the release artifacts, specifying the RC release location and staging repository
    git clean -dxf && ./gradlew -PsignArchives -PgeodeReleaseUrl=https://dist.apache.org/repos/dist/dev/geode/{version}.RC# -PgeodeRepositoryUrl=https://repository.apache.org/content/repositories/orgapachegeode-{NNNN} build runAll
     
    #if all tests pass then continue ahead, else kindly fix the issues

13. If there are no failures, then upload the artifacts to ASF dist repo using the folder created in step 6

    # then upload the release artifacts for distribution
    # go to the folder created in step 6 and copy the geode-example artifacts to it
    cd dist/dev/geode
    cp ${GEODE_EXAMPLES_SRC}/build/distributions/* {version}.RC#
    svn add {version}.RC#/apache-geode-examples-*
    svn commit -m "Adding examples to Apache Geode {version}.RC# distribution"

14. Tag the release candidate in geode-examples.

    # then tag the release candidate commit
    cd ${GEODE_EXAMPLES_SRC}
    git tag -s rel/v{version}.RC# -m "Release candidate # for 1.0.0"
    git push origin rel/v{version}.RC#

15. If you are running on linux, you may see this error message:

    error: gpg failed to sign the data

    error: unable to sign the tag

     If you get that error message, try to sign some random file so that gpg will unlock your keyring. After that the git tag -s should work.

16. Make sure you have the native client required components installed. On a Mac

    brew install cmake
    brew install doxygen

17. Checkout geode-native from https://github.com/apache/geode-native. Then `cd` into the geode-native directory, create a build directory, configure cmake and generate the source distribution via cpack

    mkdir build
    cd build
    cmake .. -DPRODUCT_VERSION={version} -DOPENSSL_ROOT_DIR=/usr/local/Cellar/openssl/1.0.2* -DGEODE_HOME=your_geode_checkout/geode-assembly/build/install/apache-geode
    cpack -G TGZ --config CPackSourceConfig.cmake

18. This should have generated a tar file and a corresponding checksum file. Let's sign the package:

    gpg --armor -u {key ID} -b apache-geode-native-*-src.tar.gz

    Verify that the command created a new file called `apache-geode-native-{version}-Source.tar.gz.asc`.

19. Copy files into release distribution

    cd dist/dev/geode
    cp ${GEODE_NATIVE_SRC}/build/apache-geode-native-{version}* {version}.RC#
    svn add {version}.RC#/apache-geode-native-*
    svn commit -m "Adding Geode Native source distribution to Apache Geode {version}.RC#"

20. # then tag the release candidate commit
    cd ${GEODE_NATIVE_SRC}
    git tag -s rel/v{version}.RC# -m "Release candidate # for 1.0.0"
    git push origin rel/v{version}.RC#

Update the version in geode-example develop branch.

Update the version in gradle.properties to the next release's snapshot

version = 1.9.0-SNAPSHOT
geodeVersion = 1.9.0-SNAPTSHOT

Send out an email announcing the RC

From JIRA find out how many JIRAs were resolved and update the links in the below template and send it to dev@geode.apache.org. Everything within {} must be updated in the below template

Hello Geode dev community,


This is the {first/second} release candidate for Apache Geode, version {version}.
Thanks to all the community members for their contributions to this
release!
Please do a review and give your feedback. The deadline is the end of day {deadline date}.  
It resolves {# issues resolved} issues on Apache Geode JIRA system. Release notes can be found at:
https://cwiki.apache.org/confluence/display/GEODE/
Release+Notes#ReleaseNotes-{version}
Please note that we are voting upon the source tags: rel/v{version}.RC{#}
Apache Geode:
https://github.com/apache/geode/tree/rel/v{version}.RC{#}
Apache Geode examples:
https://github.com/apache/geode-examples/tree/rel/v{version}.RC{#}
Commit ID:
Apache Geode:
{----------SHA256 of the last git commit of geode ---------------------}
Apache Geode Examples:
{----------SHA256 of the last git commit of geode examples---------------------}
Apache Geode Native:
{----------SHA256 of the last git commit of geode examples---------------------}
Source and binary files:
https://dist.apache.org/repos/dist/dev/geode/{version}.RC{#}/
Maven staging repo:
https://repository.apache.org/content/repositories/orgapachegeode-{repo#}
Geode's KEYS file containing PGP keys we use to sign the release:
https://github.com/apache/geode/blob/develop/KEYS
Signed the release with fingerprint:
{ PGP public keys of the release manager}
{---------------------------------------}
{-------EXAMPLE-------------------------}
{rsa4096 2018-01-04 [SC] [expires: 2022-01-04]}
{CE6CD0A89480B1B9FCB98699274C66710770C135}

PS: Command to run geode-examples: ./gradlew -PgeodeReleaseUrl=https://dist.apache.org/repos/dist/dev/geode/{version}.RC# -PgeodeRepositoryUrl=https://repository.apache.org/content/repositories/orgapachegeode-{repo id} build runAll
Regards
{release manager}

Finalizing the release

Once the release candidate has been approved 

1. Checkout the release branch for apache geode
   
   

2. Tag the commit with the final version 

   git tag -s rel/v{version} -m "Apache Geode v1.0.0 release"
   git push origin rel/v{version}

3. Checkout the release branch for geode-examples
   
   

4. Tag the commit with the final version

   git tag -s rel/v{version} -m "Apache Geode v1.0.0 release"
   git push origin rel/v{version}

5. Use svn to move the distributions from dev to release. Use the folders created in step 6 of "Creating the release candidate" section.

   # in the directory created created in step 6 of "Creating the release candidate" section.
   
   cd dist
   svn mv dev/geode/{version}.RC#/ release/geode/{version}
   svn commit -m "Releasing Apache Geode {version} distribution"

6. Update the keys in dist/release.

   # in the directory created created in step 6 of "Creating the release candidate" section.
   
   
   cp dev/geode/KEYS release/geode/KEYS
   svn commit -m "Updating Apache Geode KEYS file"

7. Review the contents of dist/release. Remove any inactive releases. https://dist.apache.org/repos/dist/release/geode/
8. Promote the Nexus staging repo → login to repository.apache.org→ Select Staging repositories → Find "orgapachegeode-####" → Click on the Release Button on the toolbar.
   
   

9. Merge release branch to develop  and master for apache geode and then delete the release branch

   git checkout master
   git merge -s recursive -Xtheirs release/{version}
   git push origin master
     
   git checkout develop
   git merge -s recursive -Xtheirs release/{version}
   git push origin develop
    
   git branch -d release/{version}
   git push origin --delete release/{version}

10. Merge release branch to develop  and master for geode-examples and then delete the release branch

    git checkout master
    git merge -s recursive -Xtheirs release/{version}
    git push origin master
      
    git checkout develop
    git merge -s recursive -Xtheirs release/{version}
    git push origin develop
     
    git branch -d release/{version}
    git push origin --delete release/{version}

Apache Geode JIRA resolved → closed

1. List all the JIRAs with fix version as the release version and status as resolved.
   
   
   
   
2. Using bulk operations → transition to closed state.
   
   
   
   
   
   
   
   
   
   
   
   

Upload the docker image to docker hub.

1. In docker/Dockerfile update the GEODE_GPG with the GPG key you used to sign the release and GEODE_SHA256 with the SHA from https://dist.apache.org/repos/dist/release/geode/{VERSION}/apache-geode-{VERSION}.tgz.sha256 (Replace VERSION with the version you are trying to release).
2. Follow the instruction present in docker/README.md

Update the brew formula

# update brew
brew update


# Navigate to homebrew core directory
cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula



# Create a release branch
git checkout -b apache-geode-{version}


#Open apache-geode.rb
vim apache-geode.rb


# add the correct version in lines below 
# in url update "https://www.apache.org/dyn/closer.cgi?path=geode/{version}/apache-geode-{version}.tgz"
# in sha256 update with the SHA256 from https://dist.apache.org/repos/dist/release/geode/{version}/apache-geode-{version}.tgz.sha256


# save the file 


#commit the file - put the right version in below command.
git add .
git commit -a -m "apache-geode {version}"


# Fork git@github.com:Homebrew/homebrew-core.git to your repositories.
#add the remote 
git remote add myfork {your forked homebrew core repo}


#push to your remote forked repo
git push myfork apache-geode-{version}


#create a PR against git@github.com:Homebrew/homebrew-core.git - monitor the the tests in the PR page


#test it locally.
git checkout master


git merge apache-geode-{version}
brew install --build-from-source apache-geode


#check for gfsh version - should be the released version
gfsh version


# uninstall apache-geode
brew uninstall apache-geode


#reset master to origin
git reset --hard origin/master

Publish javadocs and documentation to the website

1. Build website from sources as described in geode-site/website/README.md.
2. In the generated site, create the directory geode-site/content/releases/latest.
3. Obtain a copy of the javadoc directory from the binary build and put it in the geode-site/content/releases/latest directory (link from the website's Docs landing page points to ../releases/latest/javadoc).
4. Deploy the generated site by checking it into the asf-site branch of the apache-geode repo.
   
   

Sending the announce mail.

Important: Send the email from an apache email ID.

The Apache Geode community is pleased to announce the availability of
Apache Geode {version}.
Apache Geode is a data management platform that provides a database-like
consistency model, reliable transaction processing and a shared-nothing
architecture to maintain very low latency performance with high concurrency
processing.
Geode {version} contains a number of improvements and bug fixes. It includes performance improvements in OQL order-by and distinct queries in client/server when security is enabled. New GFSH commands were added to get/set cluster config and to destroy gateway receivers. A new post processor was added to the new client protocol. Pulse now supports legacy SSL options. Auto-reconnecting members no more reuse old addresses and IDs. Duplicated or member-specific receivers are removed from cluster config during rolling upgrades. Users are encouraged to upgrade to the latest release.
For the full list of changes please review the release notes:
https://cwiki.apache.org/confluence/display/GEODE/
Release+Notes#ReleaseNotes-{version}
The release artifacts can be downloaded from the project website:
http://geode.apache.org/releases/
The release documentation is available at:
http://geode.apache.org/docs/guide/{version eg. in the format as 17 or 18 or 19}/about_geode.html
We would like to thank all the contributors that made the release possible.
Regards,
{Release Manager} on behalf of the Apache Geode team

Post release:

Add the released version to  geode-old-versions/build.gradle

Eg.

addOldVersion('test160', '1.6.0', true)