Summary

Apache NiFi releases involve a Release Manager and members of the project community in various roles. The process involves both technical and procedural steps.

The Release Manager is a member of the Project Management Committee and oversees the process of preparing, reviewing, and publishing versions in coordination with community members.

Release Process Steps

Community member suggests a release timeline through a discussion email thread on dev@nifi.apache.org
A member of the PMC volunteers to act as the Release Manager for the version
The Release Manager validates the source branch and stages the distribution artifacts for a Release Candidate build
The Release Manager sends a vote email thread for the Release Candidate build to dev@nifi.apache.org
Community members verify and vote to approve or reject the Release Candidate build
1. The Release Manager can cancel a vote thread for a Release Candidate build
2. The Release Manager identifies issues for resolution and prepares a new Release Candidate build
The Release Manager publishes the distribution artifacts for an approved Release Candidate build

Prerequisites

Git
GnuPG with published singing key
Java Development Kit 21
Maven settings configured with server entries for Apache Nexus Repository

Maven Settings

<settings>
  <servers>
    <server>
      <id>apache.releases.https</id>
      <username>APACHE_USERNAME</username>
      <password>ENCRYPTED_APACHE_PASSWORD</password>
    </server>
    <server>
      <id>repository.apache.org</id>
      <username>APACHE_USERNAME</username>
      <password>ENCRYPTED_APACHE_PASSWORD</password>
    </server> 
  </servers>
</settings>

Release Candidate Build

The Release Manager is responsible for creating, signing, and staging artifacts for a Release Candidate build.

Version Tracking

Create an Apache NiFi Jira issue for tracking the release process with a Fix Version field set to the Release Version
Create a new version under Apache NiFi Jira Releases with the Version name set to the next minor version
Create a new version section on the project Release Notes page highlighting notable features and fixes

Build Variables

The build process includes several variables that are specific to each Release Candidate build.

#!/bin/bash

# Apache NiFi Jira issue for tracking the release process
JIRA_ISSUE=NIFI-00000

# Release Candidate number for each build
CANDIDATE_NUMBER=1

# Release version number
RELEASE_VERSION=2.0.0

# Next version number applied after the release process
NEXT_SNAPSHOT_VERSION=2.1.0-SNAPSHOT

# Apache Git username for committing changes
export GIT_COMMITTER_NAME=${USER}
export GIT_USER=${USER}

# Apache Git email address for committing changes
export GIT_COMMITTER_EMAIL="${USER}@apache.org"

# GPG key identifier for signing artifacts and commits
GIT_SIGNING_KEY=$(gpg --list-secret-keys --keyid-format LONG ${GIT_COMMITTER_EMAIL}|grep SC|cut -d \/ -f 2|cut -d ' ' -f 1)

# Working directory for building and staging distribution artifacts
WORK_DIR=~/releases/nifi-work
ARTIFACTS_DIR=${WORK_DIR}/artifacts
DIST_DIR=${WORK_DIR}/dist

Build Configuration

The build configuration consists of properties that apply to various steps in the release process.

#!/bin/bash

# Load build variables
source ./variables.sh

# Set artifact identifier for build elements
ARTIFACT_ID=nifi

# Set Git properties
GIT_AUTHOR="${GIT_COMMITTER_NAME} <${GIT_COMMITTER_EMAIL}>"
GIT_REPO=nifi

# Set Git source location and branch for building release candidate artifacts
GIT_ORIGIN_URL=https://github.com/${GIT_USER}/${GIT_REPO}.git
GIT_SOURCE_BRANCH=main

# Set Git upstream location for committing changes
GIT_UPSTREAM_URL=https://gitbox.apache.org/repos/asf/${GIT_REPO}.git

# Set Git tag and branch names
GIT_CANDIDATE_TAG="${GIT_REPO}-${RELEASE_VERSION}-RC${CANDIDATE_NUMBER}"
GIT_CANDIDATE_BRANCH="${JIRA_ISSUE}-RC${CANDIDATE_NUMBER}"
GIT_RELEASE_TAG="rel/nifi-${RELEASE_VERSION}"

# Set Subversion distribution locations
DIST_DEV_REPO=https://dist.apache.org/repos/dist/dev/nifi/
DIST_RELEASE_REPO=https://dist.apache.org/repos/dist/release/nifi/

Checkout Sources

Cloning the source repository and checking out the source branch provides the baseline for the release candidate build.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Create work directory
mkdir -p ${WORK_DIR}
cd ${WORK_DIR}

# Remove previous clones when necessary
rm -rf ${GIT_REPO}

# Clone source repository
git clone ${GIT_ORIGIN_URL}

# Checkout source branch as candidate branch
cd ${GIT_REPO}
git checkout -b ${GIT_CANDIDATE_BRANCH} ${GIT_SOURCE_BRANCH}

Version Sources

Set release version and commit changes to local release candidate branch.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Change to working cloned directory
cd ${WORK_DIR}/${GIT_REPO}

# Set Release Version
./mvnw versions:set -DgenerateBackupPoms=false -DnewVersion=${RELEASE_VERSION}

# Commit Release Version changes
git commit --gpg-sign=${GIT_SIGNING_KEY} -a --author "${GIT_AUTHOR}" -m "${JIRA_ISSUE} Updated version to ${RELEASE_VERSION}"

Stage Artifacts

Build and sign artifacts then deploy to the Nexus Repository. This step can take over an hour based on the number of artifacts.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Change to working cloned directory
cd ${WORK_DIR}/${GIT_REPO}

# Build and deploy artifacts to Nexus Repository with required release profile
./mvnw deploy -P apache-release -DskipTests -Dgpg.keyname=${GIT_COMMITTER_EMAIL}

Stage Binaries

Stage binary archives for signing.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Create Artifacts directory
mkdir -p ${ARTIFACTS_DIR} 

# Define Source Release Binary and signatures
SOURCE_RELEASE_ZIP=${ARTIFACT_ID}-${RELEASE_VERSION}-source-release.zip
SOURCE_RELEASE_ZIP_ASC=${SOURCE_RELEASE_ZIP}.asc
SOURCE_RELEASE_ZIP_SHA512=${SOURCE_RELEASE_ZIP}.sha512
SOURCE_DIR=${WORK_DIR}/${GIT_REPO}

# Copy Source Release files to artifacts directory
cp ${SOURCE_DIR}/target/${SOURCE_RELEASE_ZIP} ${ARTIFACTS_DIR}
cp ${SOURCE_DIR}/target/${SOURCE_RELEASE_ZIP_ASC} ${ARTIFACTS_DIR}
cp ${SOURCE_DIR}/target/${SOURCE_RELEASE_ZIP_SHA512} ${ARTIFACTS_DIR}

# Copy binary archives to artifacts directory
cp ${SOURCE_DIR}/minifi/minifi-assembly/target/minifi-${RELEASE_VERSION}-bin.zip ${ARTIFACTS_DIR}
cp ${SOURCE_DIR}/minifi/minifi-toolkit/minifi-toolkit-assembly/target/minifi-toolkit-${RELEASE_VERSION}-bin.zip ${ARTIFACTS_DIR}
cp ${SOURCE_DIR}/nifi-assembly/target/nifi-${RELEASE_VERSION}-bin.zip ${ARTIFACTS_DIR}
cp ${SOURCE_DIR}/nifi-registry/nifi-registry-assembly/target/nifi-registry-${RELEASE_VERSION}-bin.zip ${ARTIFACTS_DIR}
cp ${SOURCE_DIR}/nifi-registry/nifi-registry-toolkit/nifi-registry-toolkit-assembly/target/nifi-registry-toolkit-${RELEASE_VERSION}-bin.zip ${ARTIFACTS_DIR}
cp ${SOURCE_DIR}/nifi-stateless/nifi-stateless-assembly/target/nifi-stateless-${RELEASE_VERSION}-bin.zip ${ARTIFACTS_DIR}
cp ${SOURCE_DIR}/nifi-toolkit/nifi-toolkit-assembly/target/nifi-toolkit-${RELEASE_VERSION}-bin.zip ${ARTIFACTS_DIR}

Sign Binaries

Sign binary archives and generate hashes for subsequent publication.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Change to artifacts directory
cd ${ARTIFACTS_DIR}

# Sign binary archives
gpg -a -b --local-user ${GIT_SIGNING_KEY} --yes --digest-algo=SHA512 minifi-${RELEASE_VERSION}-bin.zip
gpg -a -b --local-user ${GIT_SIGNING_KEY} --yes --digest-algo=SHA512 minifi-toolkit-${RELEASE_VERSION}-bin.zip
gpg -a -b --local-user ${GIT_SIGNING_KEY} --yes --digest-algo=SHA512 nifi-${RELEASE_VERSION}-bin.zip
gpg -a -b --local-user ${GIT_SIGNING_KEY} --yes --digest-algo=SHA512 nifi-registry-${RELEASE_VERSION}-bin.zip
gpg -a -b --local-user ${GIT_SIGNING_KEY} --yes --digest-algo=SHA512 nifi-registry-toolkit-${RELEASE_VERSION}-bin.zip
gpg -a -b --local-user ${GIT_SIGNING_KEY} --yes --digest-algo=SHA512 nifi-stateless-${RELEASE_VERSION}-bin.zip
gpg -a -b --local-user ${GIT_SIGNING_KEY} --yes --digest-algo=SHA512 nifi-toolkit-${RELEASE_VERSION}-bin.zip

# Generate SHA-512 hash for binary archives
sh -c 'sha512sum $1 | cut -d " " -f 1 > $1.sha512' -- minifi-${RELEASE_VERSION}-bin.zip
sh -c 'sha512sum $1 | cut -d " " -f 1 > $1.sha512' -- minifi-toolkit-${RELEASE_VERSION}-bin.zip
sh -c 'sha512sum $1 | cut -d " " -f 1 > $1.sha512' -- nifi-${RELEASE_VERSION}-bin.zip
sh -c 'sha512sum $1 | cut -d " " -f 1 > $1.sha512' -- nifi-registry-${RELEASE_VERSION}-bin.zip
sh -c 'sha512sum $1 | cut -d " " -f 1 > $1.sha512' -- nifi-registry-toolkit-${RELEASE_VERSION}-bin.zip
sh -c 'sha512sum $1 | cut -d " " -f 1 > $1.sha512' -- nifi-stateless-${RELEASE_VERSION}-bin.zip
sh -c 'sha512sum $1 | cut -d " " -f 1 > $1.sha512' -- nifi-toolkit-${RELEASE_VERSION}-bin.zip

Distribute Binaries

Distribute binary artifacts to to the development distribution location using Subversion.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Change to distribution directory
mkdir -p ${DIST_DIR}
cd ${DIST_DIR}

# Checkout development distribution repository to local directory
svn checkout ${DIST_DEV_REPO} ${ARTIFACT_ID}
RELEASE_VERSION_DIR=${ARTIFACT_ID}-${RELEASE_VERSION}
RELEASE_DIR=${ARTIFACT_ID}/${RELEASE_VERSION_DIR}

# Copy binary artifacts to version directory
cd ${ARTIFACT_ID}
cp -r ${ARTIFACTS_DIR} ${RELEASE_VERSION_DIR}

# Commit and push binary artifacts to Subversion
svn update
svn add ${RELEASE_VERSION_DIR}
svn commit -m "${JIRA_ISSUE} Staging artifacts for ${RELEASE_VERSION_DIR}-RC${CANDIDATE_NUMBER}" ${RELEASE_VERSION_DIR}

Push Sources

Create source tag and push both tag and branch to upstream source repository.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Change to working cloned directory
cd ${WORK_DIR}/${GIT_REPO}

# Tag and sign version
git tag --sign --local-user=${GIT_SIGNING_KEY} -m "${JIRA_ISSUE} Tagged Release Candidate ${CANDIDATE_NUMBER}" ${GIT_CANDIDATE_TAG}

# Add remote repository location
git remote add upstream ${GIT_UPSTREAM_URL}

# Push candidate branch and candidate tag
git push upstream ${GIT_CANDIDATE_BRANCH}
git push upstream ${GIT_CANDIDATE_TAG}

Prepare Email

Prepare email for voting on release candidate build. The REPOSITORY_NUMBER must match the value from the Apache Nexus Repository.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Get Apache Nexus Repository number from staging artifacts
REPOSITORIES_URL=https://repository.apache.org/content/repositories/
REPOSITORY_NAME=$(curl -s ${REPOSITORIES_URL}|grep orgapachenifi|cut -d '>' -f 3|cut -d / -f 1)

# Get Jira Version and Issues
VERSIONS_JSON=$(curl -s https://issues.apache.org/jira/rest/api/2/project/NIFI/versions)
JIRA_VERSION=$(echo ${VERSIONS_JSON}|jq -r ".[] | select(.name==\"${RELEASE_VERSION}\") | .id" )
PROJECT_ID=$(echo ${VERSIONS_JSON}|jq -r ".[] | select(.name==\"${RELEASE_VERSION}\") | .projectId" )
JIRA_RELEASE_NOTES="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=${PROJECT_ID}&version=${JIRA_VERSION}"

# Get Total Issues
ISSUES_JSON=$(curl -s https://issues.apache.org/jira/rest/api/2/version/${JIRA_VERSION}/relatedIssueCounts)
TOTAL_ISSUES=$(echo ${ISSUES_JSON}|jq -r ".issuesFixedCount")

# Set total number of Jira issues changed
TOTAL_ISSUES=0

# Set SHA-512 hash of source release archive
SHA512_HASH=`cat ${WORK_DIR}/${GIT_REPO}/target/${ARTIFACT_ID}-${RELEASE_VERSION}-source-release.zip.sha512`

# Change to cloned working directory
cd ${WORK_DIR}/${GIT_REPO}

# Get current Git commit ref
GIT_COMMIT_ID=`git rev-parse HEAD`
PROJECT_LABEL="Apache NiFi"

echo """
[VOTE] Release ${PROJECT_LABEL} ${RELEASE_VERSION} (RC${CANDIDATE_NUMBER})

Team,

I am pleased to be calling this vote for the source release of ${PROJECT_LABEL} ${RELEASE_VERSION}.

Please review the following guide for how to verify a release candidate build:

https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification

The source being voted on the and the convenience binaries are available on the Apache Distribution Repository:

${DIST_DEV_REPO}${ARTIFACT_ID}-${RELEASE_VERSION}

The build artifacts are available on the Apache Nexus Repository:

${REPOSITORIES_URL}${REPOSITORY_NUMBER}/

Git Tag: ${GIT_REPO}-${RELEASE_VERSION}-RC${CANDIDATE_NUMBER}
Git Commit ID: ${GIT_COMMIT_ID}
GitHub Commit Link: https://github.com/apache/${GIT_REPO}/commit/${GIT_COMMIT_ID}

Hashes of ${ARTIFACT_ID}-${RELEASE_VERSION}-source-release.zip

SHA512: ${SHA512_HASH}

Release artifacts are signed with the following key:

https://people.apache.org/keys/committer/${GIT_COMMITTER_NAME}.asc

KEYS file is available on the Apache Distribution Repository:

https://dist.apache.org/repos/dist/release/nifi/KEYS

Issues resolved for this version: ${TOTAL_ISSUES}

${JIRA_RELEASE_NOTES}

Release note highlights can be found on the project wiki:

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version${RELEASE_VERSION}

The vote will be open for 72 hours.

Please download the release candidate and evaluate the necessary items including checking hashes, signatures, build from source, and test. Then please vote:

[] +1 Release this package as ${ARTIFACT_ID}-${RELEASE_VERSION}
[] +0 no opinion
[] -1 Do not release this package because...
"""

Distribute Binary Release Artifacts

After a positive vote thread approving the release, move the binary release artifacts from the development directory to the release directory.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Change to distribution directory
cd ${DIST_DIR}

# Set locations
DIST_DEV_URL=${DIST_DEV_REPO}/${ARTIFACT_ID}-${RELEASE_VERSION}
DIST_RELEASE_URL=${DIST_RELEASE_REPO}/${RELEASE_VERSION}

# Move binary artifacts with Subversion
svn move -m "${JIRA_ISSUE}" ${DIST_DEV_URL} ${DIST_RELEASE_URL}

Create Signed Tag

Create a signed Git tag and push to source repository.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Change to source directory
cd ${WORK_DIR}/${GIT_REPO}

# Create signed Git tag
git tag -u ${GIT_SIGNING_KEY} -s ${GIT_RELEASE_TAG} -m "${JIRA_ISSUE} Released version ${RELEASE_VERSION}" ${GIT_CANDIDATE_TAG}

# Push tag to source repository
git push upstream ${GIT_RELEASE_TAG}

Merge Changes

Merge version changes from release branch to main branch.

#!/bin/bash

# Load configuration
source ./configuration.sh

# Change to source directory
cd ${WORK_DIR}/${GIT_REPO}

# Fetch current upstream branch
git fetch upstream

# Rebase upstream changes
git checkout ${GIT_SOURCE_BRANCH}
git rebase upstream/${GIT_SOURCE_BRANCH}

# Merge changes
git merge --no-ff ${GIT_CANDIDATE_BRANCH} --gpg-sign=${GIT_SIGNING_KEY} -m "${JIRA_ISSUE} Merged ${GIT_CANDIDATE_BRANCH}"

# Push changes to source repository
git push upstream ${GIT_SOURCE_BRANCH}