First of all, thanks for taking the time to validate an Apache StreamPipes (incubating) release!

The following steps should help you in performing the necessary steps to validate a release. Any comments or improvements to this guide are highly appreciated!

Introduction

An Apache StreamPipes release currently consists of three different source releases:

• incubator-streampipes is the core of Apache StreamPipes
• incubator-streampipes-extensions contains several extensions (> 20 adapters and > 70 pipeline elements)
• incubator-streampipes-installer contains installation files for Docker, a CLI and helm charts for running StreamPipes in Kubernetes.

Using the validation script

(sorry, the validation script is not yet available and will hopefully be ready after the first ASF release)

Manual validation

Prerequisites

For fully testing a release candidate, you should have the following installed on your system:

• JDK (tested with JDK 8+)
• Maven (tested with 3.6)
• NodeJS + NPM (tested with v12+/ v6+)
• Docker + Docker-Compose

Step 1: Download release artifacts

Download all staged artifacts under the url specified in the release vote email into a directory we’ll now call download-dir.

The artifacts are available at https://dist.apache.org/repos/dist/dev/incubator/streampipes 

Typically, this includes three zip files (core, extensions and installer) along with checksum, signature, README, RELEASE_VALIDATION and RELEASE_NOTES.

You can either download the release artifacts one by one, or use the convenient single line command below:

# download all release artifacts (core, extensions, installer)
wget --recursive --no-host-directories -e robots=off --cut-dirs=5 --no-parent --reject "index.html*" https://dist.apache.org/repos/dist/dev/incubator/streampipes/

Step 2: Validate incubator-streampipes

First, open the core folder (core/{version}/{rc}).

Check signatures and checksums

Verify the signature

If the output says "public key not found", follow the instructions in this guide: Validate a release for the first time

gpg --verify apache-streampipes-{current-full-version}-incubating-source-release.zip.asc apache-streampipes-{current-full-version}-incubating-source-release.zip

# example
gpg --verify apache-streampipes-0.66.0-incubating-source-release.zip.asc apache-streampipes-0.66.0-incubating-source-release.zip

Verify the checksum

sha512sum -c apache-streampipes-{current-full-version}-incubating-source-release.zip.sha512

# example:
sha512sum -c apache-streampipes-0.66.0-incubating-source-release.zip.sha512

Make sure the output says OK.

Check source release

Unzip the source file

Unzip and switch to the parent directory (where the README and pom files are located) 

unzip apache-streampipes-{current-full-version}-incubating-source-release.zip
cd apache-streampipes-{current-full-version}-incubating/

# example:
unzip apache-streampipes-0.66.0-incubating-source-release.zip
cd apache-streampipes-0.66.0-incubating/

Verify legal information and required files

• Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES and RELEASE_VALIDATION files in the extracted source bundle.
• Check the LICENSE file
• Check the NOTICE file (e.g., make sure the year is correct)

Run RAT

mvn rat:rat

Search for SNAPSHOT dependencies

Search for any unexpected SNAPSHOT dependencies in the bundle, e.g.:

grep -rnw './' -e 'SNAPSHOT'

# Check the output

Search for unexpected binaries

Have a look at the source code to check there are no unexpected binaries, e.g., in the ui folder there shouldn't be any node_modules folder or dist folder.

Build backend & UI

From the parent folder of the core, do the following to build the Java-based core:

# cd download-dir/apache-streampipes-#.##.#-incubating/
mvn clean install

# we use install here as these dependencies will be later required by streampipes-extensions (with the core modules not yet being available on Maven Central at release validation time)

This build should be successful.

The next step is to build the UI, switch to the ui directory and do the following:

# cd ui
npm install
npm run build-rc

Start test system 

If you want to test the system, there is a convenience docker-compose file that will help you starting the release candidate for testing.

Go back to the main directory of the core release artifact (where the docker-compose.yml file is located) and do the following:

# Set the correct EOL encoding for the UI entrypoint file (docker-entrypoint.sh) file to LF:
awk 'BEGIN{RS="^$";ORS="";getline;gsub("\r","");print>ARGV[1]}' ui/docker-entrypoint.sh

# Alternative: Open the file ui/docker-entrypoint.sh, change the eol encoding to "LF" (e.g., in Notepad++ by clicking on "Windows CRLF" in the bottom status bar).

docker-compose up --build -d

# Use docker-compose up to run it in foreground
# Docker images will be locally built

Once you're done with that, proceed to the next step. Before we test the whole system, we'll build and run the extensions.

Step 3: Validate incubator-streampipes-extensions

Open the extensions folder (extensions/{version}/{rc}.

Repeat the following steps for the extensions project, similar as explained above:

Verify signature

gpg --verify apache-streampipes-extensions-{current-full-version}-incubating-source-release.zip.asc apache-streampipes-extensions-{current-full-version}-incubating-source-release.zip

# example
gpg --verify apache-streampipes-extensions-0.66.0-incubating-source-release.zip.asc apache-streampipes-extensions-0.66.0-incubating-source-release.zip

Verify checksum

sha512sum -c apache-streampipes-extensions-{current-full-version}-incubating-source-release.zip.sha512

# example:
sha512sum -c apache-streampipes-extensions-0.66.0-incubating-source-release.zip.sha512

Make sure the output says OK.

Unzip the source file

Unzip and switch to the parent directory (where the README and pom files are located)

unzip apache-streampipes-extensions-{current-full-version}-incubating-source-release.zip
cd apache-streampipes-extensions-{current-full-version}-incubating/

# example:
unzip apache-streampipes-extensions-0.66.0-incubating-source-release.zip
cd apache-streampipes-extensions-0.66.0-incubating/

Verify legal information and required files

• Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES and RELEASE_VALIDATION files in the extracted source bundle.
• Check the LICENSE file
• Check the NOTICE file (e.g., make sure the year is correct)

Run RAT

mvn rat:rat

Search for SNAPSHOT dependencies

Search for any unexpected SNAPSHOT dependencies in the bundle, e.g.:

grep -rnw './' -e 'SNAPSHOT'

# Check the output

Build extensions

Build the extensions project as follows:

mvn clean package

Start test system

Go back to the main directory of the extensions release artifact (where the docker-compose.yml file is located) and do the following:

docker-compose up --build -d

# Use docker-compose up to run it in foreground
# Docker images will be locally built

See the instructions under "test" to test the release candidate.

Step 3: incubator-streampipes-installer

Open the installer folder (installer/{version}/{rc}).

The installer contains no files that need to be built, just some bash scripts, docker-compose files and helm charts.

Repeat the following steps for the installer project, similar as explained above:

Verify signature

gpg --verify apache-streampipes-installer-{current-full-version}-incubating-source-release.zip.asc apache-streampipes-installer-{current-full-version}-incubating-source-release.zip

# example
gpg --verify apache-streampipes-installer-0.66.0-incubating-source-release.zip.asc apache-streampipes-installer-0.66.0-incubating-source-release.zip

Verify checksum

sha512sum -c apache-streampipes-installer-{current-full-version}-incubating-source-release.zip.sha512

# example:
sha512sum -c apache-streampipes-installer-0.66.0-incubating-source-release.zip.sha512

Make sure the output says OK.

Unzip the source file

Unzip and switch to the parent directory.

unzip apache-streampipes-installer-{current-full-version}-incubating-source-release.zip
cd apache-streampipes-installer-{current-full-version}-incubating/

# example:
unzip apache-streampipes-installer-0.66.0-incubating-source-release.zip
cd apache-streampipes-installer-0.66.0-incubating/

Verify legal information and required files

• Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES and RELEASE_VALIDATION files in the extracted source bundle.
• Check the LICENSE file
• Check the NOTICE file (e.g., make sure the year is correct)

Step 4: Test

If you have started the core and extensions in Docker as explained in the previous step, you can now switch to the user interface to do functional testing:

Open your browser and go to http://localhost (port 80) to get to the StreamPipes UI. You'll see the setup screen where you can start the setup procedure using a test user, e.g.,

• dummy user email: streampipes@apache.org
• dummy password: password

If all goes well, the StreamPipes core as well as extensions are initialized and you should only see "checkmarks". After this step, click "Go to Login Page" and login to the system with the before specified credentials.

Validation Template

See also the Incubator Release Checklist (which is not official policy, but may help)

Here is a template that can help you valiating a release:

• Download all staged artifacts under the url specified in the release vote email
• Verify the signature is correct
• Check if the signature references an Apache email address
• Verify the SHA512 checksum
• Unzip the archive
• Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES, RELEASE_VALIDATION files in the extracted source bundle.
• Run RAT
• Search for SNAPSHOT dependencies
• Build Backend
• Build UI
• Build Extensions