First of all, thanks for taking the time to validate an Apache StreamPipes (incubating) release!

The following steps should help you in performing the necessary steps to validate a release. Any comments or improvements to this guide are highly appreciated!

...

An Apache StreamPipes release currently consists of three different one source releases:

• incubator-streampipes is the core of Apache StreamPipes
  incubator-streampipes-extensions contains several
  • This also contains all the extensions (> 20 adapters and >
  70
  • 100 pipeline elements)
  incubator-streampipes-installer contains installation
  • & the installer (containing files for
  Docker
  • docker, a CLI, and helm charts for
  running StreamPipes in Kubernetes.

  Using the validation script

  (sorry, the validation script is not yet available and will hopefully be ready after the first ASF release)

  Manual validation

  • • Kubernetes)

  Validation

  Prerequisites

  For fully testing a release candidate, you should have the following installed on your system:

  • JDK (required: JDK 17+)
  • Maven (tested with 3.6)
  • NodeJS + NPM (tested with v16+/ v8+)
  • Docker + Docker-Compose

  Step 1: Download release artifacts

  ...

  Download all staged artifacts under the url specified in the release vote email into a directory we’ll now call download-dir.

  The artifacts are available at at https://dist.apache.org/repos/dist/dev/incubatorstreampipes/streampipes 

  Typically, this includes three one zip files (core, extensions and installer) along with checksum, signature, README, RELEASE_VALIDATION and RELEASE_NOTES.

  incubator-streampipes

  You can either download the release artifacts one by one, or use the convenient single line command below:

  Code Block
  language bash title Download release artifacts
  # download all release artifacts wget --recursive --no-host-directories -e robots=off --cut-dirs=5 --no-parent --reject "index.html*" https://dist.apache.org/repos/dist/dev/streampipes/

  Step 2: Validate streampipes

  First, open the core folder (core/{version}/{rc}) (e.g., 0.70.0/rc1).

  Check signatures and checksums

  
  

  Verify the signature

  If the output says "public key not found", follow the instructions in this guide: Validate a release for the first time

  Code Block
  language bash title Verify signature
  gpg --verify apache-streampipes-{current-full-version}-source-release.zip.asc apache-streampipes-{current-full-version}-source-release.zip # example gpg --verify apache-streampipes-0.92.0-source-release.zip.asc apache-streampipes-0.92.0-source-release.zip

  
  

  Verify the checksum checksum:

  Code Block
  title Verify checksum
  sha512sum -c apache-streampipes-{current-full-version}-incubating-source-release.zip.sha512 # example: sha512sum -c apache streampipesapache-streampipes-0.92.0-source-release.zip.sha512 # alternative (if you get an error such as no properly formatted SHA512 checksum lines found) # print the checksum cat apache-streampipes-0.6692.0-incubating-source-release.zip.sha512 # print the checksum of the zip file sha512sum apache-streampipes-0.92.0-source-release.zip # compare both checksums

  Make sure the output says OK.

  Check source release

  
  

  Unzip the source file

  Unzip and switch to the parent directory (where the README and pom files are located) 

  Code Block
  title Unzip
  unzip apache-streampipes-{current-full-version}-source-release.zip cd apache-streampipes-{current-full-version}/ # example: unzip apache-streampipes-0.92.0-source-release.zip cd apache-streampipes-0.92.0/

  
  

  Verify legal information and required files

  •  Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES and RELEASE_VALIDATION files in the extracted source bundle.
  •  Check the LICENSE file
  •  Check the NOTICE file (e.g., make sure the year is correct)

  
  

  Run RAT

  Code Block
  title Run RAT
  mvn rat:rat

  
  

  Search for SNAPSHOT dependencies

  Search for any unexpected SNAPSHOT dependencies in the bundle, e.g.:

  Code Block
  title Search for SNAPSHOT dependencies
  grep -rnw './' -e 'SNAPSHOT' # Check the output

  
  

  Search for unexpected binaries

  Have a look at the source code to check there are no unexpected binaries, e.g., in the ui folder there shouldn't be any node_modules folder or dist folder.

  Build backend & UI

  incubator-streampipes-extensions

  incubator-streampipes-installer

  Verify

  gpg --verify zip.asc

  if not found

  gpg --keyserver hkp://keyserver.ubuntu.com:80 --rev-key KEY

  From the parent folder of the core, do the following to build the Java-based core:

  Code Block
  title Build Maven
  # in download directory mvn clean package

  This build should be successful.

  The next step is to build the UI, switch to the ui directory and do the following:

  Code Block
  title Build UI
  # cd ui npm install npm run build

  Start test system 

  If you want to test the system, there is a convenience docker-compose file that will help you starting the release candidate for testing.

  Go back to the main directory of the core release artifact (where the docker-compose.yml file is located) and do the following:

  Code Block
  title Set proper line encoding in docker entrypoint
  # in most cases, this step is optional - seems to only occur in rare cases on Windows systems # Set the correct EOL encoding for the UI entrypoint file (docker-entrypoint.sh) file to LF: awk 'BEGIN{RS="^$";ORS="";getline;gsub("\r","");print>ARGV[1]}' ui/docker-entrypoint.sh # Alternative: Open the file ui/docker-entrypoint.sh, change the eol encoding to "LF" (e.g., in Notepad++ by clicking on "Windows CRLF" in the bottom status bar).

  
  

  Code Block
  title Build Docker images and start
  # from root directory docker-compose up --build -d # Use docker-compose up to run it in foreground # Docker images will be locally built

  
  

  Once you're done with that, proceed to the next step. 

  Step 2: Test

  If you have started the core and extensions in Docker as explained in the previous step, you can now switch to the user interface to do functional testing:

  Open your browser and go to http://localhost (port 80) to get to the StreamPipes UI. You'll see the login screen:

  • Default user email: admin@streampipes.apache.org
  • Default password: admin

  Once you are logged in, you can test the functionalities of the software.

  Info
  title Testing with the release validation docker-compose file
  The docker-compose.yml file that is part of the source is only intended for developers. There is no persistence enabled, when you stop the Docker containers, all information will be lost. Users are intended to use the installer.

  Step 5: Clean Up

  After finishing testing the StreamPipes, one can stop and remove all running containers and clean up your host using the following two commands.

  Code Block
  language bash title Remove Docker containers and clean up
  docker stop $(docker ps -a -q) docker system prune --volumes

  ...

  
  

  Validation Template

  See also the Incubator Release Checklist (which is not official policy, but may help)

  Here is a template that can help you valiating validating a release:

  •  Download all staged artifacts under the url specified in the release vote email
  •  Verify the signature is correct
  •  Check if the check is successful.
  •  Check if the signature references an Apache email address.
  •  Verify the SHA512 hashes.checksum
  •  Unzip the archive.
  •  Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in the extracted source bundle. Verify the content of LICENSE, NOTICE, READMENOTES, RELEASE_NOTES files in VALIDATION files in the extracted source bundle.
  •  Run RAT
  •  Search for SNAPSHOT
  •  Build the project according to the information in the README.md file.
  • dependencies
  •  Build Backend
  •  Build UI
  •  Build and Run Test system on Docker