First of all, thanks for taking the time to validate an Apache StreamPipes (incubating) release!
The following steps should help you in performing the necessary steps to validate a release. Any comments or improvements to this guide are highly appreciated!
...
An Apache StreamPipes release currently consists of three different one source releases:
- incubator-streampipes is the core of Apache StreamPipes
- This also contains all the extensions (> 20 adapters and >
- 100 pipeline elements)
- & the installer (containing files for
- docker, a CLI, and helm charts for
- Kubernetes
Using the validation script
(sorry, the validation script is not yet available and will hopefully be ready after the first ASF release)
...
- )
Validation
Prerequisites
For fully testing a release candidate, you should have the following installed on your system:
- JDK (tested with required: JDK 817+)
- Maven (tested with 3.6)
- NodeJS + NPM (tested with v12v16+/ v6v8+)
- Docker + Docker-Compose
...
The artifacts are available at at https://dist.apache.org/repos/dist/dev/incubatorstreampipes/streampipes
Typically, this includes three one zip files (core, extensions and installer) along with checksum, signature, README, RELEASE_VALIDATION and RELEASE_NOTES.
...
Code Block | ||||
---|---|---|---|---|
| ||||
# download all release artifacts (core, extensions, installer) wget --recursive --no-host-directories -e robots=off --cut-dirs=5 --no-parent --reject "index.html*" https://dist.apache.org/repos/dist/dev/incubator/streampipes/ |
Step 2: Validate
...
streampipes
First, open the core folder (core/{version}/{rc}) (e.g., 0.70.0/rc1).
Check signatures and checksums
Verify the signature
If the output says "public key not found", follow the instructions in this guide: Validate a release for the first time
Code Block | ||||
---|---|---|---|---|
| ||||
gpg --verify apache-streampipes-{current-full-version}-incubating-source-release.zip.asc apache-streampipes-{current-full-version}-incubating-source-release.zip # example gpg --verify apache-streampipes-0.6692.0-incubating-source-release.zip.asc apache-streampipes-0.6692.0-incubating-source-release.zip |
Verify the checksum
Code Block | ||
---|---|---|
| ||
sha512sum -c apache-streampipes-{current-full-version}-incubating-source-release.zip.sha512 # example: sha512sum -c apache-streampipes-0.66.0-incubating92.0-source-release.zip.sha512 # alternative (if you get an error such as no properly formatted SHA512 checksum lines found) # print the checksum cat apache-streampipes-0.92.0-source-release.zip.sha512 # print the checksum of the zip file sha512sum apache-streampipes-0.92.0-source-release.zip # compare both checksums |
Make sure the output says OK.
...
Code Block | ||
---|---|---|
| ||
unzip apache-streampipes-{current-full-version}-incubating-source-release.zip cd apache-streampipes-{current-full-version}-incubating/ # example: unzip apache-streampipes-0.6692.0-incubating-source-release.zip cd apache-streampipes-0.6692.0-incubating/ |
Verify legal information and required files
...
Code Block | ||
---|---|---|
| ||
# cdin download-dir/apache-streampipes-#.##.#-incubating/ directory mvn clean install # we use install here as these dependencies will be later required by streampipes-extensions (with the core modules not yet being available on Maven Central at release validation time)package |
This build should be successful.
...
Code Block | ||
---|---|---|
| ||
# cd ui npm install npm run build-rc |
Start test system
If you want to test the system, there is a convenience docker-compose file that will help you starting the release candidate for testing.
...
Code Block | ||
---|---|---|
| ||
# in most cases, this step is optional - seems to only occur in rare cases on Windows systems
# Set the correct EOL encoding for the UI entrypoint file (docker-entrypoint.sh) file to LF:
awk 'BEGIN{RS="^$";ORS="";getline;gsub("\r","");print>ARGV[1]}' ui/docker-entrypoint.sh
# Alternative: Open the file ui/docker-entrypoint.sh, change the eol encoding to "LF" (e.g., in Notepad++ by clicking on "Windows CRLF" in the bottom status bar). |
...
Code Block | ||
---|---|---|
| ||
# from root directory
docker-compose up --build -d
# Use docker-compose up to run it in foreground
# Docker images will be locally built |
...
Once you're done with that, proceed to the next step. Before we test the whole system, we'll build and run the extensions.
Step 3: Validate incubator-streampipes-extensions
Open the extensions folder (extensions/{version}/{rc}.
Repeat the following steps for the extensions project, similar as explained above:
Verify signature
...
language | bash |
---|---|
title | Verify signature |
...
.
...
Verify checksum
Code Block | ||
---|---|---|
| ||
sha512sum -c apache-streampipes-extensions-{current-full-version}-incubating-source-release.zip.sha512
# example:
sha512sum -c apache-streampipes-extensions-0.66.0-incubating-source-release.zip.sha512 |
Make sure the output says OK.
Unzip the source file
Unzip and switch to the parent directory (where the README and pom files are located)
Code Block | ||
---|---|---|
| ||
unzip apache-streampipes-extensions-{current-full-version}-incubating-source-release.zip
cd apache-streampipes-extensions-{current-full-version}-incubating/
# example:
unzip apache-streampipes-extensions-0.66.0-incubating-source-release.zip
cd apache-streampipes-extensions-0.66.0-incubating/ |
Verify legal information and required files
...
Run RAT
Code Block | ||
---|---|---|
| ||
mvn rat:rat |
Search for SNAPSHOT dependencies
Search for any unexpected SNAPSHOT dependencies in the bundle, e.g.:
Code Block | ||
---|---|---|
| ||
grep -rnw './' -e 'SNAPSHOT'
# Check the output |
...
Build the extensions project as follows:
Code Block | ||
---|---|---|
| ||
mvn clean package |
Start test system
Go back to the main directory of the extensions release artifact (where the docker-compose.yml file is located) and do the following:
Code Block | ||
---|---|---|
| ||
docker-compose up --build -d
# Use docker-compose up to run it in foreground
# Docker images will be locally built |
See the instructions below under "Step 4: Test" to test the release candidate.
Step 3: incubator-streampipes-installer
Open the installer folder (installer/{version}/{rc}).
The installer contains no files that need to be built, just some bash scripts, docker-compose files and helm charts.
Repeat the following steps for the installer project, similar as explained above:
Verify signature
Code Block | ||||
---|---|---|---|---|
| ||||
gpg --verify apache-streampipes-installer-{current-full-version}-incubating-source-release.zip.asc apache-streampipes-installer-{current-full-version}-incubating-source-release.zip
# example
gpg --verify apache-streampipes-installer-0.66.0-incubating-source-release.zip.asc apache-streampipes-installer-0.66.0-incubating-source-release.zip |
Verify checksum
Code Block | ||
---|---|---|
| ||
sha512sum -c apache-streampipes-installer-{current-full-version}-incubating-source-release.zip.sha512
# example:
sha512sum -c apache-streampipes-installer-0.66.0-incubating-source-release.zip.sha512 |
Make sure the output says OK.
Unzip the source file
Unzip and switch to the parent directory.
Code Block | ||
---|---|---|
| ||
unzip apache-streampipes-installer-{current-full-version}-incubating-source-release.zip
cd apache-streampipes-installer-{current-full-version}-incubating/
# example:
unzip apache-streampipes-installer-0.66.0-incubating-source-release.zip
cd apache-streampipes-installer-0.66.0-incubating/ |
Verify legal information and required files
- Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES and RELEASE_VALIDATION files in the extracted source bundle.
- Check the LICENSE file
- Check the NOTICE file (e.g., make sure the year is correct)
...
Step 2: Test
If you have started the core and extensions in Docker as explained in the previous step, you can now switch to the user interface to do functional testing:
Open your browser and go to http://localhost (port 80) to get to the StreamPipes UI. You'll see the setup screen where you can start the setup procedure using a test user, e.g.,login screen:
- Default dummy user email: streampipes@apacheadmin@streampipes.apache.org
- dummy Default password: passwordadmin
Once you are logged in, you can test the functionalities of the softwareIf all goes well, the StreamPipes core as well as extensions are initialized and you should only see "checkmarks". After this step, click "Go to Login Page" and login to the system with the before specified credentials.
Info | ||
---|---|---|
| ||
The docker-compose.yml file that is part of the source is only intended for developers. There is no persistence enabled, when you stop the Docker containers, all information will be lost. Users are intended to use the installer. |
...
Here is a template that can help you valiating validating a release:
- Download all staged artifacts under the url specified in the release vote email
- Verify the signature is correct
- Check if the signature references an Apache email address
- Verify the SHA512 checksum
- Unzip the archive
- Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES, RELEASE_VALIDATION files in the extracted source bundle.
- Run RAT
- Search for SNAPSHOT dependencies
- Build Backend
- Build UI
- Build Extensionsand Run Test system on Docker