Page History

...

• Until

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-12622

  is complete, you will have to manually verify the binary artifact's LICENSE file matches its own packaged dependencies. See that ticket for an example of how to do this.
• Set environment variable `PUSH_REMOTE_NAME` to the remote you want to push to, e.g., export PUSH_REMOTE_NAME=origin (default is apache-github).
• Run the `release.py` script in the root of the kafka repository and follow the instructions. NOTE that if you are releasing a version prior to 1.0.x, you need to have minor edits on the script to change the three-digits pattern checking to four-digits pattern.
• This script will created a hidden directory called .release_work_dir. Don't delete this as you may need the contents later (for example the kafka-stream-x.x.x-test.jar)
• Troubleshooting:
  • If any step fails, make sure you have everything set up as described in the Prerequisites section and are using the correct passphrase for each config. When in doubt, remove or comment out anything in your settings that is not specifically needed for the release – for example, alternate profiles or unrelated servers, mirrors, etc in your `~/.m2/settings.xml` or `~/.gradle/gradle.properties`
  • Make sure you're running the script with Python3: sadly Python versions are not well managed especially on Macs, so you may want to explicitly invoke it with $python3 release.py 
  • Install any missing packages with pip3 , for example $pip3 install jira  – if pip can't find this, verify it's looking in the right repo by commenting out/removing any index-url settings in your ~/.pip/pip.conf 
  • Try setting an explicit path for both Java8 and Java17, rather than relying on your JAVA_HOME since this may be modified in the background by other applications (such as IDEs)

Website update process

Note: Unlike the Kafka sources (kafka repo), the content of the Apache Kafka website kafka.apache.org is backed by a separate git repository (kafka-site repo). Today, any changes to the content and docs must be kept manually in sync between the two repositories.

We should improve the release script to include these steps. In the meantime, for new releases:

• git clone git@github.com:apache/kafka-site.git
• git checkout asf-site
• Update the website content including docs:
  • The gradle target releaseTarGz generates the Kafka website content including the Kafka documentation (with the exception of a few pages like project-security.html, which are only tracked in the kafka-site repository). This build target also auto-generates the configuration docs of the Kafka broker/producer/consumer/etc. from their respective Java sources. The build output is stored in ./core/build/distributions/kafka_2.13-2.8.0-site-docs.tgz.
  • Untar the file and rename the site-docs/ folder to 28/ (or, if the latter already exists, replace its contents). That's because the docs for a release are stored in a separate folder (e.g., 27/ for Kafka v2.7 and 28/ for Kafka v2.8), which ensures the Kafka website includes the documentation for the current and all past Kafka releases.
• Update the javadocs:
  • Create the release Javadocs with the gradle target aggregatedJavadoc on JDK 17, with output under ./build/docs/javadoc/.
  • Copy the javadoc folder to 28/ (i.e., the full path is 28/javadoc/). If this is bug fix release, do this after the vote has passed to avoid showing an unreleased version number in the published javadocs.
• Commit & push

Blog Post

• For minor and major releases, consider writing a blog. Since the Apache blogs platform is now sunset, we've added a blog section to the Kafka website. Unfortunately this requires writing it in HTML, see the blog.html file in kafka-site.

• It's nice to thank as many people as we can identify.  Please use "Find all contributors" script to generate the list of contributors. 

• Consider incorporating any suggestions from the dev thread until release is announced

Announce the RC

• Send an email announcing the release candidate.

If need to roll a new RC

• Go to https://repository.apache.org/#stagingRepositories, find the uploaded artifacts and drop it.
• Go back to the beginning  - don't forget to bump the RC number.

Update the Collaborators List

While waiting for the vote to pass, this is a good time to update the Collaborators (see https://github.com/apache/kafka-site/pull/510 for reference until the site update is published).

The process is documented in

After the vote passes

Create Docker Artifact (For versions >= 3.7.0)

• Once you have executed release.py script, you'll be able to generate an RC docker image.
• This document covers details of each workflow. Refer to this for detailed examples and better understanding.
• In Github Actions of Apache Kafka repository (here) you'll find Docker Build Test Workflow.
• Click on run workflow and select the RC branch for the branch that this workkflow needs to run.
• Then provide the image type. If you are creating an RC image for apache/kafka , image type will be jvm.
• Then provide the url to RC kafka binary tarball (scala 2.13 version). This url will be used to generate the docker image.
• Run workflow and wait for it to finish. Once it finishes go back to the actions page and click on the workflow again. You'll find the uploaded artifacts. In case of jvm image type they will be report_jvm.html (docker image test report) and scan_report_jvm.txt (docker image cve report).
• This pipeline along with the downloaded artifacts can be shared with the community in the RC voting email thread.
• For creating RC docker image in apache/kafka you'll need to go to Build and Push Release Candidate Docker Image.
• Here image type and kafka url will be same as provided in previous workflow. In addition to that you also need to provide docker hub image that needs to be pushed.
• It will be apache/kafka:{rc_tag}. Here rc_tag is comprised of release version and rc version. For example apache/kafka:3.7.0-rc0 , can be the first RC docker image for 3.7.0 release.
• Ensure that the RC docker image is visible on docker hub.
• This docker image can be shared with the community in the RC voting email thread

Website update process

Note: Unlike the Kafka sources (kafka repo), the content of the Apache Kafka website kafka.apache.org is backed by a separate git repository (kafka-site repo). Today, any changes to the content and docs must be kept manually in sync between the two repositories.

We should improve the release script to include these steps. In the meantime, for new releases:

• git clone git@github.com:apache/kafka-site.git
• git checkout asf-site
• Update the website content including docs:
  • The gradle target releaseTarGz generates the Kafka website content including the Kafka documentation (with the exception of a few pages like project-security.html, which are only tracked in the kafka-site repository). This build target also auto-generates the configuration docs of the Kafka broker/producer/consumer/etc. from their respective Java sources. The build output is stored in ./core/build/distributions/kafka_2.13-2.8.0-site-docs.tgz.
  • Untar the file and rename the site-docs/ folder to 28/ (or, if the latter already exists, replace its contents). That's because the docs for a release are stored in a separate folder (e.g., 27/ for Kafka v2.7 and 28/ for Kafka v2.8), which ensures the Kafka website includes the documentation for the current and all past Kafka releases.
• Update the javadocs:
  • Create the release Javadocs with the gradle target aggregatedJavadoc on JDK 17, with output under ./build/docs/javadoc/.
  • Copy the javadoc folder to 28/ (i.e., the full path is 28/javadoc/). If this is bug fix release, do this after the vote has passed to avoid showing an unreleased version number in the published javadocs.
• Commit & push

Blog Post

• For minor and major releases, consider writing a blog. Since the Apache blogs platform is now sunset, we've added a blog section to the Kafka website. Unfortunately this requires writing it in HTML, see the blog.html file in kafka-site.

• It's nice to thank as many people as we can identify.  Please use "Find all contributors" script to generate the list of contributors. 

• Consider incorporating any suggestions from the dev thread until release is announced

Announce the RC

• Send an email announcing the release candidate.

If need to roll a new RC

• Go to https://repository.apache.org/#stagingRepositories, find the uploaded artifacts and drop it.
• Go back to the beginning  - don't forget to bump the RC number.

Update the Collaborators List

While waiting for the vote to pass, this is a good time to update the Collaborators (see https://github.com/apache/kafka-site/pull/510 for reference until the site update is published).

The process is documented in

After the vote passes

• Remember: at least 3 days, 3 +1 from PMC members (committers are not enough!) and no -1.

• Send a vote closing email:

  No Format

  To: dev@kafka.apache.org, kafka-clients@googlegroups.com, users@kafka.apache.org Subject: [RESULTS] [VOTE] Release Kafka version 0.10.0.0 This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * $Name * $Name * $Name Committers: * $Name * $Name Community: * $Name * $Name 0 votes * No votes -1 votes * No votes Vote thread: http://markmail.org/message/faioizetvcils2zo I'll continue with the release process and the release announcement will follow in the next few days. $RM

  
  

• Create a new tag for the release, on the same commit as the voted rc tag and push it:
  • Use "git show 0.10.0.0-rc6" to find the commit hash of the tag
  • git tag -a 0.10.0.0 <commit hash>
  • When asked to provide a message for the new tag, to preserve uniformity in release tags, add: Apache Kafka <version number> release. For example: Apache Kafka 0.10.0.0 release
  • Run "git show 0.10.0.0" and confirm that the tag points to the correct commit hash. 
  • git push apache 0.10.0.0
• Merge the last version change / rc tag into the release branch and bump the version to 0.10.0.1-SNAPSHOT
  • git checkout 0.10.0
  • git merge 0.10.0.0-rc6
  • Update version on the branch to 0.10.0.1-SNAPSHOT in the following places:
    • docs/js/templateData.js
    • gradle.properties
      
    • kafka-merge-pr.py
    • streams/quickstart/java/pom.xml
    • streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
    • streams/quickstart/pom.xml
    • tests/kafkatest/__init__.py (note: this version name can't follow the -SNAPSHOT convention due to python version naming restrictions, instead update it to 0.10.0.1.dev0)

    • tests/kafkatest/version.py

  • Run "git status" and "git diff" and make sure that only the files above have changed with the intended changes. (For example, there's no .release_work_dir/ from a previous RC generation).
  • git commit -a (the commit message could be "MINOR: Update 3.2 branch version to 3.2.1-SNAPSHOT")
    
  • git push apache 0.10.0

• Upload all artifacts, release notes, and docs (can be found in the .release_work_dir created by the release.py script) to https://dist.apache.org/repos/dist/release/kafka (a SVN repo, using Apache committer id/passwd).
  

  Note
  Note that only PMC members can upload to the `release` directory. If the RM is not in the PMC, they can upload the files to https://dist.apache.org/repos/dist/dev/kafka instead and ask a PMC member to move them to the release directory.

  
  

  1. Set your Apache username as environment variable

     Code Block
     APACHE_USERNAME=<apache_username>

     
     

  2. Create a directory for the release. Assuming the release is 0.10.0.0:

     Code Block
     svn mkdir --username $APACHE_USERNAME -m "Making directory for 0.10.0.0" https://dist.apache.org/repos/dist/release/kafka/0.10.0.0

     
     

  3. Checkout the directory for the new release:

     Code Block
     svn co --username $APACHE_USERNAME https://dist.apache.org/repos/dist/release/kafka/0.10.0.0 kafka-release-0-10-0-0

     
     

  4. copy the release artifacts from the latest RC (the ones which were in your home.apache.org directory) to the new release directory, but exclude the `javadoc` directory since we don't publish that anymore to https://dist.apache.org/repos/dist/release/kafka

     Code Block
     cd kafka-release-0-10-0-0 # note: there is a '~' sign in front of your APACHE_USERNAME # this step will take minutes wget --no-parent --recursive --reject "javadoc/*" "https://home.apache.org/~$APACHE_USERNAME/kafka-0.10.0.0-rc0/" mv "home.apache.org/~$APACHE_USERNAME/kafka-0.10.0.0-rc0/*" . rm -rf home.apache.org/ rm index.html*

     
     

  5. Add the files to SVN

     Code Block
     svn --username $APACHE_USERNAME add *

     
     

  6. Commit the artifacts to subversion

     Code Block
     svn commit --username $APACHE_USERNAME -m "Release 0.10.0.0"

     
     

• For PMC members only - If RM is a committer, you can move the artifacts from dev repo to release repo with the following commands. If RM is a PMC member, this step can be ignored.
  
  

  Code Block
  language bash
  # Set your env variable for apache username APACHE_USERNAME=<apache_username> # Create the directory for release in release repo svn mkdir

• Remember: at least 3 days, 3 +1 from PMC members (committers are not enough!) and no -1.

Send a vote closing email:

No Format

To: dev@kafka.apache.org, kafka-clients@googlegroups.com, users@kafka.apache.org Subject: [RESULTS] [VOTE] Release Kafka version 0.10.0.0 This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * $Name * $Name * $Name Committers: * $Name * $Name Community: * $Name * $Name 0 votes * No votes -1 votes * No votes Vote thread: http://markmail.org/message/faioizetvcils2zo I'll continue with the release process and the release announcement will follow in the next few days. $RM

• Create a new tag for the release, on the same commit as the voted rc tag and push it:
  • Use "git show 0.10.0.0-rc6" to find the commit hash of the tag
  • git tag -a 0.10.0.0 <commit hash>
  • When asked to provide a message for the new tag, to preserve uniformity in release tags, add: Apache Kafka <version number> release. For example: Apache Kafka 0.10.0.0 release
  • Run "git show 0.10.0.0" and confirm that the tag points to the correct commit hash. 
  • git push apache 0.10.0.0
• Merge the last version change / rc tag into the release branch and bump the version to 0.10.0.1-SNAPSHOT
  • git checkout 0.10.0
  • git merge 0.10.0.0-rc6
  • Update version on the branch to 0.10.0.1-SNAPSHOT in the following places:
    • docs/js/templateData.js
    • gradle.properties
      
    • kafka-merge-pr.py
    • streams/quickstart/java/pom.xml
    • streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
    • streams/quickstart/pom.xml
    • tests/kafkatest/__init__.py (note: this version name can't follow the -SNAPSHOT convention due to python version naming restrictions, instead update it to 0.10.0.1.dev0)

    • tests/kafkatest/version.py

  • Run "git status" and "git diff" and make sure that only the files above have changed with the intended changes. (For example, there's no .release_work_dir/ from a previous RC generation).
  • git commit -a (the commit message could be "MINOR: Update 3.2 branch version to 3.2.1-SNAPSHOT")
    
  • git push apache 0.10.0

• Upload all artifacts, release notes, and docs (can be found in the .release_work_dir created by the release.py script) to https://dist.apache.org/repos/dist/release/kafka (a SVN repo, using Apache committer id/passwd).
  

  Note
  Note that only PMC members can upload to the `release` directory. If the RM is not in the PMC, they can upload the files to https://dist.apache.org/repos/dist/dev/kafka instead and ask a PMC member to move them to the release directory.

  Set your Apache username as environment variable

  Code Block
  APACHE_USERNAME=<apache_username>

  Create a directory for the release. Assuming the release is 0.10.0.0:

  Code Block
  svn mkdir --username $APACHE_USERNAME -m "Making directory for 0.10.0.0" https://dist.apache.org/repos/dist/release/kafka/0.10.0.0

  Checkout the directory for the new release:

  Code Block
  svn co --username $APACHE_USERNAME https://dist.apache.org/repos/dist/release/kafka/0.10.0.0 kafka-release-0-10-0-0

  copy the release artifacts from the latest RC (the ones which were in your home.apache.org directory) to the new release directory, but exclude the `javadoc` directory since we don't publish that anymore to https://dist.apache.org/repos/dist/release/kafka

  Code Block
  cd kafka-release-0-10-0-0 # note: there is a '~' sign in front of your APACHE_USERNAME # this step will take minutes wget --no-parent --recursive --reject "javadoc/*" "https://home.apache.org/~$APACHE_USERNAME/kafka-0.10.0.0-rc0/" mv "home.apache.org/~$APACHE_USERNAME/kafka-0.10.0.0-rc0/*" . rm -rf home.apache.org/ rm index.html*

  Add the files to SVN

  Code Block
  svn --username $APACHE_USERNAME add *

  Commit the artifacts to subversion

  Code Blocksvn commit

  --username $APACHE_USERNAME -m "

  Release

  Making directory for 0.10.0.0"

   For PMC members only - If RM is a committer, you can move the artifacts from dev repo to release repo with the following commands. If RM is a PMC member, this step can be ignored.
  

  Code Block
  language bash
  # Set your env variable for apache username APACHE_USERNAME=<apache_username> # Create the directory for release in release repo svn mkdir https://dist.apache.org/repos/dist/release/kafka/0.10.0.0  # Checkout the directory for the new release svn co --username $APACHE_USERNAME https://dist.apache.org/repos/dist/release/kafka/0.10.0.0 kafka-release-0-10-0-0 # Copy the artifacts from dev  repo to local svn co --username $APACHE_USERNAME https://dist.apache.org/repos/dist/dev/kafka/0.10.0.0 -m "Making directory for 0.10.0.0" https://dist.apache.org/repos/dist/release/kafka/0.10.0.0  # Checkout the directory for the new release svn cokafka-dev-0-10-0-0 # Move files from one folder into another mv kafka-dev-0-10-0-0/* kafka-release-0-10-0-0 # Add files to SVN and commit cd kafka-release-0-10-0-0 svn --username $APACHE_USERNAME add *  svn commit --username $APACHE_USERNAME https://dist.apache.org/repos/dist/release/kafka/-m "Release 0.10.0.0 kafka-release-0-10-0-0" # CopyUpdate the artifacts from dev  repo to localPGP KEYS cd svn co --username $APACHE_USERNAME --depth empty https://dist.apache.org/repos/dist/dev/kafka/0.10.0.0 kafka-dev-0-10-0-0 # Move files from one folder into another mv kafka-dev-0-10-0-0/* kafka-release/release/kafka/ kafka-pgp-0-10-0-0 # Add files to SVN and commit cd kafka-releasepgp-0-10-0-0 svn --username $APACHE_USERNAME add * update KEYS svn commit --username $APACHE_USERNAME -m "Release 0.10.0.0" # Update the PGP KEYS cd svn co --username $APACHE_USERNAME --depth empty https://dist.apache.org/repos/dist/release/kafka/ kafka-pgp-0-10-0-0 cd kafka-pgp-0-10-0-0 svn update KEYS wget http://kafka.apache.org/KEYS mv KEYS.1 KEYS svn commit -m "Update PGP keys"

• Make sure the KEYS file in the svn repo includes the committer who signed the release.
  The KEYS must be in  https://dist.apache.org/repos/dist/release/kafka/KEYS and not just in http://kafka.apache.org/KEYS.

• wget http://kafka.apache.org/KEYS mv KEYS.1 KEYS svn commit -m "Update PGP keys"

  
  
  

• Make sure the KEYS file in the svn repo includes the committer who signed the release.
  The KEYS must be in  https://dist.apache.org/repos/dist/release/kafka/KEYS and not just in http://kafka.apache.org/KEYS.
• Go to https://repository.apache.org/#stagingRepositories, find the uploaded artifacts and release them (this will push the artifacts to maven central). You will be asked to provide a description on a pop-up menu that will allow you to add Apache Kafka 0.10.0.0 (in previous descriptions you'd normally have appended the RC identifier as well). 
  
  Image Added
  
  
• Wait for about a day for the artifacts to show up in apache mirror (releases, public group) and maven central (mvnrepository.com or maven.org).
• Release Docker Image:-
  • Run Promote Release Candidate Docker Image github action.
  • Here the RC Docker image will be the rc docker image that got voted and approved. For example if apache/kafka:3.7.0-rc1  got voted and approved, it will be the RC docker image that needs to be used.
  • Promoted image will be the final release name. For 3.7.0 release it will be apache/kafka:3.7.0 
  • Run the workflow and verify that the new docker image is visible on docker hub
  • If this is not a bug fix release, then ensure that you also release for latest tag. Which means you need to run the above pipeline again and use promoted image as apache/kafka:latest  
  Go to https://repository.apache.org/#stagingRepositories, find the uploaded artifacts and release them (this will push the artifacts to maven central). You will be asked to provide a description on a pop-up menu that will allow you to add Apache Kafka 0.10.0.0 (in previous descriptions you'd normally have appended the RC identifier as well). 
  Image Removed
  Wait for about a day for the artifacts to show up in apache mirror (releases, public group) and maven central (mvnrepository.com or maven.org). 
• In trunk update the following files with the current release number. This is needed for a feature as well as a bug-fix release (commit example)
  • KAFKA-REPO-DIR/gradle/dependencies.gradle
  • KAFKA-REPO-DIR/tests/docker/Dockerfile
  • KAFKA-REPO-DIR/tests/kafkatest/version.py
  • KAFKA-REPO-DIR/vagrant/base.sh
  • Upload the new release and kafka-stream-x.x.x-test.jar (can be found in the .release_work_dir created by the release.py script) to the S3 bucket "kafka-packages".  This is a S3 bucket owned by Confluent. If the RM is a committer from Confluent then follow the internal documentation for getting credentials. If the RM is not a committer from Confluent, please ask a committer from Confluent to do this for you.
    Use the AWS console to upload the files in the bucket or the CLI if you have appropriate keys. Update these commands to use the current release version. For example:
    • aws s3 cp .release_work_dir/<rc-version>/kafka_2.12-3.0.0.tgz s3://kafka-packages
    • aws s3 cp .release_work_dir/<rc-version>/kafka_2.13-3.0.0.tgz s3://kafka-packages
    • aws s3 cp .release_work_dir/kafka/streams/build/libs/kafka-streams-3.0.0-test.jar s3://kafka-packages

    • Make sure to update the permissions on AWS S3 so they are readable by everyone

      Code Block
      $ aws s3api put-object-acl --bucket kafka-packages --key kafka_2.13-3.0.0.tgz --acl public-read $ aws s3api put-object-acl --bucket kafka-packages --key kafka_2.12-3.0.0.tgz --acl public-read $ aws s3api put-object-acl --bucket kafka-packages --key kafka-streams-3.0.0-test.jar --acl public-read

      
      

• Update the website:
  • git clone https://github.com/apache/kafka-site
  • git checkout asf-site
  • Verify that documentation.html (in kafka-site repo) is referring to the correct release and links.
  • Verify that docs/documentation.html (in kafka repo) is similarly set up correctly.
  • If it's a feature release:
  • • Update files (e.g. documentation.html, protocol.html, quickstart.html, intro.html) to include the link for the new version (e.g. 0100/documentation.html). The full list of files can be found by:
      git grep "should always link the latest" | grep -v '^[0-9]'
      The command grep -v '^[0-9]' excludes per-release files (e.g. ./10/documentation/streams/upgrade-guide.html)
    • Verify that related html files (excluding per-release files) have been updated to use the new release version by doing
      git grep -Irn -A1 "should always link the latest" | grep -v '^[0-9]' 
      and checking that the new feature version is used.
    • Update files (e.g documentation.html, streams/quickstart.html) from the previous release (e.g current release is 2.8 so update files in the /27 folder) to change '<!--//#include virtual="'...  to '<!--#include virtual="... .  You can find the files by running
      git grep '<!--//#include virtual=' 
      from the directory of the previous feature release.  This kafka-site PR is an example of the changes that need to be made.
  • Update downloads.html to include the new download links from mirrors and change last release to use archive. Also add a paragraph with a brief feature introduction for the release.
  • git commit -am ".."
  • git push origin asf-site
  • Make sure the docs for the previous version are updated to display the "You're viewing documentation for an older version of Kafka" banner. This means un-commenting the banner out in two places: the previous version's branch of the kafka repo, and the previous version's directory in the kafka-site repo. See this commit for an example of which line displays this banner
• Mark the version as released in Kafka JIRA (from JIRA administration panel, select versions and scroll mouse towards the end of the line for the particular version. From the dropdown list, select release and set the date).

...