Page History

...

For minor and major releases, consider writing a blog. Since the Apache blogs platform is now sunset, we've added a blog section to the Kafka website. Unfortunately this requires writing it in HTML, see the blog.html file in kafka-site.

It's nice to thank as many people as we can identify. This script assumes that the previous release was 3.2.x and the current release is 3.3.0

No Format

# Configure the branch name for the current release and the previous release
CURRENT_RELEASE_BRANCH=apache-kafka/3.3
PREVIOUS_RELEASE_BRANCH=apache-kafka/3.2

# Get the latest common commit between them
MERGE_BASE=$(git merge-base $CURRENT_RELEASE_BRANCH $PREVIOUS_RELEASE_BRANCH)

# List of all contributors
git shortlog -sn --group=author --group=trailer:co-authored-by --no-merges "$MERGE_BASE..$CURRENT_RELEASE_BRANCH" | cut -f2 | sort --ignore-case | uniq > /tmp/kafka_contributors.txt

# Copy the list (don't forget to drop the trailing comma)
cat /tmp/kafka_contributors.txt | tr '\n' ',' | sed sed -e ':a' -e '$!N;s/\n/, /;ta' -e 's/,$//' -e 's/,%$/, /g' | xclip -selection clipboard

Consider incorporating any suggestions from the dev thread until release is announced

...

Remember: at least 3 days, 3 +1 from PMC members (committers are not enough!) and no -1.

Send a vote closing email:

No Format

To: dev@kafka.apache.org, kafka-clients@googlegroups.com, users@kafka.apache.org
Subject: [RESULTS] [VOTE] Release Kafka version 0.10.0.0

This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* $Name
* $Name 
* $Name

Committers:
* $Name
* $Name

Community:
* $Name
* $Name

0 votes
* No votes

-1 votes
* No votes

Vote thread:
http://markmail.org/message/faioizetvcils2zo

I'll continue with the release process and the release announcement will follow in the next few days.

$RM

Create a new tag for the release, on the same commit as the voted rc tag and push it:
- Use "git show 0.10.0.0-rc6" to find the commit hash of the tag
- git tag -a 0.10.0.0 <commit hash>
- When asked to provide a message for the new tag, to preserve uniformity in release tags, add: Apache Kafka <version number> release. For example: Apache Kafka 0.10.0.0 release
- Run "git show 0.10.0.0" and confirm that the tag points to the correct commit hash.
- git push apache 0.10.0.0
Merge the last version change / rc tag into the release branch and bump the version to 0.10.0.1-SNAPSHOT
- git checkout 0.10.0
- git merge 0.10.0.0-rc6
- Update version on the branch to 0.10.0.1-SNAPSHOT in the following places:
  - docs/js/templateData.js
  - gradle.properties
  - kafka-merge-pr.py
  - streams/quickstart/java/pom.xml
  - streams/quickstart/java/src/main/resources/archetype-resources/pom.xml
  - streams/quickstart/pom.xml
  - tests/kafkatest/__init__.py (note: this version name can't follow the -SNAPSHOT convention due to python version naming restrictions, instead update it to 0.10.0.1.dev0)
  - tests/kafkatest/version.py
- Run "git status" and "git diff" and make sure that only the files above have changed with the intended changes. (For example, there's no .release_work_dir/ from a previous RC generation).
- git commit -a (the commit message could be "MINOR: Update 3.2 branch version to 3.2.1-SNAPSHOT")
- git push apache 0.10.0

Upload all artifacts, release notes, and docs (can be found in the .release_work_dir created by the release.py script) to https://dist.apache.org/repos/dist/release/kafka (a SVN repo, using Apache committer id/passwd).

Note
Note that only PMC members can upload to the `release` directory. If the RM is not in the PMC, they can upload the files to https://dist.apache.org/repos/dist/dev/kafka instead and ask a PMC member to move them to the release directory.

Set your Apache username as environment variable
Code Block
APACHE_USERNAME=<apache_username>

Create a directory for the release. Assuming the release is 0.10.0.0:

Code Block
svn mkdir --username $APACHE_USERNAME -m "Making directory for 0.10.0.0" https://dist.apache.org/repos/dist/release/kafka/0.10.0.0

Checkout the directory for the new release:

Code Block
svn co --username $APACHE_USERNAME https://dist.apache.org/repos/dist/release/kafka/0.10.0.0 kafka-release-0-10-0-0

copy the release artifacts from the latest RC (the ones which were in your home.apache.org directory) to the new release directory, but exclude the `javadoc` directory since we don't publish that anymore to https://dist.apache.org/repos/dist/release/kafka

Code Block

cd kafka-release-0-10-0-0
# note: there is a '~' sign in front of your APACHE_USERNAME
# this step will take minutes
wget --no-parent --recursive --reject "javadoc/*" "https://home.apache.org/~$APACHE_USERNAME/kafka-0.10.0.0-rc0/"
mv "home.apache.org/~$APACHE_USERNAME/kafka-0.10.0.0-rc0/*" .
rm -rf home.apache.org/
rm index.html*

Add the files to SVN
Code Block
svn --username $APACHE_USERNAME add *

Commit the artifacts to subversion

Code Block
svn commit --username $APACHE_USERNAME -m "Release 0.10.0.0"

For PMC members only - If RM is a committer, you can move the artifacts from dev repo to release repo with the following commands. If RM is a PMC member, this step can be ignored.

Code Block

language	bash

# Set your env variable for apache username

APACHE_USERNAME=<apache_username>

# Create the directory for release in release repo

svn mkdir --username $APACHE_USERNAME -m "Making directory for 0.10.0.0" https://dist.apache.org/repos/dist/release/kafka/0.10.0.0 

# Checkout the directory for the new release

svn co --username $APACHE_USERNAME https://dist.apache.org/repos/dist/release/kafka/0.10.0.0 kafka-release-0-10-0-0

# Copy the artifacts from dev  repo to local

svn co --username $APACHE_USERNAME https://dist.apache.org/repos/dist/dev/kafka/0.10.0.0 kafka-dev-0-10-0-0

# Move files from one folder into another

mv kafka-dev-0-10-0-0/* kafka-release-0-10-0-0

# Add files to SVN and commit

cd kafka-release-0-10-0-0

svn --username $APACHE_USERNAME add * 

svn commit --username $APACHE_USERNAME -m "Release 0.10.0.0"

# Update the PGP KEYS

cd

svn co --username $APACHE_USERNAME  --depth empty https://dist.apache.org/repos/dist/release/kafka/ kafka-pgp-0-10-0-0

cd kafka-pgp-0-10-0-0

svn update KEYS

wget http://kafka.apache.org/KEYS

mv KEYS.1 KEYS

svn commit -m "Update PGP keys"

Make sure the KEYS file in the svn repo includes the committer who signed the release.
The KEYS must be in https://dist.apache.org/repos/dist/release/kafka/KEYS and not just in http://kafka.apache.org/KEYS.
Go to https://repository.apache.org/#stagingRepositories, find the uploaded artifacts and release them (this will push the artifacts to maven central). You will be asked to provide a description on a pop-up menu that will allow you to add Apache Kafka 0.10.0.0 (in previous descriptions you'd normally have appended the RC identifier as well).
Wait for about a day for the artifacts to show up in apache mirror (releases, public group) and maven central (mvnrepository.com or maven.org).

In trunk update the following files with the current release number. This is needed for a feature as well as a bug-fix release (commit example)

KAFKA-REPO-DIR/gradle/dependencies.gradle
KAFKA-REPO-DIR/tests/docker/Dockerfile
KAFKA-REPO-DIR/tests/kafkatest/version.py
KAFKA-REPO-DIR/vagrant/base.sh

Upload the new release and kafka-stream-x.x.x-test.jar (can be found in the .release_work_dir created by the release.py script) to the S3 bucket "kafka-packages". This is a S3 bucket owned by Confluent. If the RM is a committer from Confluent then follow the internal documentation for getting credentials. If the RM is not a committer from Confluent, please ask a committer from Confluent to do this for you.
Use the AWS console to upload the files in the bucket or the CLI if you have appropriate keys. Update these commands to use the current release version. For example:

aws s3 cp .release_work_dir/<rc-version>/kafka_2.12-3.0.0.tgz s3://kafka-packages
aws s3 cp .release_work_dir/<rc-version>/kafka_2.13-3.0.0.tgz s3://kafka-packages
aws s3 cp .release_work_dir/kafka/streams/build/libs/kafka-streams-3.0.0-test.jar s3://kafka-packages

Make sure to update the permissions on AWS S3 so they are readable by everyone

Code Block

$ aws s3api put-object-acl --bucket kafka-packages --key kafka_2.13-3.0.0.tgz --acl public-read
$ aws s3api put-object-acl --bucket kafka-packages --key kafka_2.12-3.0.0.tgz --acl public-read
$ aws s3api put-object-acl --bucket kafka-packages --key kafka-streams-3.0.0-test.jar --acl public-read

Update the website:
- git clone https://github.com/apache/kafka-site
- git checkout asf-site
- Verify that documentation.html (in kafka-site repo) is referring to the correct release and links.
- Verify that docs/documentation.html (in kafka repo) is similarly set up correctly.
- If it's a feature release:
- - Update files (e.g. documentation.html, protocol.html, quickstart.html, intro.html) to include the link for the new version (e.g. 0100/documentation.html). The full list of files can be found by:
    git grep "should always link the latest" | grep -v '^[0-9]'
    The command grep -v '^[0-9]' excludes per-release files (e.g. ./10/documentation/streams/upgrade-guide.html)
  - Verify that related html files (excluding per-release files) have been updated to use the new release version by doing
    git grep -Irn -A1 "should always link the latest" | grep -v '^[0-9]'
    and checking that the new feature version is used.
  - Update files (e.g documentation.html, streams/quickstart.html) from the previous release (e.g current release is 2.8 so update files in the /27 folder) to change '<!--//#include virtual="'... to '<!--#include virtual="... . You can find the files by running
    git grep '<!--//#include virtual='
    from the directory of the previous feature release. This kafka-site PR is an example of the changes that need to be made.
- Update downloads.html to include the new download links from mirrors and change last release to use archive. Also add a paragraph with a brief feature introduction for the release.
- git commit -am ".."
- git push origin asf-site
- Make sure the docs for the previous version are updated to display the "You're viewing documentation for an older version of Kafka" banner. This means un-commenting the banner out in two places: the previous version's branch of the kafka repo, and the previous version's directory in the kafka-site repo. See this commit for an example of which line displays this banner
Mark the version as released in Kafka JIRA (from JIRA administration panel, select versions and scroll mouse towards the end of the line for the particular version. From the dropdown list, select release and set the date).

Send out an announcement email.

The announcement email should be sent to announce@apache.org, users@kafka.apache.org, dev@kafka.apache.org, and kafka-clients@googlegroups.com
You need to use your apache email address to send out the email (otherwise, it won't be delivered to announce@apache.org).
- If you use gmail, you can configure it to send outbound mail from your apache address. Go to Settings → Accounts and Import → Send mail as: , and use the config:
  Server: mail-relay.apache.org Port: 587 (STARTTLS), 465 (SSL) User/Pass: {Your LDAP credentials}
- For other mail clients, see https://infra.apache.org/committer-email.html
You need to be subscribed to `kafka-clients@googlegroups.com` with your apache email address – otherwise it bounces back. Just send a message from your apache email to kafka-clients+subscribe@googlegroups.com and click `Join` in the confirmation email
Make sure to send the email as plain text, if there is any html (including basic hyperlinks) the email will bounce from announce@apache.org. For gmail, removing formatting might not be enough. Make sure you've selected the option "Plain text mode".
Double check that all the links in the email and on the downloads page work.

You can either use the release.py script to get the email contents or use the below template and instructions to construct it yourself:

Run `./release.py release-email` script in the root of the kafka repository and follow the instructions to generate the announcement email template for the release to the mailing list.

Copy the below template and fill in the sections such as the version, number-of-JIRAs, and contributor info, where:

number-of-contributors = `git shortlog -sn --no-merges <previous_release_tag>..<current_release_tag> | wc -l` (eg `git shortlog -sn --no-merges 0.8.2.2..0.9.0.0 | wc -l`)
list-of-contributors = `git shortlog -sn --no-merges <previous_release_tag>..<current_release_tag> | cut -f2 | tr '\n' ',' | sed -e 's/,/, /g'` (eg `git shortlog -sn --no-merges 0.8.2.2..0.9.0.0 | cut -f2 | sort --ignore-case | tr '\n' ',' | sed -e 's/,/, /g'`)

Check and update the Scala versions if necessary

Code Block

The Apache Kafka community is pleased to announce the release for Apache Kafka <version>.


This is a bug fix release and it includes fixes and improvements from <number-of-JIRAs> JIRAs, including a few critical bugs.


All of the changes in this release can be found in the release notes:

https://www.apache.org/dist/kafka/<version>/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.11 and Scala 2.12) from:

https://kafka.apache.org/downloads#<version>


---------------------------------------------------------------------------------------------------


Apache Kafka is a distributed streaming platform with four core APIs:

** The Producer API allows an application to publish a stream records to one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.

** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.



With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data between systems or applications.

** Building real-time streaming applications that transform or react to the streams of data.



Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others.



A big thank you for the following <number-of-contributors> contributors to this release!

<list-of-contributors>



We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/


Thank you!


Regards,
 $RM

Once the release is announced, the PMC member who committed the release artifacts to the SVN dist repository should add the release data to https://reporter.apache.org/addrelease.html?kafka (they will get a notification asking them to do this after the svn commit).
PMC member should double check if older releases should be archived (cf. http://www.apache.org/legal/release-policy.html#when-to-archive). This includes changing the relevant download links in the site's `download.html` to use the archive link.
- Cf.
  Jira
  server ASF JIRA
  serverId 5aa69414-a9e9-3523-82ec-879b028fb15b
  key KAFKA-6222
  and
  Jira
  server ASF JIRA
  serverId 5aa69414-a9e9-3523-82ec-879b028fb15b
  key KAFKA-6223
For feature releases, file a JIRA for updating compatibility/upgrade system tests to test the newly released version. Example PRs:
- Broker and clients: https://github.com/apache/kafka/pull/12210
- Streams: https://github.com/apache/kafka/pull/12209

For feature releases, publish the blog post previously shared with the dev list.

- Use the command to find all contributors for a release
- list-of-contributors = Use the command to find all contributors for a release

Check and update the Scala versions if necessary

Code Block

The Apache Kafka community is pleased to announce the release for Apache Kafka <version>.


This is a bug fix release and it includes fixes and improvements from <number-of-JIRAs> JIRAs, including a few critical bugs.


All of the changes in this release can be found in the release notes:

https://www.apache.org/dist/kafka/<version>/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.11 and Scala 2.12) from:

https://kafka.apache.org/downloads#<version>


---------------------------------------------------------------------------------------------------


Apache Kafka is a distributed streaming platform with four core APIs:

** The Producer API allows an application to publish a stream records to one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.

** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.



With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data between systems or applications.

** Building real-time streaming applications that transform or react to the streams of data.



Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others.



A big thank you for the following <number-of-contributors> contributors to this release!

<list-of-contributors>



We welcome your help and feedback. For more information on how to report problems, and to get involved, visit the project website at https://kafka.apache.org/


Thank you!


Regards,
 $RM

Once the release is announced, the PMC member who committed the release artifacts to the SVN dist repository should add the release data to https://reporter.apache.org/addrelease.html?kafka (they will get a notification asking them to do this after the svn commit).
PMC member should double check if older releases should be archived (cf. http://www.apache.org/legal/release-policy.html#when-to-archive). This includes changing the relevant download links in the site's `download.html` to use the archive link.
- Cf.
  Jira
  server ASF JIRA
  serverId 5aa69414-a9e9-3523-82ec-879b028fb15b
  key KAFKA-6222
  and
  Jira
  server ASF JIRA
  serverId 5aa69414-a9e9-3523-82ec-879b028fb15b
  key KAFKA-6223
For feature releases, file a JIRA for updating compatibility/upgrade system tests to test the newly released version. Example PRs:
- Broker and clients: https://github.com/apache/kafka/pull/12210
- Streams: https://github.com/apache/kafka/pull/12209
For feature releases, publish the blog post previously shared with the dev list.

Useful Commands

Find all contributors for a release

Code Block

language	bash

# The commands below assume that new version is 3.5.1 and last version is 3.5.

## set variables

CURRENT_RELEASE_TAG=3.5.1
OLD_RELEASE_TAG=3.5.0## get list of contributors (commit authors and co-authors) sorted in alphabetical order and separated by commagit shortlog -sn --group=author --group=trailer:co-authored-by --no-merges ${OLD_RELEASE_TAG}..${CURRENT_RELEASE_TAG} | cut -f2 | sort --ignore-case | uniq | sed -e ':a' -e '$!N;s/\n/, /;ta' -e 's/,$//' -e 's/%$//'

## get count of list of unique contributorsgit shortlog -sn --group=author --group=trailer:co-authored-by --no-merges ${OLD_RELEASE_TAG}..${CURRENT_RELEASE_TAG} | cut -f2 | sort --ignore-case | uniq | wc -l

Space shortcuts

Child pages

Versions Compared

Old Version 242

New Version 243

Key

Useful Commands

Find all contributors for a release