...
NOTE: For the purpose of illustration, this document assumes that the version being released is 1.99.1, and the following development version will become 1.99.2
...
Table of Contents | ||
---|---|---|
|
...
Prerequisites
...
Policy documents
...
The RM should then send the pointer to the umbrella issue along with the tentative timeline for branch point to the user and developer lists. Any work identified as release related that needs to be completed should be added as a subtask of the umbrella issue to allow users to see the overall release progress in one place.
No Format |
---|
To: dev@sqoop.apache.org
Subject: Work on $release release has started
We are starting the process to prepare for Sqoop $release release. I have opened JIRA $jira to cover the tasks under this release.
If you have any JIRA in progress and would like to include it in this release, please follow the process to do so.
Feel free to comment on the JIRA if you have any comments/suggestions.
Thanks,
$RM
|
...
Before a release is done, make sure that any issues that are fixed have their fix version setup correctly. Run the following JIRA query to see which resolved issues do not have their fix version set up correctly:
Code Block |
---|
project = sqoop and resolution = fixed and fixVersion is empty
|
The result of the above query should be empty. If some issues do show up in this query that have been fixed since the last release, please bulk-edit them to set the fix version to '1.499.31'.
You can also run the following query to make sure that the issues fixed for the to-be-released version look accurate:
Code Block |
---|
project = sqoop and resolution = fixed and fixVersion = '1.99.1'
|
...
1. Clone fresh repository copy
Code Block |
---|
git clone https://git-wip-us.apache.org/repos/asf/sqoop.git
cd sqoop
|
2. Checkout Sqoop 2 branch
Code Block |
---|
git checkout sqoop2
|
3. Check that current HEAD points to commit on which you want to base new release branch. Checkout particular commit if not.
Code Block |
---|
git log # Check current branch history. HEAD should point to commit that you want to be base for your release branch
git checkout abcdef123 # Check out particular commit that should be base for release branch if -^
|
4. Create new release branch with name "branch-$version"
Code Block |
---|
git branch branch-1.99.1
|
5. Push new branch to Apache repository
Code Block |
---|
git push origin branch-1.99.1
|
...
7. Update version to your release version in all pom.xml files (.pom.xml, */pom.xml and *//pom.xml). You should be changing the version only in <parent> tag (with exception of root pom.xml file that obviously do not have a parent).
simple trick to get all the pom.xmls changed
Code Block |
---|
grep 2.0.0-SNAPSHOT * -RF | grep pom.xml |
8. Check your changes by running package build
Code Block |
---|
mvn package
|
9. Commit your changes and push them to Apache repository
Code Block |
---|
git commit -a -m "Changing version to 1.99.1"
git push origin branch-1.99.1
|
...
2. You can use ruby script jira2changelog.rb attached to this page to transform this page into desired output.
Code Block |
---|
jira2changelog.rb 'https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311320&version=12323641' # Change with your version Release notes
|
...
1. Check out release branch
Code Block |
---|
git clone https://git-wip-us.apache.org/repos/asf/sqoop.git git checkout branch-1.99.1 |
2. It may be worth checking to make sure we don't have any files that are out of sync with git before we start packaging.
Code Block |
---|
git status --ignored |
3. Make sure that LICENSE. Make sure that LICENSE.txt contains license information for all third-party libraries. In order to do this check properly, you will need to quickly create all packages. You need to check jars available in client/lib, server/lib and insider sqoop server war file (server/webapps/sqoop.war).
Code Block |
---|
//mvn package -DskipTestDskipTests //mvn package -DskipTests -Pbinary -Dhadoop.profile=100 // Since release 1.99.5 we have one hadoop profile we support mvn package -DskipTests -Pbinary -Dhadoop.profile=200 |
2. Create tag on this commit to identify precise point where the RC was generated and push this tag to main repository
Code Block |
---|
git tag -a release-1.99.1-rc0 -m "Sqoop 1.99.1-rc0 release"
git push origin release-1.99.1-rc0
|
3. Create temporary directory where you'll be preparing all required artifacts
Code Block |
---|
mkdir -p /tmp/sqoop-release-preparations
|
4. Create source artifact and move it to your temporary directory
Code Block |
---|
mvn clean verify package mv // This script will help determine which dependencies are used but not in the existing license file ls -1 dist/target/sqoop-12.99.1.tar.gz /tmp/sqoop-release-preparations |
5. Create binary artifact for each supported hadoop version and move generated artifacts to your temporary directory
Code Block |
---|
for version in 200 100; do mvn clean verify package -Pbinary -Dhadoop.profile=$version; mv dist/target/sqoop-1.99.1-bin-hadoop${version}.tar.gz /tmp/sqoop-release-preparations ; done
|
Update KEYS file
If your PGP key is not yet in the project's KEYS file, you need to first add that in. To do this, checkout the KEYS file and update it using the following commands:
Code Block |
---|
$ svn co https://svn.apache.org/repos/asf/sqoop/dist sqoop-dist
...
$ cd sqoop-dist
$ (gpg --list-sigs <KEY-ID> && gpg --armor --export <KEY-ID> ) >> KEYS
$ svn commit -m "Adding PGP public key to KEYS file" KEYS
...
|
Once this file has been updated, you need to publish it in the appropriate dist directory for the project on http://www.apache.org/dist/sqoop/KEYS
. To do this, you must copy the file as follows:
Code Block |
---|
$ scp KEYS people.apache.org:/www/www.apache.org/dist/sqoop/KEYS
|
This will take some time to propagate in which you can continue with the other steps of the release process.
Create signatures and check sums
1. Change your working directory to the temporal one
Code Block |
---|
cd /tmp/sqoop-release-preparations
|
2. Sing each file with your key
Code Block |
---|
for file in *.tar.gz; do gpg --armor --output $file.asc --detach-sig $file; done
|
3. You can immediately verify your signature
Code Block |
---|
for file in *.tar.gz; do gpg --verify $file.asc $file; done
|
4. Create md5 check sum
Code Block |
---|
for file in *.tar.gz; do md5sum $file > $file.md5; done
|
5. Create sha1 check sum
Code Block |
---|
for file in *.tar.gz; do sha1sum $file > $file.sha1; done
|
6. Upload artifacts and all created check sums with signatures to your own web-space on people.apache.org
Code Block |
---|
ssh people.apache.org "mkdir -p ~/public_html/sqoop-1.99.1"
scp * people.apache.org:~/public_html/sqoop-1.99.1
|
Running the vote
Call for sqoop dev list votes
Send an email to dev@ list. For example,
...
0.0-SNAPSHOT-bin-hadoop200/*/lib/ | grep ".jar" | grep -v "SNAPSHOT.jar" | sort -u | xargs -I {} ruby -e "file = '{}'; cutoff = file.index(/\d+\./); if cutoff then puts file.slice(0..cutoff-2) else puts file.slice(0..file.index('.jar')-1) end" | xargs -I {} sh -c 'if ! grep --quiet "For {}[-<version>]*.jar" LICENSE.txt ; then echo {} ; fi'
// This script will help determine which dependencies are in the license file but are not currently being used
grep "For .*jar" LICENSE.txt | xargs -I {} ruby -e "jar = '{}'; jar = jar.slice(4..jar.length-1); if jar.include?('<version>.jar') then puts jar.slice(0..jar.index('<version>.jar')-2) else puts jar.slice(0..jar.index('.jar')-1) end" | xargs -I {} sh -c 'if ! ls -1 dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200/*/lib/ | grep --quiet '{}' ; then echo '{}' ; fi' | sort
|
3. Create tag on this commit to identify precise point where the RC was generated and push this tag to main repository
Code Block |
---|
git tag -a release-1.99.1-rc0 -m "Sqoop 1.99.1-rc0 release"
git push origin release-1.99.1-rc0
|
4. Create temporary directory where you'll be preparing all required artifacts
Code Block |
---|
mkdir -p /tmp/sqoop-release-preparations
|
5. Create source artifact and move it to your temporary directory
Code Block |
---|
mvn clean verify package -Psource
mv dist/target/sqoop-1.99.1.tar.gz /tmp/sqoop-release-preparations
|
6. Create binary artifact for each supported hadoop version and move generated artifacts to your temporary directory
Code Block |
---|
// Since release 1.99.5 we have one hadoop profile we support
for version in 200; do mvn clean verify package -Pbinary -Dhadoop.profile=$version; mv dist/target/sqoop-1.99.1-bin-hadoop${version}.tar.gz /tmp/sqoop-release-preparations ; done
|
Update KEYS file
If your PGP key is not yet in the project's KEYS file, you need to first add that in. To do this, checkout the KEYS file and update it using the following commands:
Code Block |
---|
$ svn co https://dist.apache.org/repos/dist/release/sqoop sqoop-release
...
$ cd sqoop-release
$ (gpg --list-sigs <KEY-ID> && gpg --armor --export <KEY-ID> ) >> KEYS
$ svn commit -m "Adding PGP public key to KEYS file" KEYS
...
|
...
Once this file has been updated, it will automatically be pushed to
...
...
Need 3 +1 votes from PMC members.
Rolling out the Release
Upload the artifacts
Sqoop is using SVN based release procedure. In order to release you have to checkout release repository located on https://dist.apache.org/repos/dist/release/sqoop and add release artifacts there.
No Format |
---|
svn co https://dist.apache.org/repos/dist/release/sqoop sqoop-release
cd sqoop-release
mkdir 1.4.3
cp $source_to_your_artifacts 1.4.3
svn add 1.4.2
# Optionally change KEYS file in case that you've added your key for the first time
svn commit
|
Wait for 24 Hours
It may take up to 24 hours for all mirrors to sync up.
Announce the release
Send an email to announce@apache.org (the from: address must be @apache.org). For example,
No Format |
---|
To: announce@apache.org, user@sqoop.apache.org, dev@sqoop.apache.org
Subject: [ANNOUNCE] Apache Sqoop 1.4.3 released
The Apache Sqoop team is pleased to announce the release of Sqoop 1.4.3.
This is the second release of Apache Sqoop as a TLP project. Sqoop is a tool designed
for efficiently transferring bulk data between Apache Hadoop and structured datastores,
such as relational databases.
The release is available here:
http://www.apache.org/dyn/closer.cgi/sqoop/
The full change log is available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311320&version=12317345
Your help and feedback is more than welcome. For more information on how to report problems
and to get involved, visit the project website at http://sqoop.apache.org/.
The Apache Sqoop Team
|
Publish binary artifacts in Maven Repository
Note | ||
---|---|---|
| ||
Currently Sqoop is built using ant and thus the process necessary for publishing its binary artifacts in Maven repository is manual. Once Sqoop migrates to using Maven build (which is the default in Sqoop 2), these steps will need to be updated accordingly. |
Sqoop already has a Nexus repository setup, thanks to INFRA-4310. The following steps outline how you can publish Sqoop binaries to this repository.
Setup Maven settings
Follow the details provided in Maven Password Encryption Guide to set up a master password for your system.
Once the master password has been created, update your maven settings file (~/.m2/settings.xml) to have the following server entries:
Code Block |
---|
<settings>
...
<servers>
<server>
<id>apache.snapshots.https</id>
<username>your-asf-user-name</username>
<password>your-encrypted-asf-password</password>
</server>
<server>
<id>apache.releases.https</id>
<username>your-asf-user-name</username>
<password>your-encrypted-asf-password</password>
</server>
<server>
<id>apache.staging.https</id>
<username>your-asf-user-name</username>
<password>your-encrypted-asf-password</password>
</server>
</servers>
...
</settings>
|
Prepare artifacts to publish
In a newly created directory, checkout the sources from the appropriate release tag location. For example, for the release 1.4.0-incubating the sources will need to be checked out from https://svn.apache.org/repos/asf/incubator/sqoop/tags/release-1.4.0-rc1/
. Once you have the sources, build them to generate the binaries. For example:
Code Block |
---|
$ svn co https://svn.apache.org/repos/asf/incubator/sqoop/tags/release-1.4.0-rc1/
...
$ cd release-1.4.0-rc1
$ ant jar-all
...
BUILD SUCCESSFUL
Total time: 11 seconds
$
|
Install to local maven cache
Before you do this step, remove any previously installed Sqoop artifacts from your local maven cache. You can do this by running the following command:
Code Block |
---|
$ rm -rf ~/.m2/repository/com/cloudera/sqoop ~/.m2/repository/org/apache/sqoop
|
Now run the following command to install the newly checked-out and built artifacts of Sqoop from the previous step to your local maven cache.
Code Block |
---|
$ ant mvn-install
...
BUILD SUCCESSFUL
Total time: 6 seconds
$
|
This will result in copying of the built artifacts and generated POM file into your local maven cache.
Deploy main artifact to staging repo
Once the artifacts are in your local cache, you can now deploy them to Nexus staging repository.
Start out by going into the directory where the main artifact is generated. Once there, make sure that the generated POM file looks accurate and if not, fix it manually. Then run the following commands to deploy the main artifact.
Code Block |
---|
$ cd ~/.m2/repository/org/apache/sqoop/sqoop/1.4.0-incubating
$ ls -l
total 1108
-rw-r--r-- 1 arvind staff 532981 Jan 11 16:32 sqoop-1.4.0-incubating-sources.jar
-rw-r--r-- 1 arvind staff 590086 Jan 11 16:32 sqoop-1.4.0-incubating.jar
-rw-r--r-- 1 arvind staff 1670 Jan 11 16:32 sqoop-1.4.0-incubating.pom
$ mvn gpg:sign-and-deploy-file -Dfile=./sqoop-1.4.0-incubating.jar -DrepositoryId=apache.staging.https -Durl=https://repository.apache.org/service/local/staging/deploy/maven2/ -DpomFile=./sqoop-1.4.0-incubating.pom -Dgpg.keyname=<your-gpg-keyname>
...
[INFO] Metadata[project org.apache.sqoop:sqoop].filename = sqoop-1.4.0-incubating.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.169s
[INFO] Finished at: Wed Jan 11 16:58:35 PST 2012
[INFO] Final Memory: 6M/81M
[INFO] ------------------------------------------------------------------------
|
This will result in the signing and deployment of the main artifact sqoop-1.4.0-incubating.jar
in this case, to the staging repository. If all goes well you should be able to point your browser to Apache Nexus Repository and clicking on the Staging Repository link on the left navigation column. When you select the staging repository in the right pane, the pane will split into a content pane below that shows how the content of the staging repository created by the above command. The following screenshot illustrates what this looks like.
Deploy test artifact to staging repo
To deploy the test artifact to the staging repo, you must copy the generated test artifact to the same directory as that of the main directory so that you can use the main artifact POM and specify a classifier. Here are the commands to do that:
Code Block |
---|
$ ls -l
total 1116
-rw-r--r-- 1 arvind staff 532981 Jan 11 16:32 sqoop-1.4.0-incubating-sources.jar
-rw-r--r-- 1 arvind staff 590086 Jan 11 16:32 sqoop-1.4.0-incubating.jar
-rw-r--r-- 1 arvind staff 833 Jan 11 16:58 sqoop-1.4.0-incubating.jar.asc
-rw-r--r-- 1 arvind staff 1668 Jan 11 16:58 sqoop-1.4.0-incubating.pom
-rw-r--r-- 1 arvind staff 833 Jan 11 16:58 sqoop-1.4.0-incubating.pom.asc
$ cp ../../sqoop-test/1.4.0-incubating/sqoop-test-1.4.0-incubating.jar .
$ ls -l
total 1392
-rw-r--r-- 1 arvind staff 532981 Jan 11 16:32 sqoop-1.4.0-incubating-sources.jar
-rw-r--r-- 1 arvind staff 590086 Jan 11 16:32 sqoop-1.4.0-incubating.jar
-rw-r--r-- 1 arvind staff 833 Jan 11 16:58 sqoop-1.4.0-incubating.jar.asc
-rw-r--r-- 1 arvind staff 1668 Jan 11 16:58 sqoop-1.4.0-incubating.pom
-rw-r--r-- 1 arvind staff 833 Jan 11 16:58 sqoop-1.4.0-incubating.pom.asc
-rw-r--r-- 1 arvind staff 278892 Jan 11 17:06 sqoop-test-1.4.0-incubating.jar
$ mvn gpg:sign-and-deploy-file -Dfile=./sqoop-test-1.4.0-incubating.jar -DrepositoryId=apache.staging.https -Durl=https://repository.apache.org/service/local/staging/deploy/maven2/ -DpomFile=./sqoop-1.4.0-incubating.pom -Dgpg.keyname=<your-gpg-keyname> -Dclassifier=tests
...
[INFO] Metadata[project org.apache.sqoop:sqoop].filename = sqoop-1.4.0-incubating.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.160s
[INFO] Finished at: Wed Jan 11 17:10:01 PST 2012
[INFO] Final Memory: 6M/81M
[INFO] ------------------------------------------------------------------------
$
|
If this command succeeds, you can now see the test artifacts deployed in the staging repository as well. Go the the browser and click on the refresh button on the content area to reload the contents of the staging repository. The following screen shot is an example of how the staging repository may look like after these steps.
Close and Release staging repository
In the Nexus main content pane, make sure the staging repository you are working with is selected with a check mark, and click the "Close" button on top. This causes Nexus to verify the signatures and if everything is valid, it will result in successful closing of the repository. If there are any failures due to invalid signatures, you will have to drop the repository and redo the steps outlined in this section again.
Once the repository is closed, you can promote it to release status by clicking on the "Release" button on top. This allows the artifacts in this staging repository to be published to release repository immediately. Once the release has propagated, you should receive a confirmation email from Nexus stating that the artifacts are now available in the release repository.
Updating the website and the wiki with the new Sqoop bits:
...
.
This will take some time to propagate in which you can continue with the other steps of the release process.
Create signatures and check sums
1. Change your working directory to the temporal one
Code Block |
---|
cd /tmp/sqoop-release-preparations
|
2. Sing each file with your key
Code Block |
---|
for file in *.tar.gz; do gpg --armor --output $file.asc --detach-sig $file; done
|
3. You can immediately verify your signature
Code Block |
---|
for file in *.tar.gz; do gpg --verify $file.asc $file; done
|
4. Create md5 check sum
Code Block |
---|
for file in *.tar.gz; do md5sum $file > $file.md5; done
|
5. Create sha1 check sum
Code Block |
---|
for file in *.tar.gz; do sha1sum $file > $file.sha; done
|
Warning |
---|
The structure of the file names is given by Apache Software foundation and can't be changed. The name must be in form |
6. Upload artifacts and all created check sums with signatures to to repos/dist/dev/sqoop
Code Block |
---|
svn checkout https://dist.apache.org/repos/dist/dev/sqoop sqoop-dev
mkdir 1.99.1_rc0
cd 1.99.7_rc0
cp /tmp/sqoop-release-preparations/* .
svn add .
svn commit -m "Stage Sqoop 1.99.1 RC0 artifacts" |
Running the vote
...
Call for sqoop dev list votes
Send an email to dev@ list. For example,
No Format |
---|
To: dev@sqoop.apache.org
Subject: [VOTE] Release Sqoop version 1.99.1
This is the first cut of Sqoop 2 branch, version 1.99.1. Purpose of this release is to provide early bits to our users so that they can test the product and provide early feedback.This is the first cut of Sqoop 2 branch, version 1.99.1. Purpose of this release is to provide early bits to our users so that they can test the product and provide early feedback.
*** Please cast your vote by Friday 2012-12-21 ***
The list of fixed issues:
https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=blob;f=CHANGELOG.txt;h=c22af4131a3061dc78922a82a68eedc7e9ac599e;hb=85a8e1a8b3445360d66c8f812947cb0a7e8230c9
The tarball (*.tar.gz), signature (*.asc), checksum (*.md5, *.sha):
http://people.apache.org/~jarcec/sqoop-1.99.1-rc0/
The tag to be voted upon:
https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=tag;h=refs/tags/release-1.99.1-rc0
The KEYS file:
http://www.apache.org/dist/sqoop/KEYS
|
Need 3 +1 votes from PMC members.
You can close the vote after voting period expires when you accumulate sufficient votes. Example close email:
No Format |
---|
To: dev@sqoop.apache.org
Subject: [CLOSED] [VOTE] Release Sqoop version 1.99.1
Thanks everyone for voting for this release, I'm closing the vote now. I'll send the results shortly.Thanks everyone for voting for this release, I'm closing the vote now. I'll send the results shortly.
$RM
|
After closing the vote, you need to send results. Example email:
No Format |
---|
To: dev@sqoop.apache.org
Subject: [RESULTS] [VOTE] Release Sqoop version 1.99.1
This vote passes with 8 +1 votes (4 bindings) and no 0 or -1 votes.
+1 votes
PMC Members:
* Bilung Lee
* Olivier Lamy
* Arvind Prabhakar
* Jarek Jarcec Cecho
Committers:
* Cheolsoo Park
* Abhijeet Gaikwad
Community:
* Hari Shreedharan
* Venkatesan Ranganathan
0 votes
* No votes
-1 votes
* No votes
Vote thread:
http://markmail.org/message/faioizetvcils2zo
I'll continue in the release process.
Jarcec
|
Rolling out the Release
...
Close JIRA version
You need to close the release in JIRA so that everyone knows that your version should not be used as "fixVersion" for new bugs. Go to JIRA "Administer project" page and follow "Versions" in left menu. Table with list of all releases should appear, click on additional menu on the right of your release and choose "Release" option. Submit release date and you're done.
Upload the artifacts
Sqoop is using SVN based release procedure. In order to release you have to checkout release repository located on https://dist.apache.org/repos/dist/release/sqoop and add release artifacts there. Note: this step can only be performed by a PMC member.
No Format |
---|
svn co https://dist.apache.org/repos/dist/release/sqoop sqoop-release
cd sqoop-release
mkdir 1.99.1
cp $source_to_your_artifacts 1.99.1/
svn add 1.99.1
# Optionally change KEYS file in case that you've added your key for the first time
svn commit -m "Release 1.99.1"
|
Wait for 24 Hours
It may take up to 24 hours for all mirrors to sync up.
Remove old releases from dist area
Don't worry, they will be archived on archive.apache.org. We need to release the pressure on all Apache mirrors so only most recent releases are allowed to be in the distribution area. Note: this step can only be performed by a PMC member.
Publish binary artifacts in Maven Repository
1. Configure your Apache credentials in your maven settings.xml file (usually in ~/.m2/settings.xml). More information is available in official guide.
Code Block |
---|
<settings>
<servers>
<server>
<id>apache.releases.https</id>
<username>APACHE-ID</username>
<password>APACHE-PASSWORD</password>
</server>
</servers>
</settings> |
2. Go to release branch (tag that was released)
3. Create, sign and deploy artifacts into staging repository
Code Block |
---|
for version in 200; do mvn clean deploy -Psign -Dhadoop.profile=$version ; done
|
4. Login to https://repository.apache.org and select "Staging Repositories" under "Build Promotion" menu on the left.
5. Select org.apache.sqoop from the list of repositories and click "Close" using "Apache Sqoop 1.99.1" as description.
6. Select org.apache.sqoop from the list of repositories and click "Release" using "Apache Sqoop 1.99.1 artifacts" as description
Update webpages
You need to update Sqoop webpages to reflect new release. Instructions are on separate wiki page.
Announce the release
Send an email to announce@apache.org (the from: address must be @apache.org). For example,
No Format |
---|
To: announce@apache.org, user@sqoop.apache.org, dev@sqoop.apache.org
Subject: [ANNOUNCE] Apache Sqoop 1.99.1 released
The Apache Sqoop team is pleased to announce the release of Sqoop 1.99.1.
This is the first cut of Sqoop2 branch. Major step forward of the project.
Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores, such as relational databases.
The release is available here:
http://www.apache.org/dyn/closer.cgi/sqoop/1.99.1
The full change log is available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311320&version=12323641
Your help and feedback is more than welcome. For more information on how to report problems
and to get involved, visit the project website at http://sqoop.apache.org/.
The Apache Sqoop Team
|