You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 50 Next »

This document describes how to make a release of Sqoop. It is a work in progress and should be refined by the Release Manager (RM) as they come across aspects of the release process not yet documented here.

NOTE: For the purpose of illustration, this document assumes that the version being released is 1.4.0-incubating, and the following development version will become 1.4.1-incubating.

Prerequisites

Policy documents

The policy on releasing artifacts from an incubating Apache project is stated in the Guide to Release Management During Incubation. If any stated tasks in this document conflict with what is stated in the policy document, it must be brought to the notice of PPMC for further resolution.

The Release Manager (RM) must go through the policy document to understand all the tasks and responsibilities of running a release.

Give a heads up

The RM should first create an umbrella issue and then setup a timeline for release branch point. The time for the day the umbrella issue is created to the release branch point must be at least two weeks in order to give the community a chance to prioritize and commit any last minute features and issues they would like to see in the upcoming release.

The RM should then send the pointer to the umbrella issue along with the tentative timeline for branch point to the user and developer lists. Any work identified as release related that needs to be completed should be added as a subtask of the umbrella issue to allow users to see the overall release progress in one place.

We are starting the process to prepare for Sqoop $release release. I open a JIRA $jira to cover tasks needed for that matter.

If you have any JIRA in progress and would like to get into the release, please follow the process to get it done.


Feel free to comment on the JIRA if you have any comments/suggestions.

Thanks,
$RM

Sanitize JIRA

Before a release is done, make sure that any issues that are fixed have their fix version setup correctly. Run the following JIRA query to see which resolved issues do not have their fix version set up correctly:

project = sqoop and resolution = fixed and fixVersion is empty

The result of the above query should be empty. If some issues do show up in this query that have been fixed since the last release, please bulk-edit them to set the fix version to '1.4.0-incubating'.

You can also run the following query to make sure that the issues fixed for the to-be-released version look accurate:

project = sqoop and resolution = fixed and fixVersion = '1.4.0-incubating'

Monitor active issues

It is important that between the time that the umbrella issue is filed to the time when the release branch is created, no experimental or potentially destabilizing work is checked into the trunk. While it is acceptable to introduce major changes, they must be thoroughly reviewed and have good test coverage to ensure that the release branch does not start of being unstable.

If necessary the RM can discuss if certain issues should be fixed on the trunk in this time, and if so what is the gating criteria for accepting them.

Creating Release Artifacts

Communicate with the community

1. Send an email to sqoop-dev@incubator.apache.org to

  • Notify that you are about to branch.
  • Ask to hold off any commits until this is finished.
I'm about to branch Sqoop for $release release in couple of minutes.  Another email will
be sent out once the branching is done.  Please hold off any commit until
branching is finished.

Just let me know if you have any questions.

Thanks,
$RM

2. Send another email after branching is done.

I am done branching for $release release.  The trunk is open for commit
business again.

$RM

Create a release branch

1. Checkout the trunk.

svn co https://svn.apache.org/repos/asf/incubator/sqoop/trunk

2. Update CHANGES.txt in the trunk.

  • For example,
Apache Sqoop Change Log

Release 1.4.0

  NEW FEATURES

    SQOOP-305 Support export from Avro Data Files

    ...

3. Update the "version" value of build.xml in trunk to "1.4.0-incubating".

4. Commit back to the trunk.

svn commit -m "Preparing for release 1.4.0-incubating"

5. Create a branch.

svn copy https://svn.apache.org/repos/asf/incubator/sqoop/trunk \
  https://svn.apache.org/repos/asf/incubator/sqoop/branches/branch-1.4.0 \
  -m "Branching for 1.4.0 releases"

6. Update the "oldversion" value of build.xml in the trunk to "1.4.0-incubating".

7. Update the "version" value of build.xml in the trunk to "1.4.1-incubating-SNAPSHOT".

8. Commit back to the trunk.

svn commit -m "Preparing for release 1.4.1-incubating development"

9. Tag a candidate, where R (starting from 0) is the iteration number for this candidate:

svn copy https://svn.apache.org/repos/asf/incubator/sqoop/branches/branch-1.4.0 \
  https://svn.apache.org/repos/asf/incubator/sqoop/tags/release-1.4.0-rcR \
  -m "Sqoop 1.4.0-rcR release."

Create release tarballs

1. Check out the candidate.

svn co https://svn.apache.org/repos/asf/incubator/sqoop/tags/release-1.4.0-rc0

2. Make sure that LICENSE.txt contains license information for all third-party libraries and codes to be included in source tarball. You can overwrite with LICENSE.txt from source tarball of previous release if there is no change.

3. Generate the source tarball

ant srctar

4. Make sure that LICENSE.txt contains license information for all third-party libraries and codes to be included in binary tarball. You can overwrite with LICENSE.txt from binary tarball of previous release if there is no change.

5. Generate the binary tarball

ant tar

Perform sanity check

1. Unpack the source tarball

tar xvf sqoop-1.4.0-incubating-src.tar.gz

2. Run tests.

3. Check license header

ant releaseaudit

4. Do the same for the binary tarball.

Sign the release artifacts

Update KEYS file

If your PGP key is not yet in the project's KEYS file, you need to first add that in. To do this, checkout the KEYS file and update it using the following commands:

$ svn co https://svn.apache.org/repos/asf/incubator/sqoop/dist sqoop-dist
...
$ cd sqoop-dist
$ (gpg --list-sigs <KEY-ID> && gpg --armor --export <KEY-ID> ) >> KEYS
$ svn commit -m "Adding PGP public key to KEYS file" KEYS
...

Once this file has been updated, you need to publish it in the appropriate dist directory for the project on http://www.apache.org/dist/incubator/sqoop/KEYS. To do this, you must copy the file as follows:

$ scp KEYS people.apache.org:/www/www.apache.org/dist/incubator/sqoop/KEYS

This will take some time to propagate in which you can continue with the other steps of the release process.

Create signatures

For every release artifact, there should be a corresponding signature file (*.asc) that can be generated using PGP as follows:

$ gpg --armor --output sqoop-1.4.0-incubating-src.tar.gz.asc --detach-sig sqoop-1.4.0-incubating-src.tar.gz

The above command will generate the new file sqoop-1.4.0-incubating-src.tar.gz.asc which contains the signature that can be verified using the following command:

$ gpg --verify sqoop-1.4.0-incubating-src.tar.gz.asc sqoop-1.4.0-incubating-src.tar.gz

Create md5sum

The md5sum allows a user to ensure that the bits they have downloaded are consistent and same as what were produced during the release. You must create md5sum for every artifact that is being released. Use the following command to generate the md5sum:

$ md5sum sqoop-1.4.0-incubating-src.tar.gz > sqoop-1.4.0-incubating-src.tar.gz.md5sum

Stage the release artifacts

In order to allow other developers and contributors to verify the release artifacts, you need to stage them in a place that is accessible to them to download it from. It is preferred if you post these artifacts on your own web-space on people.apache.org. Please make sure that you post all the release artifacts, their signature files (.asc) and their md5sum (.md5sum) files.

Running the vote

Call for sqoop dev list votes

Send an email to sqoop-dev@ list. For example,

To: sqoop-dev@incubator.apache.org
Subject: [VOTE] Release Sqoop version 1.4.0-incubating

This is the first incubator release for Apache Sqoop, version 1.4.0-incubating.

*** Please cast your vote by [3 working days after sending] ***

The list of fixed issues:
http://svn.apache.org/repos/asf/incubator/sqoop/branches/branch-1.4.0/CHANGES.txt

The tarball (*.tar.gz), signature (*.asc), checksum (*.md5),
and test result (log/*.ant_test.log):
http://people.apache.org/~blee/sqoop-1.4.0-incubating-rc0/

The tag to be voted upon:
http://svn.apache.org/repos/asf/incubator/sqoop/tags/release-1.4.0-rc0

The KEYS file:
http://www.apache.org/dist/incubator/sqoop/KEYS

Need 3 +1 votes from PPMC members.

Call for incubator general list votes

Send an email to incubator-general@ list. Please remember that you need to be an member of that list to be able to send the vote (To subscribe, send a message to general-subscribe at incubator dot apache dot org). For example,

To: general@incubator.apache.org
Subject: [VOTE] Release Sqoop version 1.4.0-incubating

This is the first incubator release for Apache Sqoop, version 1.4.0-incubating.

*** Please cast your vote by [3 working days after sending] ***

Dev list vote thread:
http://markmail.org/message/jehsoo2vi6xboovu

The list of fixed issues:
http://svn.apache.org/repos/asf/incubator/sqoop/branches/branch-1.4.0/CHANGES.txt

The tarball (*.tar.gz), signature (*.asc), checksum (*.md5),
and test result (log/*.ant_test.log):
http://people.apache.org/~blee/sqoop-1.4.0-incubating-rc0/

The tag to be voted upon:
http://svn.apache.org/repos/asf/incubator/sqoop/tags/release-1.4.0-rc0

The KEYS file:
http://www.apache.org/dist/incubator/sqoop/KEYS

Need 3 +1 votes from IPMC members.

You can also save time by sending out one email to both sqoop dev list and incubator general list at the same time to call for votes.

Rolling out the Release

Upload the artifacts

ssh people.apache.org
cp -r ~/public_html/sqoop-1.4.0-incubating-rc0 /www/www.apache.org/dist/incubator/sqoop/sqoop-1.4.0-incubating
cd /www/www.apache.org/dist/incubator/sqoop
chgrp -R incubator sqoop-1.4.0-incubating
chmod -R g+w sqoop-1.4.0-incubating
rm stable
ln -s sqoop-1.4.0-incubating stable
chgrp incubator stable
chmod g+w stable
rm -r /www/www.apache.org/dist/incubator/sqoop/sqoop-$PREVIOUS_VERSION-incubating # remove older versions

Wait for 24 Hours

It may take up to 24 hours for all mirrors to sync up.

Announce the release

Send an email to announce@apache.org (the from: address must be @apache.org). For example,

To: announce@apache.org, sqoop-user@incubator.apache.org, sqoop-dev@incubator.apache.org
Subject: [ANNOUNCE] Apache Sqoop 1.4.0-incubating released

The Apache Sqoop team is pleased to announce the release of Sqoop 1.4.0-incubating
from the Apache Incubator.

This is the first incubator release of Apache Sqoop, a tool designed for efficiently
transferring bulk data between Apache Hadoop and structured datastores, such as
relational databases.

The release is available here:
http://www.apache.org/dyn/closer.cgi/incubator/sqoop/

The full change log is available here:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311320&version=12317345

Your help and feedback is more than welcome. For more information on how to report problems
and to get involved, visit the project website at http://incubator.apache.org/sqoop/.

The Apache Sqoop Team

Publish binary artifacts in Maven Repository

These instructions will change when Sqoop moves to maven builds

Currently Sqoop is built using ant and thus the process necessary for publishing its binary artifacts in Maven repository is manual. Once Sqoop migrates to using Maven build (which is the default in Sqoop 2), these steps will need to be updated accordingly.

Sqoop already has a Nexus repository setup, thanks to INFRA-4310. The following steps outline how you can publish Sqoop binaries to this repository.

Setup Maven settings

Follow the details provided in Maven Password Encryption Guide to set up a master password for your system.

Once the master password has been created, update your maven settings file (~/.m2/settings.xml) to have the following server entries:

<settings>
  ...
  <servers>
    <server>
      <id>apache.snapshots.https</id>
      <username>your-asf-user-name</username>
      <password>your-encrypted-asf-password</password>
    </server>
    <server>
      <id>apache.releases.https</id>
      <username>your-asf-user-name</username>
      <password>your-encrypted-asf-password</password>
    </server>
    <server>
      <id>apache.staging.https</id>
      <username>your-asf-user-name</username>
      <password>your-encrypted-asf-password</password>
    </server>
  </servers>
  ...
</settings>

Prepare artifacts to publish

In a newly created directory, checkout the sources from the appropriate release tag location. For example, for the release 1.4.0-incubating the sources will need to be checked out from https://svn.apache.org/repos/asf/incubator/sqoop/tags/release-1.4.0-rc1/. Once you have the sources, build them to generate the binaries. For example:

$ svn co https://svn.apache.org/repos/asf/incubator/sqoop/tags/release-1.4.0-rc1/
...
$ cd release-1.4.0-rc1

$  ant jar-all
...
BUILD SUCCESSFUL
Total time: 11 seconds
$

Install to local maven cache

Before you do this step, remove any previously installed Sqoop artifacts from your local maven cache. You can do this by running the following command:

$ rm -rf ~/.m2/repository/com/cloudera/sqoop ~/.m2/repository/org/apache/sqoop

Now run the following command to install the newly checked-out and built artifacts of Sqoop from the previous step to your local maven cache.

$ ant mvn-install
...
BUILD SUCCESSFUL
Total time: 6 seconds
$

This will result in copying of the built artifacts and generated POM file into your local maven cache. Currently there is a bug in the build system that results in the copying of these artifacts with namespace com.cloudera.sqoop instead of org.apache.sqoop. This is tracked by issue SQOOP-426.

Deploy main artifact to staging repo

Once the artifacts are in your local cache, you can now deploy them to Nexus staging repository.

Note: If the issue SQOOP-426 is not fixed yet, the artifacts will be available in the directories /.m2/repository/com/cloudera/sqoop/sqoop/1.4.0-incubating and /.m2/repository/com/cloudera/sqoop/sqoop-test/1.4.0-incubating. In this case, you will have to manually modify the generated pom file ~/.m2/repository/com/cloudera/sqoop/sqoop/1.4.0-incubating/sqoop-1.4.0-incubating.pom to change the groupId from com.cloudera.sqoop to org.apache.sqoop.

Start out by going into the directory where the main artifact is generated. Once there, make sure that the generated POM file looks accurate and if not, fix it manually. Then run the following commands to deploy the main artifact.

$ cd ~/.m2/repository/org/apache/sqoop/sqoop/1.4.0-incubating

$ ls -l
total 1108
-rw-r--r-- 1 arvind staff 532981 Jan 11 16:32 sqoop-1.4.0-incubating-sources.jar
-rw-r--r-- 1 arvind staff 590086 Jan 11 16:32 sqoop-1.4.0-incubating.jar
-rw-r--r-- 1 arvind staff   1670 Jan 11 16:32 sqoop-1.4.0-incubating.pom
$ mvn gpg:sign-and-deploy-file -Dfile=./sqoop-1.4.0-incubating.jar -DrepositoryId=apache.staging.https -Durl=https://repository.apache.org/service/local/staging/deploy/maven2/ -DpomFile=./sqoop-1.4.0-incubating.pom -Dgpg.keyname=<your-gpg-keyname>
...
[INFO] Metadata[project org.apache.sqoop:sqoop].filename = sqoop-1.4.0-incubating.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.169s
[INFO] Finished at: Wed Jan 11 16:58:35 PST 2012
[INFO] Final Memory: 6M/81M
[INFO] ------------------------------------------------------------------------

This will result in the signing and deployment of the main artifact sqoop-1.4.0-incubating.jar in this case, to the staging repository. If all goes well you should be able to point your browser to Apache Nexus Repository and clicking on the Staging Repository link on the left navigation column. When you select the staging repository in the right pane, the pane will split into a content pane below that shows how the content of the staging repository created by the above command. The following screenshot illustrates what this looks like.

Deploy test artifact to staging repo

To deploy the test artifact to the staging repo, you must copy the generated test artifact to the same directory as that of the main directory so that you can use the main artifact POM and specify a classifier. Here are the commands to do that:

$ ls -l
total 1116
-rw-r--r-- 1 arvind staff 532981 Jan 11 16:32 sqoop-1.4.0-incubating-sources.jar
-rw-r--r-- 1 arvind staff 590086 Jan 11 16:32 sqoop-1.4.0-incubating.jar
-rw-r--r-- 1 arvind staff    833 Jan 11 16:58 sqoop-1.4.0-incubating.jar.asc
-rw-r--r-- 1 arvind staff   1668 Jan 11 16:58 sqoop-1.4.0-incubating.pom
-rw-r--r-- 1 arvind staff    833 Jan 11 16:58 sqoop-1.4.0-incubating.pom.asc

$ cp ../../sqoop-test/1.4.0-incubating/sqoop-test-1.4.0-incubating.jar .

$ ls -l
total 1392
-rw-r--r-- 1 arvind staff 532981 Jan 11 16:32 sqoop-1.4.0-incubating-sources.jar
-rw-r--r-- 1 arvind staff 590086 Jan 11 16:32 sqoop-1.4.0-incubating.jar
-rw-r--r-- 1 arvind staff    833 Jan 11 16:58 sqoop-1.4.0-incubating.jar.asc
-rw-r--r-- 1 arvind staff   1668 Jan 11 16:58 sqoop-1.4.0-incubating.pom
-rw-r--r-- 1 arvind staff    833 Jan 11 16:58 sqoop-1.4.0-incubating.pom.asc
-rw-r--r-- 1 arvind staff 278892 Jan 11 17:06 sqoop-test-1.4.0-incubating.jar

$ mvn gpg:sign-and-deploy-file -Dfile=./sqoop-test-1.4.0-incubating.jar -DrepositoryId=apache.staging.https -Durl=https://repository.apache.org/service/local/staging/deploy/maven2/ -DpomFile=./sqoop-1.4.0-incubating.pom -Dgpg.keyname=<your-gpg-keyname> -Dclassifier=tests
...
[INFO] Metadata[project org.apache.sqoop:sqoop].filename = sqoop-1.4.0-incubating.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.160s
[INFO] Finished at: Wed Jan 11 17:10:01 PST 2012
[INFO] Final Memory: 6M/81M
[INFO] ------------------------------------------------------------------------
$

If this command succeeds, you can now see the test artifacts deployed in the staging repository as well. Go the the browser and click on the refresh button on the content area to reload the contents of the staging repository. The following screen shot is an example of how the staging repository may look like after these steps.

Close and Release staging repository

In the Nexus main content pane, make sure the staging repository you are working with is selected with a check mark, and click the "Close" button on top. This causes Nexus to verify the signatures and if everything is valid, it will result in successful closing of the repository. If there are any failures due to invalid signatures, you will have to drop the repository and redo the steps outlined in this section again.

Once the repository is closed, you can promote it to release status by clicking on the "Release" button on top. This allows the artifacts in this staging repository to be published to release repository immediately. Once the release has propagated, you should receive a confirmation email from Nexus stating that the artifacts are now available in the release repository.

  • No labels