Content moved from 'working with' (todo: integrate)

You can send in diffs of your changes to the Cloudstack development mailing list, where your diffs will be reviewed and committed to the develop branch.

The master git branch is what Apache Cloudstack developers work with. To clone this branch, you will need to do -

git clone https://git-wip-us.apache.org/repos/asf/cloudstack.git

You may want to pull a specific sub-branch of cloudstack-oss. To do this, you will need to first pull the cloudstack-oss branch using the git clone command above, and then, do -

       git checkout <sub_branch_you_want_to_pull>

For example, 3.0.1 is the sub branch that was released to the Apache Software Foundation, so you would pull this branch using -

       git checkout vpc

If you ever need to clean up temp files not tracked by git that git may complain about when pulling sub branches or changing git branches, execute the command below (but please backup any of these files if you need them since this command will wipe out any files not tracked by git!) -

       git clean -d -fx ""

NOTE: If you issue the above clean command, your build/override/ directory would be removed, so you would need to re-create it. Refer to the section "How to build the cloudstack code" below to see how to create this directory and what to put into it.

Say you wish to make changes to your local sub-branch, it is always a good idea to checkout another branch from your newly created local sub-branch. 

Let's summarize all the typical steps you will need to carry out, below - 

Step 1) Go to the master -

        git checkout master

Step 2) Either make your changes on this branch (for a simple bugfix patch), or, create a new local branch from master -

$ git checkout -b my-feature
Switched to a new branch 'my-feature'

Step 3) Confirm that you are on your new my-feature branch -

$ git branch
master 
* my-feature

Step 4) Make your changes to the new my-feature branch, and git commit your changes.

Step 5) Ready your patch for submission upstream

If you did this on the master branch:

1. •  $ git format-patch origin   # this will provide a patch for each commit that is on your local copy of the branch that isn't upstream.    

If you did this on a topic branch

• • $ git format-patch origin master # this will provide a patch for each commit that's on your local branch that isn't on upstreams master

Step 6) Submit your patches for review. 

There are three different methods by which patches may be submitted upstream. 

• Create a review request in ReviewBoard and upload your patches
• Use git-send-email to the cloudstack-dev mailing list.
• Create a bug in Jira and upload your patch to that bug

Other useful git commands

git stash

  - Lets you stash away the changes if you don't want to commit them yet but want to change your branch

git stash apply

git stash drop

git status

Quick Start

Here is how to kick start feature development:

1. Create an account on GitHub.
   See Setting Up Git
2. Fork the Apache Repo:
   1. Go to https://github.com/apache/cloudstack.
   2. Click Fork.
      See Fork a Repo
3. Clone your fork:
       git clone --recursive https://github.com/<username>/cloudstack.git
4. If you use git-flow, initialize it with the defaults
       git flow init
5. Configure remote:
       git remote add upstream https://git-wip-us.apache.org/repos/asf/cloudstack.git
   The GitHub repo is a mirror and not the definitive source, so we use https://git-wip-us.apache.org/repos/asf/cloudstack.git and not https://github.com/apache/cloudstack
6. Do your work on a feature or bugfix branch
       git checkout -b <feature-branch>
or if you use git-flow
       git flow feature start <feature>
7. Sync from upstream periodically (for long running features, read:daily)
       git checkout <feature-branch>
This sets your local working set to the branch you want to pull the latest changes into.
       git fetch upstream
   This fetches remote changes into the git index but not yet into the checkout.
       git rebase upstream/<feature-branch>
   This rewrites your local changes on top of the latest upstream.
   Don't rebase if you have pushed your local changes somewhere, instead merge:
       git merge upstream/<feature-branch> --no-ff
8. Submit your changes
   1. As a non-committer, when you have some code to contribute, submit changes to the Review Board
      1. Create an Apache Review Board account at https://reviews.apache.org
      2. Install RBTools
      3. Set the default reviewboard.url, e.g. git config reviewboard.urlhttps://reviews.apache.org
      4. Call post-review
      5. Edit the resulting URL to assign a reviewing group and to describe the change.
         RBTools automates creating a diff file that captures the changes you've made and publishing them to the ReviewBoard site.
   2. As a committer, you can follow the above process for larger commits, if you haven't gotten review from someone else yet. Otherwise, submit changes upstream
          git push upstream <feature-branch>
9. Finish the feature
   After testing of the new feature has finished, it has been integrated into develop (and any release branches), and once you are confident the feature branch won't be needed anymore, you can remove the branch locally
       git branch -d <feature-branch>
   or with git-flow
       git flow feature finish <feature>

Committers will not be looking at your repo by default. Therefore, alert the cloudstack-dev mailing list of your work and periodically request the incremental commits you make be reviewed.

It is also best to work on feature branches rather than working directly on develop, even in your private fork. When the patch is ready for submission push the squashed patch through reviewboard. If the patch is sizeable and touches many projects in the repo, best to break it down into smaller chunks for easier and quicker review by multiple committers. Don't use github's pull request feature, but in the reviewboard patch notes, you can include the upstream you request a committer to pull from to preserve useful history.

Holler for help on the lists if you're stuck and ever need help.

Using Git in CloudStack

Git Config

# Set the name of the user for all git instances on the system
git config --global user.name "Firstname Lastname"
# Set the email-address of the user for all git instances on the system
git config --global user.email "your_email@youremail.com"

Committer workflow

Your own work can be committed easily - git push is all you need. Be mindful of the branch that you are working in and ensure that it is the current branch for development. Please use feature branches for bigger changes, and support/hotfix branches for bugfixes that have to go into multiple releases. Please be mindful that just because you have commit privileges does not relieve you of the obligation to develop in the open, which means communication on the cloudstack-dev list.

Using the git-flow command line tools is recommended but completely optional.

Handling patches

As a committer one of your responsibilities is to review and commit patches from others. The patches must pass through reviewboard, the bug tracker or the mailing list so they can be tracked and so they can be considered a CONTRIBUTION per the apache license/CLA.

Patches should be applied via git-am (in other words, don't just use a diff, it records no author information). 

Move to git-flow

Discussion thread | Vote thread

After the 4.4.0 release, cloudstack has decided to move to (a variant of) the git-flow branching model. The switch was made XXTBC august, 2014. The setup is as follows:

This is the proposal for first cut

1. branch develop from master
2. branch release/4.5 from the develop
3. 4.3.x workflow does not change and continues to built using cherry-picks from 4.4/master/develop
   (4.3.1 should be last 4.3.x) 
4. 4.4.x workflow considers 4.4 the long-term support branch

Creating a hotfix

A hotfix is a bugfix that is done to an existing release and/or release branch.

Hotfix for 4.4.x

Since 4.4 can't be merged to/from master, bugfixes to 4.4 need to be done twice, once for 4.4 and once for 4.5 and beyond:

1. branch hotfix/4.4-<jira-ticket> from 4.4
2. write fix
3. branch hotfix/<jira-ticket> from release/4.5
4. write/port fix (use git cherry-pick or do by hand)
5. push both fixes, integration test both
6. call for 4.4 release manager (Daan) to merge hotfix/4.4-<jira-ticket> into 4.4
7. TBC do all committers merge to release, one of

1. 1. call for 4.5 release manager to merge hotfix/<jira-ticket> into release/4.5
   2. merge hotfix/<jira-ticket> into release/4.5 and push
2. merge release/4.5 into develop

For bugfixes that don't need to go into 4.5.x, normal git-flow can be followed.

Hotfix for 4.5.x and beyond

In git-flow it is customary to delete release branches after a release. We won't do that. Instead minor releases get a long-term support (LTS) release branch. This means it will be a bit easier to create patch releases for multiple releases.

Imagine we have released 4.5.0 and 4.6.0, and we want to do a hotfix that is to be part of 4.5.1 and 4.6.1:

1. branch hotfix/<jira-ticket> from release/4.5
       git flow hotfix start <jira-ticket> release/4.5  
2. write fix, smoke test, commit, push, integration test
       mvn -P ... install && ... smoke test ...
    git commit -a
    git push upstream hotfix/<jira-ticket>
3. merge to release/4.5
   TBC do all committers merge to release, one of

1. 1. call for 4.5 release manager to merge hotfix/<jira-ticket> into release/4.5
   2. merge hotfix/<jira-ticket> into release/4.5 and push
          git checkout release/4.5
    git pull --rebase
    git merge --no-ff --edit hotfix/<jira-ticket>
    git push upstream release/4.5
2. merge to release/4.6
   same procedure as for release/4.5 
3. merge release/4.6 to develop
    git checkout develop
    git pull --rebase
    git merge --no-ff --edit release/4.6
    git push upstream develop
while it should be possible to merge your hotfix branch into develop, the possibility exists that other hotfixes were already merged into the release but not yet into develop. If you merge your hotfix but not those previous hotfixes, comparing the history of release and develop branches gets a bit more difficult. So, we merge all the hotfixes on 4.6 to develop.

For bugfixes that don't need to go into 4.5.x, normal git-flow can be followed (see below).

Calling git merge with --no-ff ensures a merge commit. This commit documents the history that there was a specific hotfix.

Calling git merge with --edit allows editing the merge commit message. Use this to add any comments about the application of this hotfix to this release.

Writing a new feature

See above under 'Quick Start'. This should be the default way of working for most changes.

Writing a new test for existing features

New features should come with tests and those should go onto the feature branch.

When improving test coverage for existing functionality, you should still create a feature branch if there's a lot to the test (moving code around, refactoring to reuse existing code, changes to test tools, etc). For simply adding a new test, see below.

Simple bugfix or improvement or test case

Fixing a simple bug that

• does not need to be backported to any other release
• consists of one reasonable-sized (reviewable) commit
• is unlikely to destabilize the build

Can be done directly on develop.

However, since when you start the bugfix you typically don't know exactly what the fix will be, it's simply good practice to create a local bugfix branch anyway. Basically, whenever you do a context-switch to pick up a new task or JIRA issue, you probably want to branch. The workflow difference in this case is that the branch you create is local to your machine; you never push it upstream. For example:

Start work:

    git checkout develop
    git pull --rebase
    git checkout -b bugfix/<jira-ticket>
    ...write test...
    git commit -a
    ...code code code...
    git commit -a
    ...test fix test...
    git commit -a

Check that you're done:
    mvn -P ... install && ... smoke test ...

Cleanup and make neat clean commit(s)

    git pull
    git rebase -i develop
    git commit --amend
    git checkout develop
    git merge --ff-only bugfix/<jira-ticket>
    git branch -d bugfix/<jira-ticket>

Push the changes:

    git push upstream develop

Release management

Besides the normal git-flow commands, remember to update version numbers (in maven pom.xml, in marvin setup.py, in systemvm scripts, ...should have a confluence page...).

Starting a new x.y.0 minor release

    git checkout develop
    git pull --rebase
    git flow release start x.y develop
    ...update version number to x.y.0-SNASPHOT...
    git push upstream release/x.y
    git checkout develop
    ...update version number to x.y'-SNAPSHOT...
    git commit -a
    git push upstream develop

Finishing a x.y.0 release

Don't use git flow release finish, it will delete the release branch.

    git checkout release/x.y
    git pull --rebase
    ...update version number to x.y.z...
    git push upstream release/x.y
    git checkout master
    git pull --rebase
    git merge --no-ff --edit release/x.y
    git tag -a vx.y
    git push upstream master
    git push --tags
    git checkout release/x.y
    ...update version number to x.y.z'-SNAPSHOT...

Starting a minor x.y.z (patch/hotfix) release

This is very similar to starting minor release, but you start from the release branch, not from the develop branch.

    git checkout release/x.y
    git pull --rebase
    git flow release start x.y.z release/x.y
    git push upstream release/x.y.z
    git checkout release/x.y
    ...update version number to x.y.z'-SNAPSHOT...
    git push upstream release/x.y

Finishing a minor x.y.z (patch/hotfix) release

This is very similar to finishing a minor release.

Reviewer guidelines

1. Every commit should have unit tests
2. every feature/merge request should have unit and marvin integration tests
3. A commit should not have check style or find bugs issues
4. Any coverity issues reported in the new code should be addressed immediately
5. Developer should run the BVT on the simulator before doing a checkin 
   1. Validating check-ins for your local changes, using Simulator

Non-committer workflow

Cloning

First you need to have a copy of the source - you should be able to acquire that by cloning the repo:

$ git clone https://git-wip-us.apache.org/repos/asf/cloudstack.git

Branch

The CloudStack repo has multiple branches, some version specific, you'll need to figure out which one you need to work with, but we'll make the assumption that master is OK for this purpose. Now, when you are ready to actually hack on CloudStack you should create your own topic branch locally. Lets say you wish to work on a specific bug, then you should work on that bug (and only that bug) in its own local branch (known as a topic branch), and you should use a descriptive name for it.

Suppose you are working on bug CS-15081 - you'd create a branch by the same name: $ git checkout -b CS-15081   (this both creates the branch and switches to using it) 

Now you can make your changes - and once done submit patches. 

Commit Messages

Please make your commit messages descriptive:

• Commit messages should be clear, concise and provide a reasonable summary to give an indication of what was changed and why.

• Commit messages should provide enough information to enable a third party to decide if the change is relevant to them and if they need to read the change itself.

• Avoid committing several unrelated changes in one go. It makes merging difficult, and also makes it harder to determine which change is the culprit if a bug crops up.

• Avoid committing style or whitespace fixes and functionality fixes in one go. It makes merging difficult, and also makes it harder to understand just what functional changes were made. In the case of documentation files, it can make the job of the translation teams more complicated, as it becomes difficult for them to determine exactly what content changes need to be translated.

• Avoid committing changes to multiple files in one go with a generic, vague message. Instead, commit each file (or small, related groups of files) with tailored commit messages.

A good commit message should look like this:

Header line: Explain the commit in one line

Body of commit message is a few lines of text, explaining things in more detail,

possibly giving some background about the issue being fixed, etc etc. Use bullets if possible:
 - Line 1
 - Line 2 etc.
 * Start works too, instead of -

The body of the commit message can be several paragraphs, and please do proper word-wrap

and keep columns shorter than about 80 characters or so. That way "git log" will show things

nicely even when it's indented.

Reviewed-by: Individuals who reviewed, or link to review on review.apache.org
Reported-by: whoever-reported-it, if applicable (usually this is recorded in the Jira bug)
Submitted-by: give credit to other individual for any patch submission committed by you
Signed-off-by: Your Name <youremail@yourhost.com>

Note that there exists a git hook to prepare a commit form for you, located at tools/git/prepare-commit-msg. One simply needs to link to it in order to use it:

ln -s ../../tools/git/prepare-commit-msg .git/hooks/prepare-commit-msg

Please use a logical prefix, which should be:

• CLOUDSTACK-XXX-bug-id prefix: Explain the bugfix in one line; This is for only bug fixes.
• Maven: If this changes the build system
• Doc: If this is doc related change
• awsapi: If the code changes several things inside a subproject, use artifactId sans the cloud- prefix such as: plugin-netapp; this should be lowercase
• NameOfTheClass: If this patch purely changes to ThisClass, this should be CamelCase

Patches

Patches encapsulated all the changes you want to make to another branch. Patches are used to communicate the changes that you want incorporated into Apache CloudStack.

Creating patches

Patches should be created with git-format-patch - and you should specifically be basing that against the branch you are targeting for inclusion, so for instance using our above example from the CS-15081 branch we'd run:

$ git format-patch -s master

The above would produces a patch file for each commit in the directory (unless you specified one with -o /path/to/patch/folder).
If you've a series of commits that you want to put into a single file, try

$ git format-patch -s master --stdout > ./mycommits.patch

Sending patches

CloudStack currently doesn't utilize Github pull requests.

One alternative is to post the patch on Review Board.

1. Create an Apache Review Board account at https://reviews.apache.org
2. Install RBTools
3. Set the default reviewboard.url, e.g. git config reviewboard.url https://reviews.apache.org
4. Call post-review
5. Edit the resulting URL to assign a reviewing group and to describe the change.
   RBTools automates creating a diff file that captures the changes you've made and publishing them to the ReviewBoard site. The drawback is that you loose the meta data associated with each commit that you've made. Howevever, Apach Review Board can't handle a commit that references the same file multiple times.

More materical on submitting a patch to Review Board

FIXME: This needs checking

An alternative to Review Board is to email that patch to the cloudstack-dev mailing list with [PATCH] in the subject line.

You should expect some feedback on your patch within a week. If you don't receive any - please reply to the original message to make sure folks didn't miss it.

Sending patches can be done with the git send-email tool:

$ git send-email your-newly-generated.patch

How to setup git-send-email tool:

• yum install git-email or sudo apt-get install git-email (this will install git send-email tool)

The below will configure your .gitconfig globally:

• git config --global sendemail.smtpserver <smtp-server-name>
• git config --global sendemail.to <emailing-list>
• git config --global sendemail.from <email-id>

Apply patches

If the patch to be applied gets the commit details in the header, then use

• git am < <patch> to apply the patch, which will automatically add newly added files to tracked files.
• git commit -as --author='Foo Bar <foo@bar.com>' for proper commit attribution

If it's just a raw patch, then use

• git apply <patch>
• git add <new files>
• git commit -as --author='Foo Bar <foo@bar.com>' for proper commit attribution

Words of wisdom

Never use the --force

Git assumes that if you have commit privileges that you are to be trusted, and generally this is a good thing. However, occasionally people are lured by the dark side of the force and when something doesn't merge cleanly, or their push doesn't work - they are tempted to whip out the --force and make it work. PLEASE NEVER DO THIS. It's is almost universally wrong. Please tell us about your problem on cloudstack-dev and let us help you fix it. 

Line endings

The coding standard for CloudStack says that unix line endings (LF) are used instead of CRLF (Windows line endings). You should set your editor and git configuration to behave properly. 

For more information: 

http://help.github.com/line-endings/

http://git-scm.com/docs/git-config

Fixing things for folks

In short, please don't. If non-committers have a patch 95% of the way - comment and tell them what is necessary to make the patch acceptable. Let them fix their own patch and resubmit. Yes you can probably fix things more quickly, but it's important for community growth (both in number and experience) that folks do this themselves.