I've changed the name of this from 'Release Criteria' to 'Release Branch Criteria' - as I really think they are one and the same. I think that it is vital that we guard master and release branches to ensure that other developers and users do not find them broken. I think that we must proactively prove that changes aren't harmful. I think that the below is what we as a project should demand at all times from our master/release branches. Code that breaks any of the below should never hit master/release branch. 

Think of each step in this process as part of a pipeline - with the end being a 'releasable'; but not necessarily released candidate. 

Each phase gets slightly more complex, and consumes slightly more time. These definitions are fluid, and will change over time. 

The simple stuff:

• RAT must pass succesfully. 
• It must successfully build 

The slightly more complex: 

Packaging for debs/rpms works. 
Installation of packages works

Simulator: 

• Simulator is deployable
• Complete VM Lifecycle works within the simulator
• Regions can communicate
• Service offerings can be created, edited, deleted. 
• Each deployment planner works - and can deploy VMs and does so accordingly
• Affinity/anti-affinity settings are honored. 
• Usage
  • ISO Accounting is correct
  • NAT accounting is correct 
  • Public IP accounting is correct
  • VM accounting is correct
  • Volume accounting is correct
  • VPN accounting is correct
  • Snapshot accounting is correct
  • Template accounting is correct

Hypervisors (hopefully running in parallel) 

KVM (version?/platform?): 

• Complete VM Lifecycle 
• Secondary Storage
  • Boots
  • Deploys template
  • Copies snapshot from primary storage
  • Allows snapshot to be downloaded
  • Receives upload of a template
  • Respawned after destruction. 
  • Scales when load applied. 
• CPVM
  • Boots
  • Permits console access
  • Respawned after destruction
  • Scales when load applied.
• Virtual Router
  • Boots
  • All interfaces come up and are actually connected appropriately. 
  • Network services
    • LB works with all weighting/routing options
    • Firewall works
    • NAT/Portforwarding works
    • DHCP works
    • Security Groups work
  • RVR works
  • HA works
  • VLAN allocation works (and interacts with the hypervisor) 
  • Security groups work properly. 
  • VPC deployments work
  • Respawned after destruction
• Snapshotting works
• Networking

XenServer (which version?) 

• Complete VM Lifecycle 
• Secondary Storage** Boots
  • Deploys template
  • Copies snapshot from primary storage
  • Allows snapshot to be downloaded
  • Receives upload of a template
  • Respawned after destruction. 
  • Scales when load applied. 
• CPVM
  • Boots
  • Permits console access
  • Respawned after destruction
  • Scales when load applied.
• Virtual Router
  • Boots
  • All interfaces come up and are actually connected appropriately. 
  • Network services
    • LB works with all weighting/routing options
    • Firewall works
    • NAT/Portforwarding works
    • DHCP works
    • Security Groups work
  • RVR works
  • HA works
  • VLAN allocation works (and interacts with the hypervisor) 
  • Security groups work properly. 
  • VPC deployments work
  • Respawned after destruction
• Snapshotting works
• Networking

VMware (which version?) 

• Complete VM Lifecycle 
• Secondary Storage** Boots
  • Deploys template
  • Copies snapshot from primary storage
  • Allows snapshot to be downloaded
  • Receives upload of a template
  • Respawned after destruction. 
  • Scales when load applied. 
• CPVM
  • Boots
  • Permits console access
  • Respawned after destruction
  • Scales when load applied.
• Virtual Router
  • Boots
  • All interfaces come up and are actually connected appropriately. 
  • Network services
    • LB works with all weighting/routing options
    • Firewall works
    • NAT/Portforwarding works
    • DHCP works
    • Security Groups work
  • RVR works
  • HA works
  • VLAN allocation works (and interacts with the hypervisor) 
  • VPC deployments work
  • Respawned after destruction
• Snapshotting works
• Networking

LXC 

• Complete VM Lifecycle 

Baremetal

• IPMI lifecycle (machine powered on, powered off, network booted, booted from hard drive) 
• Deployment to machine works
• Machine cleanup functions well

API

API is tested for consistency with preceding versions to confirm consistency. 

AWS API

• Installation works
• Network configured
• VM lifecycle

Secondary Storage

• Object storage integration works properly
• Object storage + NFS integration works properly. 
• NFS integration works properly.