I've changed the name of this from 'Release Criteria' to 'Release Branch Criteria' - as I really think they are one and the same. I think that it is vital that we guard master and release branches to ensure that other developers and users do not find them broken. I think that we must proactively prove that changes aren't harmful. I think that the below is what we as a project should demand at all times from our master/release branches. Code that breaks any of the below should never hit master/release branch. 

Think of each step in this process as part of a pipeline - with the end being a 'releasable'; but not necessarily released candidate. 

Each phase gets slightly more complex, and consumes slightly more time. These definitions are fluid, and will change over time. 

The simple stuff:

  • RAT must pass succesfully. 
  • It must successfully build 

The slightly more complex: 

Packaging for debs/rpms works. 
Installation of packages works

Simulator: 

  • Simulator is deployable
  • Complete VM Lifecycle works within the simulator
  • Regions can communicate
  • Service offerings can be created, edited, deleted. 
  • Each deployment planner works - and can deploy VMs and does so accordingly
  • Affinity/anti-affinity settings are honored. 
  • Usage
    • ISO Accounting is correct
    • NAT accounting is correct 
    • Public IP accounting is correct
    • VM accounting is correct
    • Volume accounting is correct
    • VPN accounting is correct
    • Snapshot accounting is correct
    • Template accounting is correct

Hypervisors (hopefully running in parallel) 

KVM (version?/platform?): 

  • Complete VM Lifecycle 
  • Secondary Storage
    • Boots
    • Deploys template
    • Copies snapshot from primary storage
    • Allows snapshot to be downloaded
    • Receives upload of a template
    • Respawned after destruction. 
    • Scales when load applied. 
  • CPVM
    • Boots
    • Permits console access
    • Respawned after destruction
    • Scales when load applied.
  • Virtual Router
    • Boots
    • All interfaces come up and are actually connected appropriately. 
    • Network services
      • LB works with all weighting/routing options
      • Firewall works
      • NAT/Portforwarding works
      • DHCP works
      • Security Groups work
    • RVR works
    • HA works
    • VLAN allocation works (and interacts with the hypervisor) 
    • Security groups work properly. 
    • VPC deployments work
    • Respawned after destruction
  • Snapshotting works
  • Networking

XenServer (which version?) 

  • Complete VM Lifecycle 
  • Secondary Storage** Boots
    • Deploys template
    • Copies snapshot from primary storage
    • Allows snapshot to be downloaded
    • Receives upload of a template
    • Respawned after destruction. 
    • Scales when load applied. 
  • CPVM
    • Boots
    • Permits console access
    • Respawned after destruction
    • Scales when load applied.
  • Virtual Router
    • Boots
    • All interfaces come up and are actually connected appropriately. 
    • Network services
      • LB works with all weighting/routing options
      • Firewall works
      • NAT/Portforwarding works
      • DHCP works
      • Security Groups work
    • RVR works
    • HA works
    • VLAN allocation works (and interacts with the hypervisor) 
    • Security groups work properly. 
    • VPC deployments work
    • Respawned after destruction
  • Snapshotting works
  • Networking

VMware (which version?) 

  • Complete VM Lifecycle 
  • Secondary Storage** Boots
    • Deploys template
    • Copies snapshot from primary storage
    • Allows snapshot to be downloaded
    • Receives upload of a template
    • Respawned after destruction. 
    • Scales when load applied. 
  • CPVM
    • Boots
    • Permits console access
    • Respawned after destruction
    • Scales when load applied.
  • Virtual Router
    • Boots
    • All interfaces come up and are actually connected appropriately. 
    • Network services
      • LB works with all weighting/routing options
      • Firewall works
      • NAT/Portforwarding works
      • DHCP works
      • Security Groups work
    • RVR works
    • HA works
    • VLAN allocation works (and interacts with the hypervisor) 
    • VPC deployments work
    • Respawned after destruction
  • Snapshotting works
  • Networking

LXC 

  • Complete VM Lifecycle 

Baremetal

  • IPMI lifecycle (machine powered on, powered off, network booted, booted from hard drive) 
  • Deployment to machine works
  • Machine cleanup functions well

API

API is tested for consistency with preceding versions to confirm consistency. 

AWS API

  • Installation works
  • Network configured
  • VM lifecycle

Secondary Storage

  • Object storage integration works properly
  • Object storage + NFS integration works properly. 
  • NFS integration works properly.
  • No labels