Introduction

Purpose

This is a functional specification of the Open vSwitch (OVS) controller within CloudStack

References

• Bugzilla reference: http://bugs.cloud.com/show_bug.cgi?id=4407
• Background info on OVS in Citrix Xenserver: http://blogs.citrix.com/2010/08/29/the-open-vswitch-key-ingredient-of-enterprise-ready-clouds/
• Extending CloudStack networking: background info on the design: http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/Extending_CloudStack_Networking

Document History

• v 0.1 Initial cut
• v 0.99 First complete draft (04/04/2012)

Glossary

• OVS: Open vSwitch. Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols 
• SDN: Software Defined Networking. http://en.wikipedia.org/wiki/Software_Defined_Networking
• VLAN: Virtual LAN http://en.wikipedia.org/wiki/Virtual_LAN
• Traffic trombone: http://blog.ioshints.info/2011/02/traffic-trombone-what-it-is-and-how-you.html

Use Cases

1. Cloud operator downloads CloudStack release and installs CloudStack. She wishes to use the GRE tunnel method of isolation. She enables this flag in the configuration database and restarts CloudStack management servers. When she creates a zone with Advanced Networking and creates a physical network within the zone, she is presented with a choice to use GRE isolation
2. Cloud operator wants to enable firewall and load balancing service on top of the GRE-isolated network. She adds a public VLAN and a public IP range to this network. She creates a network offering with these services. End-users now use this offering to create networks.
3. Cloud operator wants to enable firewall services using Juniper SRX on top of the GRE-isolated network. Not supported
4. Cloud operator wants to enable both VLANs and GRE tunnels as isolation methods. Not supported

Feature Specifications

Goals

The goal of this feature is to broadly support all of CloudStack's virtual networking functions while removing the limitations associated with VLANs. Among the limitations are:

• Scaling: a maximum of 4094 VLANs per datacenter is possible. This number is however a theoretical maximum. The actual number of VLANs that can be configured is often limited by the capabilities of the physical switches in the data center, as they need to maintain a distinct forwarding table for each VLAN. 
• Configuration complexity: VLAN information has to be consistently provisioned on all networking hardware
• Broadcast containment: broadcasts within one VLAN causes needless flooding on links not using that VLAN
• Flexibility: Since VLAN are terminated at layer-2, they do not allow to define virtual networks with span across different L2 broadcast domainsunless VLANs are allowed to transverse the aggregation and core layer of the data center, which can cause traffic "tromboning".
  

Unsupported/ToDo items

• TODO moved here: Proposed Enhancements
• The feature currently supports the virtual router only. 

Known issues

• Occasional packet loss / out of order delivery can occur on active connections while GRE tunnels are being created. For TCP connections, this is simply handled with re-transmission, leading to a negligible and temporary network performance degradation.
• The 'enableXenServerNetwork' script sometimes fails, leading to vm start failure detected by CloudStack management server. The HostAllocator picks another host in this case.(bug 
• Part of the GRE key for the network is displayed in the VLAN column (see related bug 14501)

Failure modes

Failures in setting up the bridge, or configuring the GRE tunnels will not cause a failure of the VM startup process. The VM will be started anyway, even if networking might be compromised. When starting a subsequent VM, the tunnel manager will try again to create the tunnels which previously failed.

NOTE: This is the behaviour as currently implemented, but not yet committed. The alternative approach would be to fail VM startup if an error occurs while setting up either the OVS bridge, the tunnels, or the broadcast storm prevention rules. Alternately, a synchronization framework (like the one used by the SecurityGroupManager) can use eventual consistency to (re)create the tunnels.

Logging and Debugging

• For components running in the CS management server (OVS element, OVS tunnel manager, ServerResource), check vmops.log
• For components running on the hypervisor (ovstunnel plugin), log outputs by default to /var/log/ovstunnel.log. Logging on the hypervisor is now configurable, and can be tweaked by changing the appropriate configuration file.

...

• Check for invocations of the _prepare _and _release _methods in com.cloud.network.element.OVSElement. They will invoke methods on the OVS tunnel manager for setting up and tearing down tunnels. 
• The relevant commands classes start with the 'Ovs' prefix. 
• The relevant xapi plugin is ovstunnel. However, most of the broadcast prevention code is in a ovs-vif-flows.pyscript, which is triggered by the hypervisor every time a VIF is plugged or unplugged from a OVS bridge managed by CS OVS controller. They can be easily spotted by querying XS networks for other-config:is-ovs-tun-network.

Configuration Characteristics

The OVS tunnel manager is disabled by default, and should be explicitly enabled in the configuration.
To this aim, the sdn.ovs.controller configuration flag should be set to true.
After enabling it, the management server should be restarted. 

Also, a Vnet range should be configured. Vnet identifiers are used as GRE keys for tunnel networks. The network manager implementation has a check for validating the maximum vnet id. By default this maximum is 4096, unless GRE is explicitly specified as the isolation mode for the physical network on top of which the OVS controller operates.

Deployment requirements

• XenServer 5.6FP1 or newer.
• Open vSwitch networking stack.  Enable it with:echo openvswitch > /etc/xensource/network.conf

  Performance and scalability considerations

The implemented full mesh topology ensures each VM can be reached with at most 1 hop across different hosts. This means that bottleneck issues which are common in star or ring topologies do not occur in this case.
We deliberately avoided using STP for avoiding loops in the traffic. Instead, we prevent issues such as broadcast storm ensuring that broadcast on ingress tunnels are not forwarded on egress tunnels.

...

• GRE overhead, due to encapsulation of the frame in a L3 payload with the GRE header. This overhead amounts to about 3% of the throughput when using standard frames, and to about 0.5% when using jumbo frames.
• Wiki MarkupInability to leverage TSO (TCP segmentation offload). As L4-L7 traffic is encapsulated into L3 (GRE) envelopes, hardware TSO, which is provided by the vast majority of NICs, cannot be leveraged. This is a non-negligible scalability issue as Fragmentation is entirely performed in software. \ [Will add graphs and results from scalability analysis\]. Adopting jumbo frames mitigates this problem.

Security considerations

Preamble: I am not a security expert, nor any security experts has performed a security assessment for this feature.

...

We assume the physical infrastructure to be under exclusive control of the admin of the data center. Cloudstack does not encrypt or perform any operation for protecting the traffic once it has left the host.

Architecture and Design description

Software design and architecture:

...

NOTE: A prototype for this feature was originally developed by Chiradeep; the proposed implementation has been developed starting from that prototype. Not all code relevant to that prototype has been removed, even though it is not anymore in use. When looking at the branch on the git server, please disregard all classes pertaining to OVS not included in the list above.

Web services APIs

No changes introduced to the API

UI flow

Exactly the same as the flow for starting a VM instance.