Page History

...

In the current implementation the overall number of tunnels is quite a limit to the scalability of the overlay network; this because information about each tunnel needs to be stored in the management server DB. The number of tunnels per tenant is also a concern as we need update this part of the state every time a VM is started/migrated/resumed on a ‘new’ host or the last VM on a given host is stopped/paused/moved/terminated. We currently can do a lot of more in terms of managing state updates; there is large room for improvement over the way we do it today as we currently create tunnels serially and wait for a response from the host for each tunnel.

Wiki MarkupThe actual number of tunnels is given by SUM\[i=0...m, ni(ni-1)\] where m is the number of networks and ni is the number of hosts where vms for the i{_}th_ network are deployed. We could calculate the total number of tunnels as a statistical distribution, but this is going to be very complex. The worst case is when VMs for each network are spread across all hosts, in which case we’ll have, for instance, about 50M tunnels per network on 7K hosts. In the best case however, when each network is deployed on a distinct host, we’ll have no tunnels at all. \\ The numbers can look quite scary; the current database structure is, after all, very trivial. It is however useful as it allows us to not create a tunnel between two host if it is already in place.  The size of the state currently grows linearly with the number of networks (tenants), and in a quadratic way with the number of hosts where VMs in a given network are deployed. th network are deployed. We could calculate the total number of tunnels as a statistical distribution, but this is going to be very complex. The worst case is when VMs for each network are spread across all hosts, in which case we’ll have, for instance, about 50M tunnels per network on 7K hosts. In the best case however, when each network is deployed on a distinct host, we’ll have no tunnels at all.
The numbers can look quite scary; the current database structure is, after all, very trivial. It is however useful as it allows us to not create a tunnel between two host if it is already in place.  The size of the state currently grows linearly with the number of networks (tenants), and in a quadratic way with the number of hosts where VMs in a given network are deployed. 

Here are some examples:

So things are obviously much worse as networks start to get spread over a large number hosts. This is typical of scenario with fewer, larger tenants. 
A tradeoff that it might be considered is whether we want to have a single mesh for all networks, and then distinguish each tenant’s traffic within the mesh, or have a distinct mesh for each tenant as we do today.

• Numeric examples:

  Hosts	Networks	Hosts/Network	Tunnels (per tenant mesh)	Tunnels (single mesh)
  500	1000	10	90,000	249,500
  5000	10000	10	900,000	24,995,000
  500	100	100	990,000	249,500
  5000	50	2000	199,900,000	24,995,000

  It is interesting how the single mesh would work better in cases with fewer but larger tenants. Distinguishing each tenant's traffic within the mesh is also a non-negligible problem. VLAN tags can be used, as long as the number of tenants within each mesh is less than 4K. 

...

Here are some example numbers:

...

• Enable Security Groups
  • There are several issues with the current implementation of security groups over OVS networks, mainly concerning the inability of OVS to forward packets for processing to netfilter. Apart from that, it would be better from an architectural point of view to implement security groups within the OVS flow table. 
    The most important issue here is that OVS does not allow to specify port ranges in flow table entries (CIDRs can be used for network addresses). This means that rules which apply to port ranges needs to be translated into multiple flow table entries. 
    This has two consequences: i) managing abstract rules and their mapping into OVS flow entries; ii) a potentially very high number of flow table entries which can cause a consistent number of user-mode/kernel-mode context switches due to the limited size of the kernel-level flow table. At the moment we don't have any solution to the management problem, neither we do have an estimate of the performance impact due to the increased number of rules in the flow table. To cut a long story short, we probably need a prototype implementation to state whether an OVS-based approach for security groups is feasible or not.
    It is worth noting that with the latest release of Open vSwitch, which introduces bitmask-like mechanisms for specifying source and destination port attribute, the total number of rules needs is significantly reduced. 

• Anchor
  bypass bypass

  Bypass the virtual router
  • In a nutshell, this enhancement is aimed at allowing inter-subnet routing without going through the virtual router. Once routing rules have been configured through the API, the OVS flow table can be programmed in order to send packet to the appropriate destination host, even if that host is in a different tunnel mesh (virtual subnet). 
    For more information about inter-network routing, a feature which is currently being implemented on the VLAN backend, please have a look at the following specification from Alena: Inter-VLAN Routing functional spec
• Consider alternatives to GRE overlays
  • GRE does not scale particularly well when the traffic directed to the physical tunnel interface increases. Apart from the performance overhead due to the GRE header, which amount to about 3%, the real issue is the fact that with GRE encapsulation it is not possible to leverage TCP segmentation offload. This means that GRE packets are not fragmented in hardware, but in software. This affects the overall throughput, which decreases as the amount of traffic sent over the tunnel increases. Please see the attached documents (GRE-overhead-analysis.docx, GRE-overhead-analysis-no-TSO.docx) for more information.

...