Page History

...

In the current implementation the overall number of tunnels is quite a limit to the scalability of the overlay network; this because information about each tunnel needs to be stored in the management server DB. The number of tunnels per tenant is also a concern as we need update this part of the state every time a VM is started/migrated/resumed on a ‘new’ host or the last VM on a given host is stopped/paused/moved/terminated. We currently can do a lot of more in terms of managing state updates; there is large room for improvement over the way we do it today as we currently create tunnels serially and wait for a response from the host for each tunnel.

Wiki MarkupThe actual number of tunnels is given by SUM\[i=0...m, ni(ni-1)\] where m is the number of networks and ni is the number of hosts where vms for the i{_}th_ network are deployed. We could calculate the total number of tunnels as a statistical distribution, but this is going to be very complex. The worst case is when VMs for each network are spread across all hosts, in which case we’ll have, for instance, about 50M tunnels per network on 7K hosts. In the best case however, when each network is deployed on a distinct host, we’ll have no tunnels at all. \\ The numbers can look quite scary; the current database structure is, after all, very trivial. It is however useful as it allows us to not create a tunnel between two host if it is already in place.  The size of the state currently grows linearly with the number of networks (tenants), and in a quadratic way with the number of hosts where VMs in a given network are deployed. th network are deployed. We could calculate the total number of tunnels as a statistical distribution, but this is going to be very complex. The worst case is when VMs for each network are spread across all hosts, in which case we’ll have, for instance, about 50M tunnels per network on 7K hosts. In the best case however, when each network is deployed on a distinct host, we’ll have no tunnels at all.
The numbers can look quite scary; the current database structure is, after all, very trivial. It is however useful as it allows us to not create a tunnel between two host if it is already in place.  The size of the state currently grows linearly with the number of networks (tenants), and in a quadratic way with the number of hosts where VMs in a given network are deployed. 

Here are some examples:

So things are obviously much worse as networks start to get spread over a large number hosts. This is typical of scenario with fewer, larger tenants. 
A tradeoff that it might be considered is whether we want to have a single mesh for all networks, and then distinguish each tenant’s traffic within the mesh, or have a distinct mesh for each tenant as we do today.

• Numeric examples:

  Hosts	Networks	Hosts/Network	Tunnels (per tenant mesh)	Tunnels (single mesh)
  500	1000	10	90,000	249,500
  5000	10000	10	900,000	24,995,000
  500	100	100	990,000	249,500
  5000	50	2000	199,900,000	24,995,000

  It is interesting how the single mesh would work better in cases with fewer but larger tenants. Distinguishing each tenant's traffic within the mesh is also a non-negligible problem. VLAN tags can be used, as long as the number of tenants within each mesh is less than 4K. 

...

Here are some example numbers:

...