Introduction
Purpose
This is a draft spec for utilizing Q-in-Q to provide scalable isolated networks in the cloudstack environment.
Q-in-Q, also referred to as "double-tagged" vlans, is the concept of nesting tagged vlans inside each other. Thus, each
of the 4096 available vlans can also host 4096 vlans. This provides a way to scale isolated networks that is more compatible
with standard network equipment and lower overhead than GRE or other technologies that create point-to-point tunnels.
This has been designed and tested on the KVM platform, there's no reason to believe it wouldn't work on others, but the actual implementation would likely vary. More time and expertise will be required to implement other platforms.
Document History
Glossary
Feature Specifications
Use cases
Administrators of businesses who want to use the VPC feature in a scalable way are going to be looking for a way to deploy potentially thousands of what CloudStack refers to as "Isolated Networks". Each VPC requires several, so even a company expecting to have 1,000 customers could reasonably expect to exhaust the 4000 vlan limit of their standard network deployment. This allows you to have 4096 for each vlan in their standard network, potentially 16 million.
Architecture and Design description
A traditional CloudStack advanced network environment might look like the below chart, with management, storage, and a public network created by the admin, and multiple vlans provisioned by cloudstack as needed:
This functional spec extends this design by allowing tagged interfaces to be utilized at the physical interface level:
The admin simply needs to create any 'vlan#' devices, and CloudStack uses them as physical devices.
CloudStack doesn't support defining actual physical devices to be used; instead it uses "traffic labels" to allow the admin to specify which bridges to use for what traffic, and then cloudstack determines the physical devices from those bridges. It then uses those physical devices to create subsequent tagged interfaces/bridges dynamically. Normally, if CloudStack finds that a bridge is on a tagged interface, it then looks up the parent of that interface. A small patch simply keeps CloudStack from looking up the parent if the device is a vlan# device, so that the vlan# dev is instead treated as a physical device and subsequent tagged networks are created there.
This implementation was chosen because it leverages much of the existing CloudStack code. In other words, it's a simple hack. Alternative ways of implementing this feature would include adding the ability for CloudStack to track/configure individual physical devices (my impression is that avoiding this was a conscious decision), which would make the feature independent of the "vlan#" device. This could be as simple as having a 'force-phys-dev' list that we check against.
MTU
Actual MTU values and what is included in them vary slightly between manufacturers and operating systems, but in general it should be kept in mind that Q-in-Q requires extra space in order to contain the new vlan tag, so switches that provide physical connections between KVM hosts need to accomodate that. The table below provides an idea of what MTU values have been tested to work. Generally speaking, an admin should increase the MTU on the applicable switch hardware in order to be compatible with existing MTUs in CloudStack system VMs and guest instances.
Switch MTU |
1500 |
1532 |
9000 |
9032 |
---|---|---|---|---|
Instance MTU 1468* |
Y |
Y |
Y |
Y |
Instance MTU 1500 |
N |
Y |
Y |
Y |
Instance MTU 8968* |
N |
N |
Y |
Y |
Instance MTU 9000* |
N |
N |
N |
Y |
* Special consideration is required for MTU != 1500 for any virtual routers. Will need to add the ability to set MTU in the VR, perhaps in the same way that SSVM gets MTU set in the global settings.
Appendix
Appendix A:
Appendix B: