Page History

*

*This guide is geared

 more towards *{color:#99cc00}{*}CS 4.0{*}{color} *(build 140) and* {color:#99cc00}{*}vSphere5{*}{color}*, however, the same concept should apply to other hypervisors types like KVM and XEN.* *This is

* {color:#99cc00}{*}NOT{*}{color} *a beginners guide,* *{+}please use your judgement and substitute values as necessary{+}{*}*.*


h2. *Disclaimer*


h6. The {color:#ff0000}CS4.0 build 140{color} is defined on purpose in this document - there are known UI issues that have been already addressed - however, since no update/patch has been officially released to address these issues (at the time of this writing)  \- in order to configure Advanced Networking - we must use API calls to substitute what GUI could not accomplish. The newer releases should address the UI limitations and API calls may not be necessary. Nevertheless, its a good learning experience.


h5.


h2. *What This Guide Covers*

* CloudStack Advanced Network Concepts
* Physical Networks and Mapping
* VLAN Tagging
* Usage of Virtual Switches
* Network Offerings
* API Usage and CloudMokey CLI
* Troubleshooting 

h2. *Assumptions*

* Used CloudStack previously and have basic network setup functional
* Able to provision VMs in basic network setup and your setup is fully operational
* Have understanding of basic terminology - go to this page for [CS4 Documentation|http://incubator.apache.org/cloudstack/docs/en-US/index.html]
* Root access to your hypervisors/virtual center and know your network topology
* Minimal experience with using APIs and/or CloudMonkey CLI
* _{+}Able to tell what config options must change to suit your environment{+}_

h2. Getting Started

*If you attempted to create Advanced Networking - but for some reason it* *{+}did not work{+}* *and you would like to* *{+}start over{+}*
* If you've tried previously to configure CS within your environment and it has not worked, before you begin, make sure you environment is clean from all previous attempts. I will use VSphere 5 as an example, translate it to a hypervisor of your choice.
* Remove all network modifications on your Hypervisors/Cluster created by CS
* Remove all storage modifications on your Hypervisors/Cluster created by CS
* Remove all the content from NFS primary and secondary datastore (i'm making assumption that data is not needed) or present a new primary and secondary NFS datastore
* Remove all the references from CS about the zone/pod/cluster/storage/networks/hosts you attempted to add but failed

*If there are remnants of previous setup - there is a chance your Zone Deployment may fail - therefore, please confirm that your environment is in

 pristine state as it was pre-CloudStack Zone Deployment

 operation*



h2. Pretext

The example environment in this tutorial consists of:

*Physical gear and OS:*

* 3 x HP BL465 G8 Servers as hypervisors
* Each hypervisor has 6 NICs
* VmWare vSphere 5.0 u1 and Virtual Center 5.0

*Hypervisor Network Layout -  same on all 3 hosts:*

\-------------------------------------------------------------\-

{code}


h2. Step 2: Create a network topology map for your reference

It is recommended to create layout of how you believe CloudStack IP allocation will take place, hence a quick write up of what networks get what assignments is very helpful. In my case, i wanted to be able to deploy DEV and MGMT VMs on vSwitch2 that uses VLAN tagging, while CS Managment/Storage VMs would reside on vSwitch0 (w/o VLAN tag). If you would like to move your CS Management VMs to VLAN tag'ed network

 -  you will need to confirm that you meet minimum requirements for CS described in Setup Guide and alter the CS Global Settings to change the default portgroup name "Management Network" for vmware - if its different. 

Network we need to define before we begin:

* *CloudStack Networks*
** *Management*  \- used by management nodes
** *Storage* \- used by secondary storage node
* *VM Instance Networks *
** *Public* \- network used for VMs and Internet (used only if you do Isolated Mode)
** *Guest* \- network used for internal VM communication

Below is the sample table created for easy of IP space accounting
|| VmWare Virtual Switch || CS Physical Network || VmWare PortGroup Name || Network || Subnet || Gateway || VLAN TAG || {color:#003366}{*}CS Traffic Type{*}{color} || {color:#003366}{*}Network IP Range *{color}\\ ||
| vSwitch0 | vSwitch0-CS-Dev-MGMT-PN

 | Management Network

 | 10.25.243.0

 | 255.255.255.0

 | 10.25.243.1

 | NONE | CS Management \\ | 10.25.243.140

 \- 10.25.243.146

 |
| vSwitch0 | vSwitch0-CS-Dev-MGMT-PN

 | Management Network

 | 10.25.243.0

 | 255.255.255.0

 | 10.25.243.1

 | NONE | CS Storage | 10.25.243.147

 \- 10.25.243.149

 |
| vSwitch2 | vSwitch2-CS-Dev-Public-PN

 | MGMT | 10.25.243.0

 | 255.255.255.0

 | 10.25.243.1

 | 1045 | Public | 10.25.243.195

 \- 10.25.243.199

 |
| vSwitch2 | vSwitch2-CS-Dev-Public-PN

 | DEV | 10.28.18.0

 | 255.255.254.0

 | 10.28.18.1

 | 1075 | Public | 10.28.19.195 - 10.28.19.199

 |
| vSwitch2 | vSwitch2-CS-Dev-MGMT-PN

 | MGMT | 10.25.243.0

 | 255.255.255.0

 | 10.25.243.1

 | 1045 | Guest | 10.25.243.150

 \- 10.25.243.194

 \\ |
| vSwitch2 | vSwitch2-CS-Dev-DEV-PN

 | DEV | 10.28.18.0

 | 255.255.254.0

 | 10.28.18.1

 | 1075 | Guest | 10.28.19.150 - 10.28.19.194

 |
You will need administrative VC user and password when configuring vSphere Cluster with CS.

The above table gives you an overview of what I have setup. Please note that while i have listed 6 Physical Network names - we only need to create 4.

vSwitch0-CS-Dev-MGMT-PN,

 vSwitch2-CS-Dev-Public-PN,

 vSwitch2-CS-Dev-MGMT-PN

 and vSwitch2-CS-Dev-DEV-PN.

Moreover, the concept of Public IP space in Advanced Network Shared setup does not apply - as it is meant to be used for internal Non-Isolated network. Therefore, i've allocated a very small amount of public IP space that will be used by CS management VMs only. If you are using Isolated mode, then you should assign more public IP addresses.

h2. Step 3: Add Zone/POD/Cluster via UI

Before you begin - you can see my cluster network setup in Virtual Center. I've removed all the remnants of existing CS network setup as i've done this several times before

.
  !redo-cs-advance-setup01.png|border=1!


This screenshot shows the storage setup - also clean from previous CS storage setup attempts.
  !redo-cs-advance-setup02.png|border=1!

Next is to import the System VM image for VmWare.
  !redo-cs-advance-setup03.png|border=1!

Login to CS with user that has admin privileges, this screenshot has Citrix CloudPlatform Logo - which is a bug that will be addressed in CS4.0.1
  !redo-cs-advance-setup04.png|border=1!

Navigate to *Infrastructure > Zones*, click Add Zone
  !redo-cs-advance-setup05.png|border=1!

Choose *Advanced* Zone

.  !cs-advance-setup0.png|border=1!

Complete the required fields for *Setup Zone* page
  !redo-cs-advance-setup07.png|border=1!

When you get to *Setup Network,* define your Physical Network Names, in my case I created 4 Physical Networks - as defined in Network Map above.

Note that I've placed CS *Management* and *Storage Traffic* on *vSwitch0-CS-Dev-MGMT-PN

*

The *Public* traffic on *vSwitch2-CS-Dev-Public-PN

*

The *Guest* traffic for 2 of my tagged VLANs on *vSwitch2-CS-Dev-MGMT-PN

* and *vSwitch2-CS-Dev-DEV-PN

*.

!redo-cs-advance-setup08.png|border=1!

Press Edit on each Traffic Type and and specify the respective VmWare vSwitch (reminder you must use non distributed virtual switch - the support for dVs will be in CS version 4.1). !redo-cs-advance-setup10.png|border=1!
Once you've defined all VMWare Traffic Labels for all of your Traffic Types, proceed to the next step.

Screenshot shows how i've defined 2 Public Networks for DEV and MGMT. This is only required of your are using Isolated Mode in your Network Offering. To make an analogy of what Isolated mode means, look at how AWS does IaaS. They offer an Internal IP address as well as External IP address. If If you are planning to run your environment in traditional manner where NATing is handled by Firewalls, your don't need to define Public Network for your guest VMs. A small public network IP range is required only for Management VMs.

  !redo-cs-advance-setup11.png|border=1!

Define you POD on the page below as well as IP space for CS Management Network
  !redo-cs-advance-setup12.png|border=1!

I've purposely left VLAN Range blank for both Physical Networks on Guest Traffic and will do this through API/CLI commands later. This function has not worked as expected through UI interface and Network has not been created.
  !redo-cs-advance-setup13.png|border=1!
On the Storage Traffic page, define the IP range for your storage network. For simplicity, i did not use a VLAN tagged network.
  !redo-cs-advance-setup14.png|border=1!
The screenshot below defined the VmWare cluster information, you will need a *user* that is able to login to a virtual center with *administrative* privileges.
  !redo-cs-advance-setup16.png|border=1!
Primary storage is where your guest VMs are going to live.In this example its NFS based, make sure you have proper network access to CS and your hypervisors. !redo-cs-advance-setup17.png|border=1!
Secondary storage is where your templates are going to reside, make sure it has proper network access to CS and your hypervisors.
  !redo-cs-advance-setup18.png|border=1!
Last but not least - if you've done everything properly, you should see a Ready Page to Launch a Zone.
  !redo-cs-advance-setup23.png|border=1!
I'd strongly recommend to open Virtual Center and look for all the progress CS makes when it goes through the setup process.
 

When you get to enable the Zone page, launch an ssh session to cloudstack server and tail the /var/log/cloud/management/management-server.log.

Note that you will be getting occasional warning and errors - as it tries to deploy multiple components - this is normal. What is not normal is when you see the same error repeating over and over for extended period of time - if thats the case, proceed to troubleshooting section.

Here is an example of what you may see..

.

  !redo-cs-advance-setup24.png|border=1!


_{+}The initial setup process may take upto 10 minutes depending on your environment. I'd recommend to wait it out and proceed only once the initial setup is successful. +_

Before we proceed, make sure that Storage VM (s-XXX-VM) and Proxy VM (v-XXX-VM) have been deployed.

At this point we are half way done and will proceed to a more sophisticated part of this tutorial.

h2. Step 4: Creating API Keys and Signature for CloudMonkey CLI

CloudStack provides a CLI called CloudMonkey. To install CloudMonkey, please visit this [CloudMonkey Tutorial|https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI] for detailed instructions. Once installed, proceed with generating required keys.

Navigate to Accounts tab and select the desired user

.

!redo-cs-advance-setup30.png|border=1!

h2. Step 5: Creating Network Offering and Networks

In order for VMs to have network functinality we need to create a network offering and networks. 

Network Offering is a collection of network services VM will have an access to. For example, if you would like to have DHCPD, DNS, LoadBalancing and UserData services offered, we need to create Network Offering with following services.

In order to create the Network Offerings, we must first Tag our Physical Networks in CloudStack.

Navigate to Infrastructure, Zones, choose your Zone and press on Physical Network.

  !redo-cs-advance-setup31.png|border=1!


Press on each Physical Network and define a tag. In the example below, you will see that I use {Physical Network Name}\-tag, this naming convention is optional and only my preference. Make sure to leave VLAN information blank, we will do this portion through API/CLI shortly.


  !redo-cs-advance-setup32.png|border=1!





*RAW IMAGES - NOT YET DOCUMENTED*

     



  

  








  !redo-cs-advance-setup34.png|border=1!

h3. *THIS DOCUMENT IS STILL BEING WORKED ON\!\!*\\

*This guide is geared more towards *{color:#99cc00}{*}CS 4.0{*}{color} *(build 140) and* {color:#99cc00}{*}vSphere5{*}{color}*, however, the same concept should apply to other hypervisors types like KVM and XEN.* *This is* {color:#99cc00}{*}NOT{*}{color} *a beginners guide,* *{+}please use your judgement and substitute values as necessary{+}{*}*.*
\\
\\

h2. *Disclaimer*\\  \\

h6. The {color:#ff0000}CS4.0 build 140{color} is defined on purpose in this document - there are known UI issues that  have been already addressed - however, since no update/patch has  been officially released to address these issues (at the time of  this writing)  \- in order to configure Advanced Networking - we must use  API calls to substitute what GUI could not accomplish. The newer  releases should address the UI limitations and API calls may not be  necessary. Nevertheless, its a good learning experience.\\  \\

h5. \\  \\

h2. *What This Guide Covers*\\

* CloudStack Advanced Network Concepts
* Physical Networks and Mapping
* VLAN Tagging
* Usage of Virtual Switches
* Network Offerings
* API Usage and CloudMokey CLI
* Troubleshooting 
\\

h2. *Assumptions*\\

* Used CloudStack previously and have basic network setup functional
* Able to provision VMs in basic network setup and your setup is fully operational
* Have understanding of basic terminology - go to this page for [CS4 Documentation|http://incubator.apache.org/cloudstack/docs/en-US/index.html]
* Root access to your hypervisors/virtual center and know your network topology
* Minimal experience with using APIs and/or CloudMonkey CLI
* _{+}Able to tell what config options must change to suit your environment{+}_
\\

h2. Getting Started\\

*If you attempted to create Advanced Networking - but for some reason it* *{+}did not work{+}* *and you would like to* *{+}start over{+}*
* If you've tried previously to configure CS within your environment  and it has not worked, before you begin, make sure you environment  is clean from all previous attempts. I will use VSphere 5 as an example,  translate it to a hypervisor of your choice.
* Remove all network modifications on your Hypervisors/Cluster created by CS
* Remove all storage modifications on your Hypervisors/Cluster created by CS
* Remove all the content from NFS primary and secondary datastore  (i'm making assumption that data is not needed) or present a new primary  and secondary NFS datastore
* Remove all the references from CS about the zone/pod/cluster/storage/networks/hosts you attempted to add but failed
*If there are remnants of previous setup - there is a chance your Zone  Deployment may fail - therefore, please confirm that your environment  is in pristine state as it was pre-CloudStack Zone Deployment operation*
\\
\\
\\

h2. Pretext\\

The example environment in this tutorial consists of:
\\

*Physical gear and OS:*
\\
* 3 x HP BL465 G8 Servers as hypervisors
* Each hypervisor has 6 NICs
* VmWare vSphere 5.0 u1 and Virtual Center 5.0
\\

*Hypervisor Network Layout -  same on all 3 hosts:*
\\

\-------------------------------------------------------------\-
\\
* *vSwitch0* \- Virtual Switch for Management \- 2 Physical NICs assigned 
** *Management Network* \- Portgroup for Management vnic vmk0
*** *NO VLAN TAGGING*
\\
* *vSwitch1* \- Virtual Switch for vMotion Network - 2 Physical NICs assigned
** *vMotion* \- PortGroup for vMotion operations
*** *NO VLAN TAGGING*
\\
* *vSwitch2* \- Virtual Switch for Guest VMs TRUNK Network - 2 Physical NICs assigned
** *MGMT* \- PortGroup for VMs on Management Network
*** *VLAN TAG/ID 1045*
** *DEV* \- PortGroup for VMs on Development Network
*** *VLAN TAG/ID 1075*
\\
\\
\\
\\
\\

h2. Step 1: Import system template\\

SSH to your CS4 environment
\\
\\

Mount the secondary NFS storage to /mnt/secondary on CS4
\\

Run
\\
\\
/usr/lib64/cloud/common/scripts/storage/secondary/cloud-install-sys-tmplt \-m /mnt/secondary/ \-u [http://download.cloud.com/templates/burbank/burbank-systemvm-08012012.ova] \-h vmware
\\
\\

h2. Step 2: Create a network topology map for your reference\\

It is recommended to create layout of how you believe CloudStack IP  allocation will take place, hence a quick write up of what networks get  what assignments is very helpful. In my case, i wanted to be able to  deploy DEV and MGMT VMs on vSwitch2 that uses VLAN tagging, while CS  Managment/Storage VMs would reside on vSwitch0 (w/o VLAN tag). If you  would like to move your CS Management VMs to VLAN tag'ed network -  you  will need to confirm that you meet minimum requirements for  CS described in Setup Guide and alter the CS Global Settings to change  the default portgroup name "Management Network" for vmware - if its  different. 
\\

Network we need to define before we begin:
\\
* *CloudStack Networks*
** *Management*  \- used by management nodes
** *Storage* \- used by secondary storage node
* *VM Instance Networks *
** *Public* \- network used for VMs and Internet (used only if you do Isolated Mode)
** *Guest* \- network used for internal VM communication
\\

Below is the sample table created for easy of IP space accounting
|| VmWare Virtual Switch || CS Physical Network || VmWare PortGroup Name || Network || Subnet || Gateway || VLAN TAG || {color:#003366}{*}CS Traffic Type{*}{color} || {color:#003366}{*}Network IP Range *{color}\\ ||
| vSwitch0 | vSwitch0-CS-Dev-MGMT-PN | Management Network | 10.25.243.0 | 255.255.255.0 | 10.25.243.1 | NONE | CS Management \\ | 10.25.243.140 \- 10.25.243.146 |
| vSwitch0 | vSwitch0-CS-Dev-MGMT-PN | Management Network | 10.25.243.0 | 255.255.255.0 | 10.25.243.1 | NONE | CS Storage | 10.25.243.147 \- 10.25.243.149 |
| vSwitch2 | vSwitch2-CS-Dev-Public-PN | MGMT | 10.25.243.0 | 255.255.255.0 | 10.25.243.1 | 1045 | Public | 10.25.243.195 \- 10.25.243.199 |
| vSwitch2 | vSwitch2-CS-Dev-Public-PN | DEV | 10.28.18.0 | 255.255.254.0 | 10.28.18.1 | 1075 | Public | 10.28.19.195 - 10.28.19.199 |
| vSwitch2 | vSwitch2-CS-Dev-MGMT-PN | MGMT | 10.25.243.0 | 255.255.255.0 | 10.25.243.1 | 1045 | Guest | 10.25.243.150 \- 10.25.243.194 \\ |
| vSwitch2 | vSwitch2-CS-Dev-DEV-PN | DEV | 10.28.18.0 | 255.255.254.0 | 10.28.18.1 | 1075 | Guest | 10.28.19.150 - 10.28.19.194 |
You will need administrative VC user and password when configuring vSphere Cluster with CS.
\\

The above table gives you an overview of what I have setup. Please  note that while i have listed 6 Physical Network names - we only need to  create 4.
\\

vSwitch0-CS-Dev-MGMT-PN, vSwitch2-CS-Dev-Public-PN, vSwitch2-CS-Dev-MGMT-PN and vSwitch2-CS-Dev-DEV-PN.
\\

Moreover, the concept of Public IP space in Advanced Network Shared  setup does not apply - as it is meant to be used for internal  Non-Isolated network. Therefore, i've allocated a very small amount of  public IP space that will be used by CS management VMs only. If you are  using Isolated mode, then you should assign more public IP addresses.
\\

h2. Step 3: Add Zone/POD/Cluster via UI\\

Before you begin - you can see my cluster network setup in Virtual  Center. I've removed all the remnants of existing CS network setup as  i've done this several times before.
 
\\
\\

This screenshot shows the storage setup - also clean from previous CS storage setup attempts.
 
\\

Next is to import the System VM image for VmWare.
 
\\

Login to CS with user that has admin privileges, this screenshot has  Citrix CloudPlatform Logo - which is a bug that will be addressed in  CS4.0.1
 
\\

Navigate to *Infrastructure > Zones*, click Add Zone
 
\\

Choose *Advanced* Zone
\\

. 
\\

Complete the required fields for *Setup Zone* page
 
\\

When you get to *Setup Network,* define your Physical Network Names, in my case I created 4 Physical Networks - as defined in Network Map above.
\\

Note that I've placed CS *Management* and *Storage Traffic* on *vSwitch0-CS-Dev-MGMT-PN*
\\

The *Public* traffic on *vSwitch2-CS-Dev-Public-PN*
\\

The *Guest* traffic for 2 of my tagged VLANs on *vSwitch2-CS-Dev-MGMT-PN* and *vSwitch2-CS-Dev-DEV-PN*.
\\
\\

Press Edit on each Traffic Type and and specify the respective VmWare  vSwitch (reminder you must use non distributed virtual switch - the  support for dVs will be in CS version 4.1).
Once you've defined all VMWare Traffic Labels for all of your Traffic Types, proceed to the next step.
\\

Screenshot shows how i've defined 2 Public Networks for DEV and MGMT.  This is only required of your are using Isolated Mode in your Network  Offering. To make an analogy of what Isolated mode means, look at how  AWS does IaaS. They offer an Internal IP address as well as External IP  address. If If you are planning to run your environment in traditional  manner where NATing is handled by Firewalls, your don't need to define  Public Network for your guest VMs. A small public network IP range is  required only for Management VMs.
\\

 
\\

Define you POD on the page below as well as IP space for CS Management Network
 
\\

I've purposely left VLAN Range blank for both Physical Networks on  Guest Traffic and will do this through API/CLI commands later. This  function has not worked as expected through UI interface and Network has  not been created.
 
On the Storage Traffic page, define the IP range for your storage network. For simplicity, i did not use a VLAN tagged network.
 
The screenshot below defined the VmWare cluster information, you will need a *user* that is able to login to a virtual center with *administrative* privileges.
 
Primary storage is where your guest VMs are going to live.In this  example its NFS based, make sure you have proper network access to CS  and your hypervisors.
Secondary storage is where your templates are going to reside, make sure  it has proper network access to CS and your hypervisors.
 
Last but not least - if you've done everything properly, you should see a Ready Page to Launch a Zone.
 
I'd strongly recommend to open Virtual Center and look for all the progress CS makes when it goes through the setup process.
\\

When you get to enable the Zone page, launch an ssh session to  cloudstack server and tail the  /var/log/cloud/management/management-server.log.
Note that you will be getting occasional warning and errors - as it  tries to deploy multiple components - this is normal. What is not normal  is when you see the same error repeating over and over for extended  period of time - if thats the case, proceed to troubleshooting section.
\\
\\

Here is an example of what you may see...
\\

 
\\
\\

_{+}The initial setup process may take upto 10 minutes depending on  your environment. I'd recommend to wait it out and proceed only once  the initial setup is successful. +_
\\

Before we proceed, make sure that Storage VM (s-XXX-VM) and Proxy VM (v-XXX-VM) have been deployed.
\\

At this point we are half way done and will proceed to a more sophisticated part of this tutorial.
\\

h2. Step 4: Creating API Keys and Signature for CloudMonkey CLI\\

CloudStack provides a CLI called CloudMonkey. To install CloudMonkey, please visit this [CloudMonkey Tutorial\|confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI|] for detailed instructions. Once installed, proceed with generating required keys.
\\

Navigate to Accounts tab and select the desired user (admin privileges required)
\\

h2. \\

Once on Users Tab, click on "View Users" to see additional permission and settings for this user.
\\
\\

Under Details Tab, press on the Key Icon to generate the API Keys.
\\

 
\\
\\

Launch CloudMokey CLI in your ssh terminal and setup the initial  settings/attributes in order for CLI to interact with CloudStack core  server.
\\
> set host <CS-hostname>
> set port 8080
> set apikey <put-your-api-key-for-your-user>
> set secretkey <put-your-secret-key-for-your-user>
> set prompt mycloudmonkey>

To test it, run "list zones" command, the example screenshot is below.
\\
\\

h2. Step 5: Creating Network Offering and Networks\\

In order for VMs to have network functinality we need to create a network offering and networks.&nbsp;
\\

Network Offering is a collection of network services VM will have an  access to. For example, if you would like to have DHCPD, DNS,  LoadBalancing and UserData services offered, we need to create Network  Offering with following services.
\\

In order to create the Network Offerings, we must first Tag our Physical Networks in CloudStack.
\\

Navigate to Infrastructure, Zones, choose your Zone and press on Physical Network.
\\

&nbsp;
\\
\\

Press on each Physical Network that has Guest Traffic Type and define a tag. In this example i have 2 Guest Traffic types one in each Physical Network respectively.

In the example below, you will see that I used (Physical Network Name)-tag, this naming convention is optional and only my preference.
\\
\\
\\
\\
\\
\\
\\
\\
\\
\\
\\
\\
\\

&nbsp;
\\

&nbsp;
\\

&nbsp;
\\

&nbsp;
\\

&nbsp;
&nbsp;
\\

&nbsp;
\\


&nbsp; !redo-cs-advance-setup35.png|border=1!

&nbsp; !redo-cs-advance-setup36.png|border=1!

&nbsp; !redo-cs-advance-setup37.png|border=1!

&nbsp; !redo-cs-advance-setup38.png|border=1!
&nbsp; !redo-cs-advance-setup39.png|border=1!

&nbsp; !redo-cs-advance-setup40.png|border=1!

&nbsp; !redo-cs-advance-setup41.png|border=1!